We don't need to call obj_free on certain types of objects anymore including
T_FLOAT, T_RATIONAL, T_COMPLEX, and T_OBJECT. It makes freeing those objects
faster.
Benchmark:
i = 0
while i < 10_000_000
Object.new
i += 1
end
Before:
Time (mean ± σ): 1.435 s ± 0.013 s [User: 1.706 s, System: 1.970 s]
Range (min … max): 1.418 s … 1.461 s 10 runs
After:
Time (mean ± σ): 1.230 s ± 0.027 s [User: 1.335 s, System: 1.743 s]
Range (min … max): 1.206 s … 1.294 s 10 runs
https://github.com/ruby/mmtk/commit/52f01ba90f
cl.exe's traditional preprocessor passes __VA_ARGS__ to a nested macro
as a single argument, so the multi-argument probe triggers:
gc/default/default.c(4395): warning C4003: not enough arguments for function-like macro invocation 'RUBY_DTRACE_GC_SWEEP_PAGE'
Expand the probe invocation through an identity macro so that the
arguments are re-scanned and split properly.
Previously we needed to check every object to determine whether or not
it might have an entry in the geniv table.
This commit adds a function, rb_gc_vm_weak_table_essential_p, which
allows gc.c to decide which tables to always handle in batch.
MMTK will continue to call rb_gc_vm_weak_table_foreach on all tables.
This only changes the default GC to sweep the "essential" tables first,
but continue to call rb_gc_obj_free_vm_weak_references on other objects.
This makes the geniv table "essental", as it's usually small and needs
to be tested on every type of object. It leaves the fstring table,
symbol table, callinfo table, and overloaded CME table as-is because
those tables are (mostly) larger and are only needed by specific object
types.
When gc_move relocates an object to a different slot size,
rb_gc_obj_changed_slot_size may re-embed it: the fields move from the
fields_obj into the object itself and the object references them
directly. The write-barrier history for those (possibly young) values
lived on the discarded fields_obj, so an old unremembered object then
violates the O->Y invariant, which gc_verify_internal_consistency
reports as a WB miss. Remember the object on such moves.
Remembering inside rb_gc_obj_changed_slot_size (as #17866 tried) does
not work for this path: gc_move calls it before the destination's age
bits are restored, so RVALUE_OLD_P is false there, and gc_move
overwrites the destination's remembered bits afterwards anyway.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ruby requires GCC 4.9 or later as noted in configure.ac, so these
blocks guarded by __GNUC__ == 4 && __GNUC_MINOR__ == 4 have been
preprocessed away on every supported compiler for years.
This commit implements support in ZJIT for inlining GC fast path. The GC
fast path is defined in gc_fastpath.rs for both the default GC and MMTk.
For the default GC, the fast path uses the bump pointer allocator that
was implemented in https://github.com/ruby/ruby/pull/17201. If the bump
pointer region is full, it will bail out and fall back to the slow path.
This GC fast path is used by the newarray instruction in ZJIT in the empty
array case. We can see significant improvements in the following microbenchmark,
which allocates 100 million empty arrays:
def run(max)
i = 0
while i < max
a = []
i += 1
end
end
30.times { run(2) }
run(100_000_000)
This microbenchmark runs 1.5x faster on my machine:
Benchmark 1: master
Time (mean ± σ): 583.3 ms ± 10.1 ms [User: 569.3 ms, System: 12.0 ms]
Range (min … max): 571.9 ms … 600.0 ms 10 runs
Benchmark 2: branch
Time (mean ± σ): 389.5 ms ± 8.2 ms [User: 375.0 ms, System: 12.1 ms]
Range (min … max): 380.2 ms … 405.3 ms 10 runs
Summary
branch ran
1.50 ± 0.04 times faster than master
SHAPE_MAX_CAPACITY needed to be below or equal to the maximum capacity
allocatable by the GC otherwise it would crash. This commit decouples that
by adding back max_capacity in the rb_shape_tree_t and adding a new GC
API function rb_gc_impl_max_allocation_size to report the maximum size
allocatable by the GC.
This commit implements a fast path that inlines mmtk_post_alloc for Immix.
The benchmark results show a decent speed up in allocation performance.
GC.disable
i = 0
while i < 10_000_000
Object.new
i += 1
end
Before:
Time (mean ± σ): 506.3 ms ± 5.0 ms [User: 442.8 ms, System: 108.8 ms]
Range (min … max): 497.8 ms … 513.2 ms 10 runs
After:
Time (mean ± σ): 473.9 ms ± 2.4 ms [User: 409.4 ms, System: 108.8 ms]
Range (min … max): 470.4 ms … 478.1 ms 10 runs
https://github.com/ruby/mmtk/commit/663caa25c0
Instead lets run a single sweep step per heap and see if we can reduce
malloc_increase enough to carry on without requiring a complete sweep
finish and a big pause
https://github.com/ruby/ruby/pull/16919 changed malloc counters to be
reset in `gc_sweep_finish`. However, this introduced two problems:
1. If a GC is triggered due to the `malloc_limit` being hit, then it would
run a GC with lazy sweeping, during lazy sweeping Ruby code could call
`xmalloc`, which would immediately finish sweeping by calling `gc_rest`
because the `malloc_increase` is not reset.
2. When `gc_rest` is called, it immediately completes sweeping. However,
all of the counters for the freed memory is lost because `malloc_increase`
is reset at the end. Previously, lazy sweeping would be interleaved
with Ruby code execution, meaning `xfree` was interleaved with `xmalloc`,
allowing for more memory to be allocated before `malloc_increase` hits
`malloc_limit`.
The issue can be reproduced by this script:
live = []
500.times do
live << Array.new(200) { String.new(capacity: 64 * 1024) }
end
puts GC.stat
Before:
{count: 235, ..., minor_gc_count: 196, major_gc_count: 39}
After:
{count: 196, ..., minor_gc_count: 153, major_gc_count: 43}
When Ruby is built with `--with-modular-gc`, the default GC module is
built with `BUILDING_MODULAR_GC` defined.
The existing code stubbed out `probes.h` and `RUBY_DTRACE_GC_HOOK` in
this build path, so any DTrace script attached against the modular
default GC saw zero GC events.
This PR enables the GC probes in the modular GC library so they fire
when Ruby is built with `--enable-dtrace --with-modular-gc`
To test: build with `./configure --enable-dtrace
--with-modular-gc=/tmp/modgc`, then `make install-modular-gc
MODULAR_GC=default`.
Probes are present in the shared library:
```console
$ sudo bpftrace -l 'usdt:/tmp/modgc/librubygc.default.so:ruby:*'
usdt:/tmp/modgc/librubygc.default.so:ruby:gc__mark__begin
usdt:/tmp/modgc/librubygc.default.so:ruby:gc__mark__end
usdt:/tmp/modgc/librubygc.default.so:ruby:gc__sweep__begin
usdt:/tmp/modgc/librubygc.default.so:ruby:gc__sweep__end
```
They fire under a GC-heavy workload (`test.rb` contains `100000.times { Object.new }; GC.start`):
```console
$ sudo bpftrace -e '
usdt:/tmp/modgc/librubygc.default.so:ruby:gc__mark__begin { @mark_begin++ }
usdt:/tmp/modgc/librubygc.default.so:ruby:gc__mark__end { @mark_end++ }
usdt:/tmp/modgc/librubygc.default.so:ruby:gc__sweep__begin { @sweep_begin++ }
usdt:/tmp/modgc/librubygc.default.so:ruby:gc__sweep__end { @sweep_end++ }
' -c 'env RUBY_GC_LIBRARY=default ./miniruby --disable=gems test.rb'
@mark_begin: 2
@mark_end: 2
@sweep_begin: 18
@sweep_end: 18
```
`gc_prof_mark_timer_start` and `gc_prof_mark_timer_stop` include DTrace
hooks for the `MARK_BEGIN` and `MARK_END` events, respectively.
Previously, those probes are only triggered in `gc_marks`. However,
`gc_marks_continue` and `gc_rest` also contain marking activities, but
are not captured by the probes.
We move the invocation of `gc_prof_mark_timer_start` and
`gc_prof_mark_timer_stop` into `gc_marking_enter` and `gc_marking_exit`
to ensure all marking activities are captured by the probes.
