We know the output of the pass will be roughly the same size as the
input of the pass, so don't start from scratch with an empty vec that
will require multiple re-allocations.
- Codegen tests for `ArrayMin` (mirroring the existing `ArrayMax` ones)
- HIR opt tests asserting that `ArrayMax` / `ArrayMin` are emitted
- An Array#min redefinition test mirroring `test_dont_optimize_array_max_if_redefined`
A monomorphic-specialized call site (warmed up with class A) must still
return correct results when a second class B with its own ISEQ method of
the same name flows through it. The receiver type guard should send the B
receiver to the side exit rather than dispatching to A#foo.
This side-exit-on-unexpected-class behavior was not covered by an existing
test. Distinct return values (1 vs 2) make a broken guard observable.
Reduce the number of branches and only look up the shape once instead of
twice.
Brings a profiling-heavy run (profiling for 1000 calls) of lobsters from
700ms to 660ms profile time. Unfortunately getting the current time for
stats is still 50% of all profile time.
* ZJIT: Profile caller splat length distributions
Related to: https://github.com/Shopify/ruby/issues/1004
Before specializing sends with caller splats, collect the distribution of
caller splat array lengths.
The splat length determines how many `ArrayAref` operations and positional
arguments a `SendDirect` path needs. Specializing for only one observed length
could cause repeated side exits when a call site sees varying lengths, so we
need to understand whether these profiles are monomorphic or polymorphic first.
This does not change caller splat specialization or fallback behavior yet.
* ZJIT: Refine caller splat length profiling
Store splat length distributions in a side map since only splat call
sites use them. Record unknown lengths instead of dropping those samples
so the length profile stays aligned with the operand type profile.
Co-authored-by: Max Bernstein <tekknolagi@gmail.com>
Implements GC fast path for newrange instruction when we do not need to
call <=> method (when either endpoint is nil or both are fixnums).
Benchmark:
def run(max)
i = 0
while i < max
a = i..max
i += 1
end
end
30.times { run(10) }
run(10_000_000)
Result:
Benchmark 1: master
Time (mean ± σ): 221.8 ms ± 5.2 ms [User: 214.4 ms, System: 5.8 ms]
Range (min … max): 216.4 ms … 235.8 ms 13 runs
Benchmark 2: branch
Time (mean ± σ): 62.4 ms ± 1.6 ms [User: 56.0 ms, System: 5.1 ms]
Range (min … max): 60.7 ms … 69.7 ms 46 runs
Summary
branch ran
3.56 ± 0.12 times faster than master
The invokeblock fast path (InvokeBlockIseqDirect) passes captured self
plus each block argument in C argument registers, but
block_call_inlinable_iseq never bounded the argument count. Yielding 6+
arguments to a matching-arity block made gen_invoke_block_iseq_direct
emit a CCall with more operands than C_ARG_OPNDS, which aborts the
process in the backend:
assertion failed: data.opnds.len() <= C_ARG_OPNDS.len()
at zjit/src/backend/x86_64/mod.rs:302
Reject such blocks in block_call_inlinable_iseq so both the guarded and
the inlined-caller yield paths fall back to VM dispatch.
Previously map_ptr<T>() took *const T, so a reference like &IseqPtr
silently coerced to *const IseqPtr and mapped the address of the
reference itself instead of the pointer value, which is how the bug
fixed by the previous commit slipped in.
Constrain the parameter with an ActualPtr trait implemented only for
*const T and *mut T, so passing a reference no longer compiles.
Co-authored-by: Max Bernstein <tekknolagi@gmail.com>
gen_invokeblock_ifunc drops the callinfo and passes argv straight to
rb_vm_yield_with_cfunc, which hardcodes kw_splat = 0, so keyword
arguments lost their keyword-ness. Gate the ifunc specialization on
block_call_inlinable, the same check used for the inline ISEQ path.
* ZJIT: Fix trivial inlining for optional argument entries
Use SendDirect's jit_entry_idx when reading the callee bytecode for
trivial inlining, so optional-argument calls inspect the same entry
point that SendDirect selected.
This avoids inlining default-argument initialization code when the
caller actually provided the optional positional argument.
* ZJIT: Require trivial inline target to end at leave
Limit the trivial inliner to whole-ISEQ shapes that start at bytecode
index 0 and end with the second instruction's leave. Optional argument
entries can start in the middle of the ISEQ, so keep those SendDirect
calls for the general inliner, which handles jit_entry_idx.
This avoids folding default-argument initialization code when the caller
actually supplied the optional positional argument.
Co-authored-by: Takashi Kokubun (k0kubun) <takashikkbn@gmail.com>
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
This forces the caller to write `&` before each closure for no benefits.
It's also weird because another parameter uses `impl Fn...` so you end
up with a noisy asymmetry at the call site.
Use `impl Fn`.
`Function::successors` went through `find()`, which clones the entire terminator `Insn` on every call just to read its `BlockId` targets. Reverse-post-order traversal queries every block's successors many times over the course of optimization, so that clone occurred rather frequently. The new iterator reads the targets from a borrowed terminator and does not allocate at all.
Switching this to an iterator also avoids allocating a `Vec` for callers that just want to iterate through the successors.
jmp_ptr_bytes() sets how much room to reserve for a patchable jump -- it
spaces out patch points (PadPatchPoint) and bounds conditional jump width.
It was hardcoded to 5 instructions, an absolute load-address plus br, with
the check for the cheap case left commented out.
An arm64 `b` reaches +/-128MiB, so one 4-byte branch suffices whenever the
code region fits in that range, and the default --zjit-exec-mem-size is
64MiB. emit_jmp_ptr already emitted just a `b` there; only the reservation
was oversized, padding out up to 5x more nops than needed. Enable the check,
adding the CodeBlock::virtual_region_size() accessor it calls (YJIT has the
same one).
Two instructions in an S-byte region are at most S-4 bytes apart, so the
bound tests (S-4)/4. Testing S/4 would give up at exactly 128MiB, where
S/4 is 2^25 and the widest imm26 offset is 2^25-1.
The two emit_je snapshots shrink accordingly: same instructions, fewer
trailing nops. A new unit test pins both sides of the 128MiB boundary.
On a branch-heavy while loop (five `if`s per iteration), --zjit-dump-disasm
goes from 312 nops / 968 instructions to 29 / 685, and the loop runs 1.70x
faster under benchmark-driver; a 3-way if/elsif loop 1.23x. The win is
concentrated in branchy code, which is where the padding was. x86 is
unaffected; its 5-byte jmp reservation is already minimal.
Checked with `make zjit-test` (1817 pass), test_zjit_cli.rb (23 pass), and
bootstraptest under --zjit at call thresholds 1 and 2, both at the default
size and at --zjit-exec-mem-size=128, the boundary case.
Benchmark:
i = 0
def run(max)
i = 0
while i < max
a = i..100
i += 1
end
end
30.times { run(2) }
run(10_000_000)
Result:
Benchmark 1: master
Time (mean ± σ): 229.8 ms ± 3.2 ms [User: 221.7 ms, System: 6.2 ms]
Range (min … max): 226.3 ms … 235.2 ms 12 runs
Benchmark 2: branch
Time (mean ± σ): 67.4 ms ± 2.0 ms [User: 60.4 ms, System: 5.3 ms]
Range (min … max): 65.1 ms … 75.1 ms 43 runs
Summary
branch ran
3.41 ± 0.11 times faster than master
A tag established inside a live ZJIT frame can catch a longjmp without unwinding that native frame. Record that fact on rb_vm_tag and materialize only frames above the target, preserving its JITFrame link and keeping JITFrame saves at a single store.
THe downside is that we now always reserve one 8B reference for the
IMEMO/fields, even for structs that never have any ivars.
But it is an acceptable tradeoff given that ivars on structs aren't
rare.
Co-Authored-By: John Hawthorn <john@hawthorn.email>
The HIR printer passed references to cfunc fields into PtrPrintMap, so snapshots depended on addresses of cloned instruction storage.
Normalize the stored function pointers themselves and cover all C call instruction variants.
Sometimes we have too many stack slots to fit in one sub operand on
ARM. That's fine. We can iteratively subtract instead of doing it all in
one instruction.
Reset each existing instruction profile and reinstall zjit_* instructions after compiling a non-final version. Do the same for inlined ISEQs so materialized frames can collect fresh profiles.
Let interpreter profiling instructions collect the samples and make recompile exits only observe whether the exiting instruction has finished profiling.
