THe downside is that we now always reserve one 8B reference for the
IMEMO/fields, even for structs that never have any ivars.
But it is an acceptable tradeoff given that ivars on structs aren't
rare.
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Numerous codepath have the same `switch/case` with very little
differing logic compared to `rb_obj_fields`, might as well
use it everywhere and share more code.
Followup: https://github.com/ruby/ruby/pull/17631
Now that complex `RObject` always have an IMEMO/fields, we no
longer need to have an external `st_table` to replicate the
memory layout of `RObject`.
Instead of spilling into a raw buffer, `RObject` now spills
into an `IMEMO/fields` object like other types.
From an instance variable layout standpoint, an extended `RObject`
is now identical to a `RTypedData`, as in they both store the
reference to their `IMEMO/fields` at the same offset (`VALUE * 2`).
One positive consequence of this is that the only case where a `T_OBJECT`
needs sweeping is if a finalizer was registered.
YJIT now side exit when it need to write a new ivar into a `RObject`
that is out of space. This is unlikely to cause a performance regression
as this codepath isn't supposed to happen after warmup given `max_iv_count`
is recorded on classes, so future objects should be large enough.
Hence it's best not to waste executable memory for such codepath.
ZJIT lost support for writting into extended `RObject`.
Most of the code to support it is there it's just missing an
implementation of `RBASIC_SET_SHAPE_ID`, which recently changed
to strip some bits out of the shape (see comments).
I will leave it to the ZJIT team to implement it (sorry).
There are a few future cleanups planned that I keep for followups:
- We can now get rid of the `ROBJECT_HEAP` flag, it's redundant with
the shape layout bits.
- `imemo_fields` can now embed the `st_table` when complex.
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Co-authored-by: Randy Stauner <randy@r4s6.net>
Like the capacity part, the layout part of an object shape
almost never changes. The few exceptions are:
- On allocation.
- On being compacted by GC.
- When RObject oberflows.
As such it simplifies a lot of code if `RBASIC_SET_SHAPE_ID` strips
the layout bits, as we often copy the shape from IMEMO/fields to
the owner object and vice-versa.
Also change `RBASIC_SET_SHAPE_ID_WITH_CAPACITY` into
`RBASIC_SET_FULL_SHAPE_ID` so it can be used for assigning both
capacity and layout.
Set RUBY_TYPED_THREAD_SAFE_FREE on TypedData types whose dfree function
is trivially safe, and only frees its own memory
Types not yet marked as THREAD_SAFE_FREE:
- id2ref_tbl_type: sets the process-global id2ref_tbl to NULL.
- mutex_data_type: unlinks itself from another thread's keeping_mutexes list.
- autoload_data_type: deletes nodes from a shared intrusive list.
- autoload_const_type: deletes itself from a shared list.
- rb_cont_data_type: mutates the shared fiber pool and the global first_jit_cont list.
- rb_fiber_data_type: delegates to cont_free.
- FiberPoolDataType: mutates shared fiber pool free-lists.
- ractor_data_type: reconfigures global VM event-hook flags and call caches.
- exported_object_registry: frees a global table under the VM lock.
- rb_box_data_type (Box::Entry): unlinks classext from other live class and module objects.
- box_ext_cleanup_type: dereferences another String object and calls unlink.
- monitor_data_type: uses the default free, so the flag has no effect.
Co-authored-by: Luke Gruber <luke.gruber@shopify.com>
* Reserve 2 bits for expressing object layout
We would like to make instance variable reads in the JIT compiler faster
(as well as simplify the JIT implementation). Currently, in order to
read an instance variable, we have to:
1. Test for heap object
2. Load object to a 64 bit register
3. Mask the object header
4. Bit test against the masked header
5. JNE
6. Load field
We would like to:
1. Test for heap object
2. Load object shape to a 32 bit register
3. Bit test against the shape
4. JNE
5. Load field
The way we fetch instance variables is not consistent across objects.
In order to realize our goal, we need to encode object layout inside the
shape. If we encode object layout inside the shape, then the shape
itself will guarantee that the access pattern generated by the JIT
compiler is correct.
We should encode the following load patterns into the shape tag bits.
This way we can share shapes on transitions, but be able to
differentiate the access patterns for the JIT compiler. In other words,
two objects can have an `@a -> @b -> @c` transition and share the same
shape, but the tag bits can differentiate the access pattern so that the
JIT compiler can be confident that the machine code is correct.
Here are the patterns:
1. Embedded/Extended T_OBJECT Instance Variables
Objects with direct references to instance variables or via malloc
buffer
2. Objects with fields_objects fields
These are Data and TypedData objects. They have an associated axillary
imemo/fields object that stores the instance variables. The access
pattern is `object[2] + 2`. The fields object is the 3rd field, and the
instance variables start at +2 inside the fields object. The fields
object itself is a Ruby object, so it contains the usual header bits +
class headers.
3. Non Boxable Classes / Modules
This is similar to Objects with fields_objects, but the fields object is
stored at a different offset. We’re differentiating this from boxable
classes and modules because those are harder to support.
4. Other
"Other" pattern is for objects that are rare, or have
difficult-to-implement access patterns. This includes:
* Boxable classes and modules
* Structs (for now)
* Objects that use the geniv table
Proposed shape bit layout:
```
Current shape_id_t is 32 bits:
31 28 27 26 25 24 23 22 19 18 0
+-----------+--+--+--+--+--+------------+----------------------------+
| unused |L1|L0|OI|FR|CX| heap index | shape tree offset |
+-----------+--+--+--+--+--+------------+----------------------------+
| | | | | | |
| | | | | | +-- bits 0-18: SHAPE_ID_OFFSET_MASK
| | | | | +--------------- bits 19-22: SHAPE_ID_HEAP_INDEX_MASK
| | | | +------------------ bit 23: SHAPE_ID_FL_COMPLEX
| | | +--------------------- bit 24: SHAPE_ID_FL_FROZEN
| | +------------------------ bit 25: SHAPE_ID_FL_HAS_OBJECT_ID
+--+--------------------------- bits 26-27: SHAPE_ID_LAYOUT_MASK
```
The important part about these layout patterns is that they do not
reflect the _type_ of object, only how the object is laid out in memory.
For example, we currently treat structs as "other", but we can refactor
them to have the same layout as "Objects with fields_objects", and when
we do that they should get a different bit in the shape header.
This commit only reserves the two bits, it doesn't use them in the JIT
compiler yet.
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Co-Authored-By: Max Bernstein <tekknolagi@gmail.com>
* Update gc.c
Co-authored-by: Nobuyoshi Nakada <nobu.nakada@gmail.com>
* Update shape.h
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
* fix function name
* Update shape.c
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
* fix function name
* Revert "Update shape.c"
This reverts commit 900711defc6c541a93f3393a350819ae88cf87f1.
* add comment
---------
Co-authored-by: John Hawthorn <john@hawthorn.email>
Co-authored-by: Max Bernstein <tekknolagi@gmail.com>
Co-authored-by: Nobuyoshi Nakada <nobu.nakada@gmail.com>
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
This allows us to treat the two types of T_DATA the same way when
reading instance variables.
Non-TypedData T_DATA are deprecated, so making them faster isn't
particularly important, but it is helpful to avoid needing to test for
TypedData and adding another special case.
Now that we have 1024B slots, we can store up to 126 fields inline.
Objects larger than this are rare if not non-existent, hence we can
get rid of the `malloc` path for imemo/fields and simply transition
to `TOO_COMPLEX`.
This additionally allows to shrink `attr_index_t` from 16 to 8B.
Note: the ZJIT "ivar on extended" tests are renamed as "complex" because
"extended" AKA malloc allocated imemo/fields no longer exists.
They're now complex fields, AKA st tables.
rb_class_allocate_instance: start as complex when over max_fields
If `RCLASS_MAX_IV_COUNT` is over `max_fields`, allocating a large
slot to end up transitioning to `TOO_COMPLEX` is wasteful.
We might as well start as complex directly.
Expose both `rb_obj_shape_` functions that take a `VALUE`
and `rb_shape_` functions that take a `shape_id`.
Make common transition functions such as `complex` and `frozen`
inlineable.
Also get rid of RB_SET_SHAPE_ID and rb_set_boxed_class_shape_id.
Previously, rb_ivar_foreach would walk up the shape tree, but yield
instance variables to the callback as it went. If the object shape was
modified during this callback, particularly with removing an instance
variable, it could result in reading free'd memory.
This commit solves this by adding a version which buffering all instance
variable names and values before calling the callback, giving a snapshot
of the object at the time rb_ivar_foreach_buffered is called.
The buffer is made with ALLOCV_N, so the performance difference should
be minimal, and I don't think this method is particuarly heavily used.
[Bug #21996]
[Bug #21952]
Solves the double-free or use after-free concern with boxes.
Now entries can safely be used for copy-on-write.
Also is likely necessary to make it save to read cvar from
secondary ractors, as allowed since: ab32c0e690b805cdaaf264ad4c3421696c588204
The previous invalidation walked subclasses but missed cvars from
included modules, and skipped invalidation when creating a new cvar on
modules entirely. Always invalidate when a new class variable is
created, since this should be relatively rare.
We previously bumped the global state on any module inclusion, which
should be far more common than this.
[Bug #21978]
Initialize `entry` and `var` to NULL. They are assigned inside
RB_VM_LOCKING() but used after it, and the compiler cannot prove the
locked block always executes.
Today you can read instance variables from non-main Ractors, but many
Rails applications use cvars, and we cannot read them.
For example:
```ruby
class Foo
# This is NOT allowed to be read in non-main Ractors
@@bar = 123
def self.bar; @@bar; end
# This is allowed to be read in non-main Ractors
@baz = 123
def self.baz; @baz; end
end
# This is OK
Ractor.new {
p Foo.baz
}.value
# Exception here
Ractor.new {
p Foo.bar
}.value
```
This commit changes the semantics of cvars to be the same as instance
variables:
* It's ok to read Ractor shareable objects from the non-main Ractor
* It's NOT ok to write from the non-main Ractor
[Feature #21942]
Currently we maintain the subclasses list for two separate purposes (we essentially have to different relationships we're putting into the same list):
1. On a T_MODULE, we track the T_ICLASSes created to include it into
other classes. Used for method invalidation and propagating includes
on the module that happen after it's been used
2. On a T_CLASS/T_ICLASS, we track the T_CLASS/T_ICLASS which are the
immediate children of the class. We use this for method invalidation,
some cvar things, and to iterate through subclasses.
Purpose 1 does not have any issues with box, the T_ICLASS always belongs
to one specific module and that's immutable. This list can be box-global
(always use the prime classext or hoist it out) and only needs to be
pruned during free. If we care about behaviour under a particular box
(ie. the propagating includes), we should look up the current box being
modified on the ICLASS itself.
Purpose 2 is more complicated. It currently tracks the immediate
children, the T_CLASS or T_ICLASS whose super points back. Because super
is per-box and is mutable (include/prepend insert ICLASSes into the
chain) we need to update the list on include/prepend, entries must be
per-box, and we can have multiple entries per-box. *I propose we
simplify this by no longer tracking the immediate subclass*, but instead
tracking the T_CLASS -> ... -> T_CLASS relationship, ie. the inverse of
rb_class_superclass. That relationship is the same across all boxes and
immutable after Class creation.
As a special case the ICLASS for refinements are also added to the
purpose 2 list (on T_CLASS). As those ICLASS do not chain to an eventual
leaf T_CLASS.
When we need to find the classes which have included a module, we can
use the module subclasses list to find the ICLASS and then use
RCLASS_INCLUDER. If we needed to iterate all T_ICLASS, we could then
walk up the CLASS_SUPER chain, but I didn't find anywhere we needed to
do that.
Profiling of `ruby --disable-all -e 1` shows between 5 and 10%
of the time is spent in `_platform_memset`, called from `rb_gccct_clear_table()`,
itself called from `clear_method_cache_by_id_in_class()`.
Which makes sense, during boot we define numerous methods, and after
each we have to invalidate the `gccct`, which is done by zeroing
1023 pointers, so roughtly 8kiB of memory.
By keeping track of whether the table was used since the last clear,
we can save a lot of useless memory writes.
This may also be beneficial when loading user defined code, as it's unlikely
the `gccct` would be dirtied between two `def`.
NB: profiling such a short lived process gives a lot of variance, but
with a very high sampling rate and multiple attempts, `clear_method_cache_by_id_in_class`
is always in the top 2 hotspots.
Before this patch, Ractor::IsolationError reported an incorrect constant
path when constant was found through `rb_const_get_0()`.
In this code, Ractor::IsolationError reported illegal access against
`M::TOPLEVEL`, where it should be `Object::TOPLEVEL`.
```ruby
TOPLEVEL = [1]
module M
def self.f
TOPLEVEL
end
end
Ractor.new { M.f }.value
```
This was because `rb_const_get_0()` built the "path" part referring to
the module/class passed to it in the first place. When a constant was
found through recursive search upwards, the module/class which the
constant was found should be reported.
This patch fixes this issue by modifying rb_const_search() to take a
VALUE pointer to be filled with the module/class where the constant was
found.
[Bug #21782]
The "EXIVAR" terminology has been replaced by "gen fields"
AKA "generic fields".
Exivar implies variable, but generic fields include more than
just variables, e.g. `object_id`.
