Add fallback to `require` when `require_relative` fails to load native
extensions. This addresses an issue on RHEL-based Linux distributions
where Ruby scripts and built native extension shared libraries are
installed in separate directories.
https://github.com/ruby/rubygems/commit/68599bd107
1. Introduce State to store all status.
2. Store State instance to the Ractor local storage if possible
3. Make `GET_TIME` (Method object) shareable if possible
3 is supporeted Ruby 4.0 and later, so the Rator support is works
only on Ruby 4.0 and later.
https://github.com/ruby/timeout/commit/54ff671c6c
The try-open_timeout-then-fallback-to-timeout introduced in
https://github.com/ruby/net-http/commit/1903cedd8cd0 works well, but when it errors
due to any reason in Rubies which do not support `open_timeout`, it
spits the rescued ArgumentError that is unrelated to user code and not
actionable.
Net::HTTP.start('foo.bar', 80)
/.../net-http-0.8.0/lib/net/http.rb:1691:in 'TCPSocket#initialize': Failed to open TCP connection to foo.bar:80 (getaddrinfo(3): nodename nor servname provided, or not known) (Socket::ResolutionError)
from /.../net-http-0.8.0/lib/net/http.rb:1691:in 'IO.open'
from /.../net-http-0.8.0/lib/net/http.rb:1691:in 'block in Net::HTTP#connect'
from /.../timeout-0.4.4/lib/timeout.rb:188:in 'block in Timeout.timeout'
from /.../timeout-0.4.4/lib/timeout.rb:195:in 'Timeout.timeout'
from /.../net-http-0.8.0/lib/net/http.rb:1690:in 'Net::HTTP#connect'
from /.../net-http-0.8.0/lib/net/http.rb:1655:in 'Net::HTTP#do_start'
from /.../net-http-0.8.0/lib/net/http.rb:1635:in 'Net::HTTP#start'
from /.../net-http-0.8.0/lib/net/http.rb:1064:in 'Net::HTTP.start'
(snip)
/.../net-http-0.8.0/lib/net/http.rb:1682:in 'TCPSocket#initialize': unknown keyword: :open_timeout (ArgumentError)
sock = TCPSocket.open(conn_addr, conn_port, @local_host, @local_port, open_timeout: @open_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
from /.../net-http-0.8.0/lib/net/http.rb:1682:in 'IO.open'
from /.../net-http-0.8.0/lib/net/http.rb:1682:in 'Net::HTTP#connect'
from /.../net-http-0.8.0/lib/net/http.rb:1655:in 'Net::HTTP#do_start'
from /.../net-http-0.8.0/lib/net/http.rb:1635:in 'Net::HTTP#start'
from /.../net-http-0.8.0/lib/net/http.rb:1064:in 'Net::HTTP.start'
(snip)
... 8 levels...
This patch suppresses the ArgumentError by moving the retry out of the
rescue clause.
https://github.com/ruby/net-http/commit/86232d62f5
- ### TL;DR
Bundler is heavily limited by the connection pool which manages a
single connection. By increasing the number of connection, we can
drastiscally speed up the installation process when many gems need
to be downloaded and installed.
### Benchmark
There are various factors that are hard to control such as
compilation time and network speed but after dozens of tests I
can consistently get aroud 70% speed increase when downloading and
installing 472 gems, most having no native extensions (on purpose).
```
# Before
bundle install 28.60s user 12.70s system 179% cpu 23.014 total
# After
bundle install 30.09s user 15.90s system 281% cpu 16.317 total
```
You can find on this gist how this was benchmarked and the Gemfile
used https://gist.github.com/Edouard-chin/c8e39148c0cdf324dae827716fbe24a0
### Context
A while ago in #869, Aaron introduced a connection pool which
greatly improved Bundler speed. It was noted in the PR description
that managing one connection was already good enough and it wasn't
clear whether we needed more connections. Aaron also had the
intuition that we may need to increase the pool for downloading
gems and he was right.
> We need to study how RubyGems uses connections and make a decision
> based on request usage (e.g. only use one connection for many small
> requests like bundler API, and maybe many connections for
> downloading gems)
When bundler downloads and installs gem in parallel 4f85e02fdd/bundler/lib/bundler/installer/parallel_installer.rb (L128)
most threads have to wait for the only connection in the pool to be
available which is not efficient.
### Solution
This commit modifies the pool size for the fetcher that Bundler
uses. RubyGems fetcher will continue to use a single connection.
The bundler fetcher is used in 2 places.
1. When downloading gems 4f85e02fdd/bundler/lib/bundler/source/rubygems.rb (L481-L484)
2. When grabing the index (not the compact index) using the
`bundle install --full-index` flag.
4f85e02fdd/bundler/lib/bundler/fetcher/index.rb (L9)
Having more connections in 2) is not any useful but tweaking the
size based on where the fetcher is used is a bit tricky so I opted
to modify it at the class level.
I fiddle with the pool size and found that 5 seems to be the sweet
spot at least for my environment.
https://github.com/ruby/rubygems/commit/6063fd9963
This is more concise and ruby does a better job performance-wise.
This used to be `bsearch_index` already but 6d8358c083 changed it.
https://github.com/ruby/prism/pull/1733#discussion_r1373702087 said:
> Yeah the edge case was that the value matched an element exactly
But surely there would be a test to show this behaviour?
Gets called as part of pretty-printing nodes.
Further reduces the time for `SnapshotsTest` by ~16% for me.
https://github.com/ruby/prism/commit/f448e2b995
- Depending on the native extension, it can greatly reduce compilation
time when executing recipes simultaneously.
For example on Prism:
```
# Before
time gem install prism
Building native extensions. This could take a while...
Successfully installed prism-1.6.0
1 gem installed
gem install prism 3.61s user 0.80s system 95% cpu 4.595 total
```
```
# After
time MAKEFLAGS="-j" gem install prism
Building native extensions. This could take a while...
Successfully installed prism-1.6.0
1 gem installed
MAKEFLAGS="-j" gem install prism 4.47s user 1.27s system 246% cpu 2.330 total
```
I don't think adding `-j` as a default is harmful, but I'm admitedly
not very knowledgable when it comes to compiler.
https://github.com/ruby/rubygems/commit/61340081c6
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
This allows for specifying the lockfile to read and write. It mirrors
the --gemfile option, and has higher priority than the lockfile
method in the Gemfile. It also mirrors the bundle lock --lockfile
option.
When the --lockfile option is used, it is applied twice. First, before
the Gemfile is read, to specify the lockfile to operate on, and again
after the Gemfile is read, so that if the Gemfile has a lockfile
method that overrides the defintion's lockfile, the --lockfile option
still has higher precedence.
https://github.com/ruby/rubygems/commit/17acdd4a89
This pattern is extremely common across the ecosystem, I don't think
it's reasonable to deprecate it.
I understand the performance argument, but perhaps the dependency
resolution algorithm can use another method that is private API
and only works with two `Version` instance.
https://github.com/ruby/rubygems/commit/024b4b547a
(https://github.com/ruby/rubygems/pull/9095)
* Rescue when deleting a non-existent cached gem file
When a gem was in the cache, but another process deletes it first, this
delete command fails.
To work around this, I'm rescuing from Errno::ENOENT and swalling the
error. The file is gone, and we can move on.
* Apply suggestion from @kou
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
---------
https://github.com/ruby/rubygems/commit/b30bcbc648
Co-authored-by: Hiroshi SHIBATA <hsbt@ruby-lang.org>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
If we use "system" variable in BUNDLE_VERSION on Bundler configuration,
we can use bundler version provided by system installation.
But the current logic returns the first activated version of bundler
like 2.7.2. It makes to confuse users.
https://github.com/ruby/rubygems/commit/4eb66d9549
Comparing version objects is a huge bottleneck in dependency solvers
(like inside Bundler). I would like to make comparing version objects
cheaper. Right now we support comparing version objects with strings by
trying to coerce the string to a version. So for example:
```ruby
Gem::Version.new("1") <=> "12"
```
I would like to deprecate and remove support for this feature so that we
can reduce the overhead of `def <=>`.
I'm not sure what version of RubyGems we could remove this from though.
https://github.com/ruby/rubygems/commit/81b7602183
This specifies the lockfile location. This allows for easy support
of different lockfiles per Ruby version or platform.
https://github.com/ruby/rubygems/commit/b54d65bc0a
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Colby Swandale <996377+colby-swandale@users.noreply.github.com>
This allows for the same behavior as including `lockfile false`
in the Gemfile. This allows you to get the behavior without
modifying the Gemfile, which is useful if you do not control the
Gemfile.
This is similar to the --no-lock option already supported by
`gem install -g Gemfile`.
https://github.com/ruby/rubygems/commit/6c94623881
Co-authored-by: Colby Swandale <996377+colby-swandale@users.noreply.github.com>
This allows you to specify the lockfile to use. This is useful if
you want to use different lockfiles for different ruby versions or
platforms. You can also skip writing the lockfile by using a false
value.
https://github.com/ruby/rubygems/commit/2896aa3fc2
Co-authored-by: Colby Swandale <996377+colby-swandale@users.noreply.github.com>
- The Logger is not thread safe when calling `with_level`.
This now becomes problematic because we are using multiple
threads during the resolution phase in order to fetch git gems.
https://github.com/ruby/rubygems/commit/380653ae74
- ### Problem
When you have a Gemfile that contains git gems, each repository will
be fetched one by one. This is extremelly slow.
A simple Gemfile with 5 git gems (small repositories) can take up
to 10 seconds just to fetch the repos.
We can speed this up by running multiple git process and fetching
repositories silmutaneously.
### Solution
The repositories are fetched in Bundler when `Source::Git#specs` is
called.
The problem is that `source.specs` is called in various places
depending on Gemfile.
I think the issue is that calling `source.specs` feels like that
as a "side effect" we are going to clone repositories. I believe
that fetching repositories should be an explicit call.
For instance:
```ruby
source "https://rubygems.org"
gem "foo", github: "foo/foo"
# The repository foo will be fetched as a side effect to the call to `source.spec_names`
# 6cc7d71dac/bundler/lib/bundler/source_map.rb (L21)
```
```ruby
source "https://rubygems.org"
gem "bar", source: "https://example.org"
gem "foo", github: "foo/foo"
# The repository foo will be fetched on a different codepath
# 6cc7d71dac/bundler/lib/bundler/source/rubygems_aggregate.rb (L35)
# That is because the gem "bar" has a source that doesn't have the `/dependencies` API
# endpoint and therefore Bundler enters a different branch condition.
```
I opted to add a self explanatory call to fetch the git source
repositories just before we start the resolution, and *just*
before any other calls to `source.specs` is performed.
https://github.com/ruby/rubygems/commit/f0ef526f23
For subclasses from Set, require `set/subclass_compatible`, and
extend the subclass and include a module in it that makes it more
backwards compatible with the pure Ruby Set implementation used
before Ruby 4.
The module included in the subclass contains a near-copy of the
previous Set implementation, with the following changes:
* Accesses to `@hash` are generally replaced with `super` calls. In
some cases, they are replaced with a call to another instance method.
* Some methods that only accessed `@hash` and nothing else are not
defined, so they inherit behavior from core Set.
* The previous `Set#divide` implementation is not used, to avoid
depending on tsort.
This fixes the following two issues:
* [Bug #21375] Set[] does not call #initialize
* [Bug #21396] Set#initialize should call Set#add on items passed in
It should also fix the vast majority of backwards compatibility issues
in other cases where code subclassed Set and depended on implementation
details (such as which methods call which other methods).
This does not affect Set internals, so Set itself remains fast. For
users who want to subclass Set but do not need to worry about
backwards compatibility, they can subclass from Set::CoreSet, a Set
subclass that does not have the backward compatibility layer included.
