364 Commits
Author SHA1 Message Date
a21a3b7d23 Fix StartTLS stripping vulnerability
Reported by Alexandr Savca in https://hackerone.com/reports/1178562

Co-authored-by: Shugo Maeda <[email protected]>
2021-07-07 19:48:22 +09:00
3ca1399150 Ignore IP addresses in PASV responses by default, and add new option use_pasv_ip
This fixes CVE-2021-81810.
Reported by Alexandr Savca.

Co-authored-by: Shugo Maeda <[email protected]>
2021-07-07 19:47:46 +09:00
nagachika ea347990a5 merge revision(s) 07786edd66f59a529d6febb2e0fe901782991755,1917afa34bca55ba1ea578234132b7e4479ea3c9,0c611d7f4fce67c64bee4815f263c55ef15561c4:
test/net/http/test_https.rb: Stop the error due to openssl 1.1.1h

	On some environments that uses OpenSSL 1.1.1h, the two tests now fail.

	http://rubyci.s3.amazonaws.com/android29-x86_64/ruby-master/log/20200924T062352Z.fail.html.gz
	https://github.com/ruby/ruby/runs/1159288773?check_suite_focus=true

	```
	  1) Failure:
	TestNetHTTPS#test_get [/data/data/com.termux/files/home/cb/tmp/build/20200924T062352Z/ruby/test/net/http/test_https.rb:47]:
	<"0\x82\x03\xED0\x82\x02\xD5\xA0\x03..."> expected but was
	<"0\x82\x03\xE30\x82\x02\xCB\xA0\x03...">.
	```

	Not sure why, but verify_callback now seems to receive only SERVER_CERT
	but not CA_CERT.
	It would be good to investigate the issue furthermore, but tentatively,
	I want to stop the failures.

	test/net/http/test_https.rb: the order of verify_callback seems to
	 vary

	... depending upon the environment.

	test/net/http/test_https.rb: The test logic was buggy

	The expected certs must be `[CA_CERT, SERVER_CERT]` before 1.1.1g and
	`[SERVER_CERT]` after 1.1.1h.
2020-09-30 07:50:06 +09:00
Yusuke Endoh 251f5d8226 Revert "lib/net/http/response.rb: support raw deflate correctly"
This reverts commit 5105240b1e.

In RFC 2616:

```
   deflate
        The "zlib" format defined in RFC 1950 [31] in combination with
        the "deflate" compression mechanism described in RFC 1951 [29].
```

So "Content-Encoding: deflate" means zlib format, not raw deflate.

[Bug #11268]
2019-12-16 23:39:10 +09:00
Yusuke Endoh 5105240b1e lib/net/http/response.rb: support raw deflate correctly
Net::HTTP had used `Zlib::Inflate.new(32 + Zlib::MAX_WBITS)` for all
content encoding (deflate, zlib, and gzip).
But the argument `32 + Zlib::MAX_WBITS` means zlib and gzip decoding
with automatic header detection, so (raw) deflate compression had not
been supported.

This change makes it support raw deflate correctly by passing an
argument `-Zlib::MAX_WBITS` (which means raw deflate) to
`Zlib::Inflate.new`.  All deflate-mode tests are fixed too.

[Bug #11268]
2019-12-16 23:20:42 +09:00
Kazuhiro NISHIYAMA a83c46e3fd Add more debug info
closed server doesn't have useful info.
So call inspect before close.
And add local debug code in comment.

https://rubyci.org/logs/rubyci.s3.amazonaws.com/unstable11x/ruby-master/log/20191215T092405Z.fail.html.gz
```
  1) Failure:
IMAPTest#test_connection_closed_without_greeting [/export/home/rubyci/chkbuild-tmp/tmp/build/20191215T092405Z/ruby/test/net/imap/test_imap.rb:483]:
[Net::IMAP::Error] exception expected, not #<RuntimeError: {:e=>#<Errno::EINVAL: Invalid argument - connect(2) for [::1]:41748>, :server=>#<TCPServer:(closed)>, :port=>41748, :server_addr=>"::1"}>.
```
2019-12-15 20:09:24 +09:00
Kazuhiro NISHIYAMA 68321dd1fa Move rescue for debug into block of assert_raise 2019-12-12 22:05:14 +09:00
Kazuhiro NISHIYAMA c20dd2f754 Add more debug info when Errno::EINVAL
https://rubyci.org/logs/rubyci.s3.amazonaws.com/unstable11x/ruby-master/log/20191212T072406Z.fail.html.gz
2019-12-12 17:38:27 +09:00
NARUSE, Yui af11efd377 fix ipaddr parameter of Net::HTTP.start to support proxy
54072e329c
2019-12-10 19:12:21 +09:00
Yusuke Endoh 660388f6c5 test/net/http/test_https.rb (test_get_SNI_failure): stop proxy settings
Because the test fails under HTTP proxy settings.

https://rubyci.org/logs/rubyci.s3.amazonaws.com/solaris10-gcc/ruby-master/log/20191210T000004Z.fail.html.gz
```
  1) Failure:
TestNetHTTPS#test_get_SNI_failure [/export/home/users/chkbuild/cb-gcc/tmp/build/20191210T000004Z/ruby/test/net/http/test_https.rb:81]:
[OpenSSL::SSL::SSLError] exception expected, not #<Net::HTTPServerException: 403 "Forbidden">.
```

The new SNI feature introduced at 54072e329c may need to be improved for
HTTP proxy environment.
2019-12-10 09:41:33 +09:00
NARUSE, Yui 54072e329c Add ipaddr optional parameter to Net::HTTP#start
to replace the address for TCP/IP connection [Feature #5180]

There're 3 layers of hostname:
* host address for TCP/IP
* TLS server name
* HTTP Host header value
To test DNS round robin or check server certificate from server local,
people sometimes want to connect server with given IP address but keep
TLS server name and HTTP Host header value.

closes [Feature #15215]
closes https://github.com/ruby/ruby/pull/1893
closes https://github.com/ruby/ruby/pull/1977
2019-12-09 20:21:49 +09:00
Jeremy Evans ffd0820ab3 Deprecate taint/trust and related methods, and make the methods no-ops
This removes the related tests, and puts the related specs behind
version guards.  This affects all code in lib, including some
libraries that may want to support older versions of Ruby.
2019-11-18 01:00:25 +02:00
Jeremy Evans c5c05460ac Warn on access/modify of $SAFE, and remove effects of modifying $SAFE
This removes the security features added by $SAFE = 1, and warns for access
or modification of $SAFE from Ruby-level, as well as warning when calling
all public C functions related to $SAFE.

This modifies some internal functions that took a safe level argument
to no longer take the argument.

rb_require_safe now warns, rb_require_string has been added as a
version that takes a VALUE and does not warn.

One public C function that still takes a safe level argument and that
this doesn't warn for is rb_eval_cmd.  We may want to consider
adding an alternative method that does not take a safe level argument,
and warn for rb_eval_cmd.
2019-11-18 01:00:25 +02:00
Jeremy Evans 2b6a9f3a1f Ignore Errno::EPIPE when sending requests in net/http
An EPIPE when sending the request should be ignored.  Even if you
cannot write more data, you may still be able to read the server's
response.

Fixes [Bug #14466]
2019-09-27 07:43:32 -07:00
Shugo Maeda 633ae3278e Add Net::FTP#features and Net::FTP#option
Patch by darkphnx (Dan Wentworth) .  Thanks!
[Feature #15964]
2019-09-02 14:43:51 +09:00
Espartaco PalmaandTakashi Kokubun 2665e5858f Adding missing test for Net::HTTPGenericRequest initializer (#1835)
A new exception is raised if an URI::HTTP is received and that object doesn't
have a hostname property.
Complementary to #1278
2019-08-16 01:38:46 +09:00
Nobuyoshi Nakada 11662c70b0 Test missing Content-Type warnings 2019-07-24 01:42:45 +09:00
Nobuyoshi Nakada c9826c20d2 Show the caller's location
* lib/net/http/header.rb: show the caller's location instead of
  the current lines.
2019-07-24 00:26:28 +09:00
Nobuyoshi Nakada 409b0ec4c3 Suppress unused variable warnings 2019-06-30 11:45:37 +09:00
Nobuyoshi Nakada 45ba027d08 Removed unused variables 2019-06-30 08:45:01 +09:00
Nobuyoshi Nakada 7ba14b029c Add max option to TestProtocol#create_mockio
* test/net/protocol/test_protocol.rb (TestProtocol#create_mockio):
  max option to limit maximum writable size at once, and unify
  mock method in test_write0_multibyte.
2019-06-29 17:52:57 +09:00
Yusuke Endoh 1cbc4f9876 test/net/imap/test_imap.rb: use Thread#stop? to wait for server thread
Still timeout occurs.  Retry for 5dd8fdd3f3.
https://rubyci.org/logs/rubyci.s3.amazonaws.com/unstable11x/ruby-master/log/20190625T032405Z.fail.html.gz
2019-06-25 17:22:02 +09:00
Yusuke Endoh 5dd8fdd3f3 test/net/imap/test_imap.rb: wait for the server thread to start
In some slow CI environments, the invocation of a thread seems very
slow.  This causes a test failure to attempt to connect a server that
does not start yet.

https://rubyci.org/logs/rubyci.s3.amazonaws.com/unstable11x/ruby-master/log/20190615T002420Z.fail.html.gz
https://rubyci.org/logs/rubyci.s3.amazonaws.com/unstable11x/ruby-master/log/20190611T022407Z.fail.html.gz
2019-06-15 11:06:39 +09:00
Yusuke Endoh 6096baea5d test/net/http/test_http.rb: Extend the timeout
https://rubyci.org/logs/rubyci.s3.amazonaws.com/unstable10s/ruby-master/log/20190610T071910Z.log.html.gz
2019-06-11 06:45:49 +09:00
Yusuke Endoh 60924636e5 test/net/http/test_http.rb: extend the timeout for Solaris CI
https://rubyci.org/logs/rubyci.s3.amazonaws.com/unstable10s/ruby-master/log/20190605T231909Z.fail.html.gz
2019-06-06 21:50:21 +09:00
mrkn 888ddda157 test/net/fixtures/Makefile: moved from test/net/imap
This Makefile can be used to update test certificates.  It should have been
moved at which the certificates was moved at r56836.

[ci skip]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66687 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-02 04:18:34 +00:00
kazu 1e0b49a293 Try to update cert (2nd try)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66686 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-02 03:29:01 +00:00
kazu f234e6c3d3 Try to update cert
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66685 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-01-02 03:08:20 +00:00
nobu 1680a13a92 Fix Net::Protocol::BufferedIO#write when sending large multi-byte string
This commit should fix Net::Protocol::BufferedIO#write when sending
large multi-byte string like following example.

```
$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
```

[Fix GH-2058]

From: Eito Katagiri <[email protected]>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66582 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-27 04:49:12 +00:00
naruse 1bf066dcfa Preserve HTTP header key as string [Bug #15394]
to prevent send Host header twice accidentally.

From: Sangyong Sim <[email protected]>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66319 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-10 23:54:14 +00:00
k0kubun ed42b0102d test_ftp.rb: loosen another timeout requirement
for Travis osx.
https://travis-ci.org/ruby/ruby/jobs/454864155

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65724 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-14 08:21:05 +00:00
k0kubun 2a477e0e44 test_ftp.rb: loosen timeout for Travis osx
https://travis-ci.org/ruby/ruby/jobs/454798071

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65723 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-14 07:44:30 +00:00
k0kubun 3f1524f0d2 test_ftp.rb: extend timeout for --jit-wait testing
to avoid random failures like
https://app.wercker.com/ruby/ruby/runs/mjit-test1/5be394b818310600284f2b50?step=5be394f1591ca800079b1329

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65620 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-08 04:06:15 +00:00
nobu bf26f847fa Exclude CI platform specific failures by --excludes option
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65265 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-21 03:38:52 +00:00
k0kubun 40d070843d skip tests broken with OpenSSL 1.1.1 on Travis osx
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65263 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-21 02:28:43 +00:00
normal 0783fd9366 test/net/imap/test_imap.rb (test_exception_during_idle): kill infinite looper
It is possible for Mutex#sleep (via ConditionVariable#wait)
to prematurely wake up under MJIT (because Mutex#sleep can't
handle spurious wakeups).  This affects @idle_done_cond in
Net::IMAP#idle and means the response handler may never
set `in_idle' to `true`.

In any case, ensure the infinite looping `raiser' thread
stops running when the test is done.

Will work on reducing the effect of spurious wakeups from
MJIT...

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64387 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-16 03:37:14 +00:00
rhe 33dd5d6970 net/http, net/ftp: skip SSL/TLS session resumption tests
Due to a bug in OpenSSL 1.1.0h[1] (it's only in this specific version;
it was introduced just before the release and is already fixed in their
stable branch), the callback set by SSLContext#session_new_cb= does not
get called for clients, making net/http and net/ftp not attempt session
resumption.

Let's disable the affected test cases for now. Another option would be
to fallback to using SSLSocket#session as we did before r64234. But
since only a single version is affected and hopefully a new stable
version containing the fix will be released in near future, I chose not
to add such workaround code to lib/.

[1] https://github.com/openssl/openssl/pull/5967

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64252 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-09 10:00:19 +00:00
rhe 1dfc377ae3 net/http, net/ftp: fix session resumption with TLS 1.3
When TLS 1.3 is in use, the session ticket may not have been sent yet
even though a handshake has finished. Also, the ticket could change if
multiple session ticket messages are sent by the server. Use
SSLContext#session_new_cb instead of calling SSLSocket#session
immediately after a handshake. This way also works with earlier protocol
versions.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64234 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-08 14:13:55 +00:00
naruse 8607229132 increase body size
To try to block writing on travis.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63749 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-25 12:11:31 +00:00
mame be850d7b11 test/net/http/test_httpresponse.rb: add testcases for net/httpresponse
This change adds testcases for Response#inspect and #code_type, and
improves line coverage of lib/net/http/response.rb (91.8 % to 92.94 %).

A patch from @owlworks.  https://github.com/ruby/ruby/pull/1898

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63738 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-24 09:31:09 +00:00
naruse 10485be8c2 remove debug print
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63723 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-22 11:57:06 +00:00
naruse fab030c2c1 check net.core.wmem_default and max instead of tcp
https://travis-ci.org/ruby/ruby/jobs/395416137

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63722 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-22 11:10:56 +00:00
naruse 38d0219ab0 show net.ipv4.tcp_wmem to debug failure on travis
https://travis-ci.org/ruby/ruby/builds/395318841

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63721 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-22 10:53:03 +00:00
naruse 404b155b54 assert_raise(Net::ReadTimeout) on Windows [Bug #14829]
From: MSP-Greg <[email protected]>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63613 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-08 16:32:24 +00:00
usa f69d45351b th is nil
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63607 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-08 06:15:23 +00:00
naruse 265bfdfbf5 skip write_timeout test on Windows
This test is about write_timeout.
To ensure it really raised Net::WriteTimeout, skip this test on Windows,
whose write returns immediately even for large data.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63598 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-07 06:18:23 +00:00
usa d7e8017519 Some platforms immediately returns from Socket#write
* test/net/http/test_http.rb (test_timeout_during_HTTP_session_write): on some
  platforms such as Windows immediately returns from Socket#write, and have to
  wait to read its response.  So, we can not handle Net::WriteTimeout and should
  handle Net::ReadTimeout instead.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63597 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-07 02:37:44 +00:00
naruse 8e37f17671 Add Net::HTTPClientException [Bug #14688]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63590 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-06 09:01:04 +00:00
naruse bd7c46a7aa Introduce write_timeout to Net::HTTP [Feature #13396]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63587 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-06-06 08:03:47 +00:00
shugo 59630683d6 net/imap: Fix ArgumentError in send_string_data
Thanks to ShockwaveNN (Pavel Lobashov) for reporting the bug.
[ruby-core:86990] [Bug #14750]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63396 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-05-11 12:39:23 +00:00