Skip to content

Do more conservative URL normalization #758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
tarcieri merged 5 commits into from
Oct 6, 2023
Merged

Do more conservative URL normalization #758

tarcieri merged 5 commits into from
Oct 6, 2023
+118 −2

Conversation

@c960657
Copy link
Contributor

@c960657 c960657 commented Aug 3, 2023 •
edited
Loading

http.rb normalises URLs before making a request. This normalisation may reformat a perfectly valid URL. Even though the normalised URL in theory is equivalent to the original URL, some servers may not recognise this (see e.g. mastodon/mastodon#26219).

Some of these problems are due to a bug in Addressable (see sporkmonger/addressable#366). But even with that bug fixes, there is no reason to normalise the URL before fetching it, apart from fixing obviously invalid aspects such as spaces in the path. There is no benefit, only potential problems.

I suggest we take a more conservative approach and only escape characters in path and query outside the printable ASCII range, and never decode any existing percent-escape sequences. This is the approach taken by cURL, so that seems like a safe bet.

If somebody wants more “aggressive” normalisation for better caching in HTTP proxies or whatever, and they are aware of the risks connected with this, they can provide a custom normaliser or just call Addressable::URI#normalize before invoking htp.rb. But I don't think such potentially risky behaviour should be the default.

Fixes #654

@c960657 c960657 changed the title Conservative normalization Do more conservative URL normalization Aug 3, 2023
@c960657 c960657 mentioned this pull request Aug 4, 2023
Merged
@c960657 c960657 mentioned this pull request Aug 16, 2023
Closed
ixti
ixti reviewed Aug 28, 2023
View reviewed changes
ixti
ixti requested changes Aug 28, 2023
View reviewed changes
Copy link
Member

@ixti ixti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! This is great. Can you please move NORMALIZER proc spec in a separate file though, to follow RSpec naming conventions? And, please, add @private documentation tag to self.percent_encode

@c960657 c960657 requested a review from ixti August 28, 2023 17:00
@c960657
Copy link
Contributor Author

c960657 commented Oct 4, 2023

Done :-)

@tarcieri tarcieri merged commit 8b802bf into httprb:main Oct 6, 2023
@c960657 c960657 deleted the uri-normalizer branch December 28, 2023 17:42
@ClearlyClaire ClearlyClaire mentioned this pull request Feb 6, 2024
Closed
1 task
jcoyne added a commit to sul-dlss/stacks that referenced this pull request Mar 27, 2025
@jcoyne
Remove workaround for url normalization
fc63d2d 
This issue was fixed in httprb/http#758
@jcoyne jcoyne mentioned this pull request Mar 27, 2025
Merged
jcoyne added a commit to sul-dlss/stacks that referenced this pull request Mar 27, 2025
@jcoyne
Remove workaround for url normalization
bc26316 
This issue was fixed in httprb/http#758
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Aug 11, 2025
@ryoon
www/ruby-http: Update to 5.3.1
1478a6c 
Changelog:
## [5.3.1] - 2025-06-09

### Changed

- Revert switch to the native llhttp on MRI, as it's not compatible with
  standalone bundles
  ([#802](httprb/http#802))


## [5.3.0] - 2025-06-09

### Added

- (backported) Add .retriable feature to Http
- (backported) Add more specific ConnectionError classes
- (backported) New feature: RaiseError

### Changed

- (backported) Drop depenency on base64
- (backported) Cache header normalization to reduce object allocation
- (backported) Use native llhttp on MRI


## [5.2.0] - 2024-02-05

### Added

- Add `Connection#finished_request?`
  ([#743](httprb/http#743))
- Add `Instrumentation#on_error`
  ([#746](httprb/http#746))
- Add `base64` dependency (suppresses warnings on Ruby 3.0)
  ([#759](httprb/http#759))
- Add `PURGE` HTTP verb
  ([#757](httprb/http#757))
- Add Ruby-3.3 support

### Changed

- **BREAKING** Process features in reverse order
  ([#766](httprb/http#766))
- **BREAKING** Downcase Content-Type charset name
  ([#753](httprb/http#753))
- **BREAKING** Make URI normalization more conservative
  ([#758](httprb/http#758))

### Fixed

- Close sockets on initialize failure
  ([#762](httprb/http#762))
- Prevent CRLF injection due to broken URL normalizer
  ([#765](httprb/http#765))

[unreleased]: httprb/http@v5.3.0...5-x-stable
[5.3.0]: httprb/http@v5.2.0...v5.3.0
[5.2.0]: httprb/http@v5.1.1...v5.2.0

## 5.1.1 (2022-12-17)

* [#731](httprb/http#731)
  Strip brackets from IPv6 addresses in `HTTP::URI`.
  ([@jeraki])

* [#722](httprb/http#722)
  Add `on_redirect` callback.
  ([@benubois])

## 5.1.0 (2022-06-17)

* Drop ruby-2.5 support.

* [#715](httprb/http#715)
  Set default encoding to UTF-8 for `application/json`.
  ([@drwl])

* [#712](httprb/http#712)
  Recognize cookies set by redirect.
  ([@tkellogg])

* [#707](httprb/http#707)
  Distinguish connection timeouts.
  ([@YuLeven])

## 5.0.4 (2021-10-07)

* [#698](httprb/http#698)
  Fix `HTTP::Timeout::Global#connect_ssl`.
  ([@tarcieri])

## 5.0.3 (2021-10-06)

* [#695](httprb/http#695)
  Revert DNS resolving feature.
  ([@PhilCoggins])

* [#694](httprb/http#694)
  Fix cookies extraction.
  ([@flosacca])

## 5.0.2 (2021-09-10)

* [#686](httprb/http#686)
  Correctly reset the parser.
  ([@bryanp])

* [#684](httprb/http#684)
  Don't set Content-Length for GET, HEAD, DELETE, or CONNECT requests without a BODY.
  ([@jyn514])

* [#679](httprb/http#679)
  Use features on redirected requests.
  ([@nomis])

* [#678](httprb/http#678)
  Restore `HTTP::Response` `:uri` option for backwards compatibility.
  ([@schwern])

* [#676](httprb/http#676)
  Update addressable because of CVE-2021-32740.
  ([@matheussilvasantos])

* [#653](httprb/http#653)
  Avoid force encodings on frozen strings.
  ([@bvicenzo])

* [#638](httprb/http#638)
  DNS failover handling.
  ([@midnight-wonderer])


## 5.0.1 (2021-06-26)

* [#670](httprb/http#670)
  Revert `Response#parse` behavior introduced in [#540].
  ([@DannyBen])

* [#669](httprb/http#669)
  Prevent bodies from being resubmitted when following unsafe redirects.
  ([@odinhb])

* [#664](httprb/http#664)
  Bump llhttp-ffi to 0.3.0.
  ([@bryanp])


## 5.0.0 (2021-05-12)

* [#656](httprb/http#656)
  Handle connection timeouts in `Features`
  ([@semenyukdmitry])

* [#651](httprb/http#651)
  Replace `http-parser` with `llhttp`
  ([@bryanp])

* [#647](httprb/http#647)
  Add support for `MKCALENDAR` HTTP verb
  ([@meanphil])

* [#632](httprb/http#632)
  Respect the SSL context's `verify_hostname` value
  ([@colemannugent])

* [#625](httprb/http#625)
  Fix inflator with empty responses
  ([@LukaszMaslej])

* [#599](httprb/http#599)
  Allow passing `HTTP::FormData::{Multipart,UrlEncoded}` object directly.
  ([@ixti])

* [#593](httprb/http#593)
  [#592](httprb/http#592)
  Support informational (1XX) responses.
  ([@ixti])

* [#590](httprb/http#590)
  [#589](httprb/http#589)
  Fix response headers paring.
  ([@Bonias])

* [#587](httprb/http#587)
  [#585](httprb/http#585)
  Fix redirections when server responds with multiple Location headers.
  ([@ixti])

* [#581](httprb/http#581)
  [#582](httprb/http#582)
  Add Ruby 2.7.x support.
  ([@janko])

* [#577](httprb/http#577)
  Fix `Chainable#timeout` with frozen Hash.
  ([@antonvolkoff])

* [#576](httprb/http#576)
  [#524](httprb/http#524)
  **BREAKING CHANGE**
  Preserve header names casing.
  ([@joshuaflanagan])

* [#540](httprb/http#540)
  [#538](httprb/http#538)
  **BREAKING CHANGE**
  Require explicit MIME type for Response#parse
  ([@ixti])

* [#532](httprb/http#532)
  Fix pipes support in request bodies.
  ([@ixti])

* [#530](httprb/http#530)
  Improve header fields name/value validation.
  ([@Bonias])

* [#506](httprb/http#506)
  [#521](httprb/http#521)
  Skip auto-deflate when there is no body.
  ([@Bonias])

* [#489](httprb/http#489)
  Fix HTTP parser.
  ([@ixti], [@fxposter])

* [#546](httprb/http#546)
  **BREAKING CHANGE**
  Provide initiating `HTTP::Request` object on `HTTP::Response`.
  ([@joshuaflanagan])

* [#571](httprb/http#571)
  Drop Ruby 2.3.x support.
  ([@ixti])

* [3ed0c31](httprb/http@3ed0c31)
  Drop Ruby 2.4.x support.
jcoyne added a commit to sul-dlss/stacks that referenced this pull request Aug 12, 2025
@jcoyne
Remove workaround for url normalization
8911d31 
This issue was fixed in httprb/http#758
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tarcieri tarcieri tarcieri approved these changes

@ixti ixti Awaiting requested review from ixti

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Encoded plus (+) %2B in path segment somehow not working

3 participants

@c960657 @tarcieri @ixti