Overall response timeout.

[Granitosaurus]

• The bug is reproducible against the latest release and/or master.
• There are no similar issues or pull requests to fix it yet.

Currently both async and sync client requests are succeptible to low-and-slow "attacks" unless explicitly handled. In other words when request is being streamed as long as 1 byte every 2 seconds (or aprox.) is being generated the connection will not close or timeout with none of the current available settings.
It's possible for the server to serve single html page almost idefinitely and hang the client thread.

Maybe httpx should introduce some optional or even default handlers for this? Maybe it's already possible to hook something like this in easily?

To reproduce

resp = httpx.get(
    "http://httpbin.org/drip?duration=30&numbytes=30&code=200&delay=2",
    timeout=Timeout(connect=3, read=3, write=3, pool=3, timeout=3),
)

The above code snippet will take 30+ seconds to complete (though this could be practically infinity) disregarding any timeout settings. The only way to avoid this is to explitictly wrap everything in either asyncio.wait_for() for async code and for sync code seems to be much more complicated (?).

[florimondmanca]

@Granitosaurus Hi!

This is an interesting topic. To be honest, I'm not sure yet if/how we would be adding security knobs like these. Security folks tend to be paranoid 😄 and I'm not sure if they wouldn't just like to put things like slow lorry defense measures in place themselves, rather than relying on the underlying HTTP lib to do it for them. There's also a case to be made to include some minimal support built-in, but then we need to discuss opt-in vs opt-out; if opt-out then what is the default; etc.

Luckily, HTTPX already enables people to implement these measures themselves without having to do advanced concurrency. I was pinged on the tuf project recently to expose a bit more how HTTPX deals with timeouts, and discussed slow retrieval there as well: theupdateframework/python-tuf#1213 (comment)

From my POV, the best bet for folks might be to write a custom transport that wraps the stream and checks against slow sending of bytes. Something like this:

import time
from typing import Iterator
import httpcore
import httpx


class TooSlowError(Exception):
    pass


class SlowGuardTransport(httpcore.SyncHTTPTransport):
    def __init__(self, transport: httpcore.SyncHTTPTransport, min_rate: str) -> None:
        self._transport = transport
        value, unit = min_rate.split("/")
        self._min_rate = float(value) / {"second": 1, "minute": 60}[unit]

    def _wrap(self, stream: Iterator[bytes]) -> Iterator[bytes]:
        last_chunk_date = time.time()

        for chunk in stream:
            # Compute current recv rate.
            # TODO: Maybe use a rolling average to reduce false positives.
            now = time.time()
            elapsed = now - last_chunk_date
            rate = len(chunk) / elapsed

            if rate <= self._min_rate:
                raise TooSlowError(f"Server is sending data too slowly: {rate:.2f} < {self._min_rate:.2f}")

            last_chunk_date = now

            yield chunk

    def request(self, *args, **kwargs):
        status_code, headers, stream, ext = self._transport.request(*args, **kwargs)
        return status_code, headers, self._wrap(stream), ext


transport = httpcore.SyncConnectionPool()  # See docs for kwargs, or wait for upcoming `httpx.HTTPTransport()`.
transport = SlowGuardTransport(transport, min_rate="10/second")
with httpx.Client(transport=transport) as client:
    # 20/second: OK
    response = client.get("http://httpbin.org/drip?duration=5&numbytes=100&code=200")
    print(response)
    # 1/second: fails with `TooSlowError`
    # 'TooSlowError: Server is sending data too slowly: 0.93 < 10.00'
    response = client.get("http://httpbin.org/drip?duration=5&numbytes=5&code=200")

I'm not sure yet whether something like this would benefit from being built-in, with what API, or if it's okay for folks to just copy-paste and tweak a bit of this code using custom transports. (The Transport API is here exactly for enabling this kind of high-precision behaviors.)

[lovelydinosaur]

Heya, thanks for raising this.

I'd potentially be a bit cautious about labelling this under "attacks"/"security", since controlling the server and attacking the client isn't a very typical scenario, but yes I do think there's scope for a few more resource limits, which I've so far only noted in passing.

pause - I'm going to follow up on this more in a moment, but first lunch is up...

[florimondmanca]

@tomchristie As far as security scenarios go, slowloris seems like a typical one to take into account to me. Eg one is building a service that uses HTTPX to interact with some other server-side service that you can't trust to not be compromised and controlled by malicious agent. Clearly that's entering paranoia land, but one thing I've learnt is dealing with security is a lot about dealing with paranoia. 😆

Anyway, I've dropped the "security" label for now as I've realized it might come off as "here's a security breach we found in HTTPX" whereas that's not what this issue is about (which might be what you were getting at).

[lovelydinosaur]

It's just that slowlorris makes sense as a client-side attack on the server, allowing attackers to DDOS a server. Whereas in the converse case, where the server has been owned by the attacker, it's an odd scenario to envisage an "attack" that is "let's make the clients using this service really slow". Maybe that's a thing that's happened(?), but I've never heard of it.

You'd more typically see simply this kind of issue on the client side framed as "resource limits to ensure graceful degradation vs. overloaded servers".

Anyways, I think it's valid either ways around, and I think a useful guideline for considering resource limits is to think about it in the same way as services like Heroku treat resource limiting.

The cases that seem like likely candidates to me might be:

• Maximum total time allowed for an HTTP request. (Eg. 60 seconds)
• Maximum total size allowed for the body of an HTTP response. (Eg. 100MB)
• Maximum total number of concurrent requests on a client. (Eg. 100)

[Granitosaurus]

Yeah I did put "attack" in quotes because it's not really an attack but it serves a very similar function.
For example a way for a server to maliciously deal with web-scrapers (or any unwanted connections) would be to stream bytes slowly since many of http client libraries have no default handlers for that - so web-scrapers would hang indefinitely consuming resources and blocking flows which could be interpreted as a malicious action against the client.

I think those 3 proposals by @tomchristie would be more than enough to deal with this!

[simonw]

I think about this problem a lot. I've written a lot of code which accepts an untrusted URL from a user and attempts to fetch that URL - a few examples:

• OpenID and IndieAuth, where a user enters the URL to their identity provider and my backend needs to fetch it.
• A really common one these days is retrieving a URL in order to "unfurl" it - extract the title, any og:image tags etc in order to display a summary of the link on a page. Chat programs, comment systems etc all do this now.
• Features that allow users to provide a URL to a CSV file in order to import data.

For all of these I really appreciate being able to use an HTTP client that makes it easy to "safely" consume a malicious URL. I want to be able to set aggressive timeouts AND limit the number of bytes retrieved too (see #1392 for a previous question I've had around limiting the number of bytes).

The attacks I'm worried about here are deliberate denial-of-service attacks - users providing malicious URLs that exhaust my server resources.

The three limits Tom suggests above are exactly what I'm after.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.