HTTPS Proxy support feature requests

[matteo-prosperi]

Hello,
As the support of HTTPS proxies is becoming more common, we need to execute HTTP and HTTPS requests through an HTTPS proxy.

To be clear, this is not about executing an HTTPS request through an HTTP proxy (which works well), it is about connecting to the proxy itself using HTTPS.

As an additional requirement, we are also supposed to authenticate the client using client certificates.

When I try to use a proxy created from an https URI

            HttpClientHandler handler = new HttpClientHandler();
            handler.Proxy = new WebProxy("https://127.0.0.1");
            HttpClient client = new HttpClient(handler);

I receive an Only the 'http' scheme is allowed for proxies. NotSupportedException.

I found some indication at https://github.com/dotnet/corefx/issues/20435 that this is not supported at all, so I am reopening this as a feature request.

Thanks

[richlander]

This looks like a good feature for .NET 5. I haven't heard enough requests about this scenario to warrant adding it earlier.

[scalablecory]

Can you give some use cases for HTTPS proxies?

[richlander]

Wouldn't it be the same as an HTTP proxy but with TLS? An HTTPS -> HTTP transition through an HTTP proxy would seem to undermine the value proposition of HTTPS. I cannot imagine that it is any more complicated than that.

[scalablecory]

An HTTPS -> HTTP transition through an HTTP proxy would seem to undermine the value proposition of HTTPS.

HTTPS requests have end-to-end encryption, regardless of if they go through a proxy, or if that proxy has encryption -- it is protected from the proxy as well.

A HTTPS proxy does not really add security, it merely shifts it: it makes it so that instead of your ISP being able to track what IPs you connect to, the proxy can.

[scalablecory]

Maybe @GrabYourPitchforks has some thoughts.

[matteo-prosperi]

We have been using https://github.com/fancy-owl/docker-mtls-https-proxy to test it.

CURL supports it by doing

curl --proxy-cert ./client.pem --proxy-cacert ./ca.crt --proxy https://some.uri.com https://i.want.to.see.this.com

We have customers who are setting up HTTPS proxies in their networks and requiring clients to authenticate against the proxy with client certificates.

[karelz]

Related to #17740 (Socks4/5 proxies)

[richlander]

@scalablecory -- that doesn't make sense to me. If you used an HTTP proxy, I assume any traffic after that proxy was "in the clear". That was my point, and avoiding that was my intended use case for an HTTPS proxy. If the proxy was one you trusted, then this model would seem to be fine. Right?

My mental model is this:

• There are scenarios for proxies.
• There are scenarios for HTTPS.
• This request targets the points where those scenarios overlap.

Again, is this any more complicated that that? I assume if you don't have an HTTPS proxy, then you violate the HTTPS-oriented requirements? I understand your point about proxy now being a trust point. It seems similar to using a third-party VPN. If the proxy isn't third party, then maybe no trust issue. Fair?

This seems to align with what @matteo-prosperi is saying.

[davidsh]

A HTTPS proxy does not really add security, it merely shifts it: it makes it so that instead of your ISP being able to track what IPs you connect to, the proxy can.

Besides consumer related scenarios (i.e. ISP's), we have heard from enterprise customers over the years wanting "TLS everywhere" which means that communicating with a proxy on the LAN should also use HTTPS. Currently even with communicating with a destination server over HTTPS, the initial 'CONNECT' verb to the proxy is unencrypted (HTTP right now). And that means any headers that get sent along (include 'Proxy-Authorization' headers) would be visible.

So, the scenario itself is reasonable. We would just have to prioritize this work and get a design/cost for it.

[richlander]

There has been talk about DNS with TLS for the same reason.

[davidsh]

There has been talk about DNS with TLS for the same reason.

There are significant disagreements about the usefulness of some of these related protocols (like DNS over HTTPS):
https://www.zdnet.com/article/dns-over-https-causes-more-problems-than-it-solves-experts-say/

But pure DNS over TLS (DoT) or using IPSEC would be the best for DNS solutions.

[matteo-prosperi]

I suppose in this case using client certificates also allows limiting which users or maybe even applications can interact with servers outside of a certain network boundary.

[scalablecory]

So lets draw out what a proxy connection looks like.

Client -> Client's ISP -> Proxy -> Proxy's ISP -> a series of tubes -> Server's ISP -> Server

For an HTTP proxy, the Client's ISP and the Proxy can see that you are talking to Server. Everyone else sees Proxy talking to Server.

For an HTTPS proxy, the Client's ISP can only see that you are talking to Proxy. This is the sole benefit of an HTTPS proxy -- if you can trust the proxy itself, or at least distrust your ISP, then you gain some small bit of anonymity from your ISP. Everything else remains the same.

Now, regardless of if your proxy is over TLS or not, an HTTPS request is encrypted all the way to Server: even if the Client's ISP can see that you're talking to Server, they still don't know what you're saying. You don't lose any of HTTPS benefits by going over an unencrypted proxy.

[scalablecory]

Note: not saying this is necessarily a bad idea, just that the real benefit is deceptively slim. I'd welcome some examples of when hiding from your ISP will be important under HttpClient usage scenarios.

If there are businesses that want to take a "feel good" TLS-everything approach, I'd hope they would use IPsec rather than relying on HTTP-level encryption inside their network, but I trust @davidsh knows the wants of that space better than me. That is a valid viewpoint to consider.

[richlander]

This is probably all true, but imagine that all the calls and all the end-points are running within a cloud environment. That's what this comment is making reference to: https://github.com/dotnet/corefx/issues/41658#issuecomment-539792003. There is no "ISP".