[Feature] Enhance the ability to capture headers.

[EthianWong]

Desc

I used http-cookie-agent in my project to address cookie-related issues in Node.js. Unfortunately, it currently seems unable to correctly capture cookies written by this module. (I'm not sure if it's a timing issue or a problem with other logic)
...

To Resolve?

Depending on the input-output ratio, I believe it’s worth resolving.
...

Resolutions(Optional)

After a brief debugging session, I believe this seems to be a order issue. The http-cookie-agent modifies cookies after node-network-devtools, so node-network-devtools cannot detect those cookie changes.

I think we should ensure that, just like Chrome's DevTools, the correct results are displayed regardless of how or when the user interacts with it.

I believe it might be possible to rewrite request-related methods to intercept cookie writes, thereby fulfilling this requirement. This approach not only ensures compatibility with http-cookie-agent but also enhances cookie capture capabilities in other scenarios.
...

[GrinZero]

It is a good idea to intercept setheaders to monitor subsequent header updates, especially when there are cookie updates and trigger CDP.
But I have two ideas/questions:

1. The range of field modifications can be expanded, not just to modify cookie fields, perhaps we should record any header.
2. Apart from libraries implemented based on http/https, would fetch these libraries have the same problem?

[EthianWong]

1. I agree with this perspective. However, at the time, I only encountered the cookie issue, so I focused on cookies. If this perspective is confirmed, I can also adjust the code accordingly.

2. I'm not entirely sure if methods likefetch will have issues... Theoretically, it depends on how we intercept the requests. I haven't looked into the implementation of fetch before, but after briefly reading through the code just now, it seems the same issue would likely exist. We simply overridefetch, so if users set it directly in this manner, it should currently be supported.

const response = await fetch("https://example.org/post", {
  headers: {
    "cookie": "xxxxx",
  },
  // ...
});

However, if something like http-cookie-agent is used to modify cookies through a Node.js agent, it would likely not be detectable. As for how to handle such scenarios, I haven't looked into it yet.

import { CookieJar } from 'tough-cookie';
import { CookieAgent } from 'http-cookie-agent/undici';

const jar = new CookieJar();
const agent = new CookieAgent({ cookies: { jar } });

await fetch('https://example.com', { dispatcher: agent });

The core issue might lie in the fact that requests can be altered through the configuration of an agent. Currently, many request libraries support setting anagent, though most are used for implementing network proxies, while cases like cookies using an agent are less common.

However, let's consider this: if a proxy library modifies requests, it might append some special fields, such as X-Forwarded-For: 203.0.113.195.

Ideally, we should also be able to capture these changes.

[GrinZero]

I think it is indeed necessary to capture these changes:

1. Some third-party libraries may encapsulate the original library and perform operations such as adding custom headers. We should capture these changes because one of our core goals is to track real requests.
2. Regarding libraries such as fetch, I have not conducted any research, but if these libraries can also be modified after being called, I think we need to intercept them as well.

I am looking forward to your next code submission~