Which context to run in?

[zner0L]

I always imagined the addon running as a simple addon in the toolbar, but I just discovered devtools.network.getHAR(), which can only be run on a page in the devtools. Supposedly this is also available in Chrome. We could just run our extensions in the devtools and take advantage of this, or build our own HAR, using webRequest.onBeforeRequest. There can be a case made for both, so I would like your input, @baltpeter.

[baltpeter]

I always imagined the addon running as a simple addon

If by that you mean one with a popup (like UBO for example), I don't think that would be a good idea for complex multi-step processes like ours. But you probably don't, I also imagined a button that opens the UI in a tab.

Regarding the actual question: I mean, not having to assemble the HAR ourselves would of course be a big advantage, but I fear that the limitations of running in the context of the devtools might be too big. (I haven't checked this but) I imagine that it would then only record while the devtools (tab?) are/is open, right?

I have always imagined the addon collecting all traffic in the background and then being able to give you statistics based on all your browsing (with the option to turn this off, of course).

But in light of our (still pending) discussion as to how much we want the platform and addon be a tool for information in addition to activism, maybe that's not what we want after all? And of course, always collecting all data in the background might not even be a good fit for complaints, since you probably won't even be able to say for sure whether you consented for example.

So yeah, I'm really unsure. And I don't think we can really answer this before we had the "big discussion" on what we want the platform and addon to actually do.

[zner0L]

You are right, devtools.network.getHAR() only exports a HAR which contains entries for after the devtools have been opened. But from what I imagined, we would need to reload the page anyway? How else would you attribute the requests to just one page and not many different ones with different controllers?

[baltpeter]

browser.webRequest tells you the initiator of a request. Here's a snippet from a quick and dirty "cookie collector" addon I built:

import { parse as pslParse } from 'tldts';
import { setify, desetify } from './util';

browser.browserAction.onClicked.addListener(() => browser.tabs.create({ url: '/export.html' }));

// Install listener that records contacted origins per domain.
// One thing to consider: In the current implementation, hosts contacted from within a frame are attributed to the frame
// origin, not to the outer page origin. I'm not sure whether that is the correct behavior—maybe they should be
// attributed to each origin in the chain instead?
(async () => {
    const contactedOriginsPerDomain = setify(
        (await browser.storage.local.get('contactedOriginsPerDomain'))['contactedOriginsPerDomain'] || {}
    );

    browser.webRequest.onBeforeRequest.addListener(
        async (details) => {
            // Requests without a `documentUrl` are not initiated by a site (e.g. if caused by navigation). We do
            // however still want to record the relevant cookies etc., they are their own initiator.
            const initiatorUrl = details.documentUrl || details.url;

            const initiatorDomain = pslParse(initiatorUrl).domain;
            if (!initiatorDomain) return { cancel: true };

            const contactedOrigin = new URL(details.url).origin;

            if (!contactedOriginsPerDomain[initiatorDomain]?.has(contactedOrigin)) {
                if (!contactedOriginsPerDomain[initiatorDomain]) contactedOriginsPerDomain[initiatorDomain] = new Set();
                contactedOriginsPerDomain[initiatorDomain]!.add(contactedOrigin);

                await browser.storage.local.set({ contactedOriginsPerDomain: desetify(contactedOriginsPerDomain) });
            }

            return { cancel: true };
        },
        { urls: ['<all_urls>'] }
    );
})();

[zner0L]

Yes, but that would still require us to connect different documentUrls to one controller. Which is not as easy as just using the host part of the URL. Imagine wix.com subdomains for example.

[baltpeter]

But isn't that a problem we need to outsource to the user in either case?

(And as for the wix example: That could probably be solved using the public suffix list, no?)

[zner0L]

Another thing to consider: When we use onBeforeRequest we cannot guarantee the data in the request was actually sent to any server. Other plugins or Firefox itself even could still block the request. onCompleted, however, doesn’t give us access to the request body. So we need to match up requests from both events. getHar just gives us already completed requests.

[baltpeter]

Just for the record: https://github.com/nishantmc/har-recorder/blob/main/background.js uses the chrome.debugger API which lets you send CDP commands. But that isn't an option in Firefox.