Generate static JSON of example values

[baltpeter]

We ultimately want to have a public database of tracking requests and dynamically list examples of observed values for each data path.

Until we get there, I want to generate a static JSON of such examples values based on the data from my master's thesis.

[baltpeter]

Here we go: adapter-examples.txt (it's a JSON file, but GitHub doesn't support that, for some reason)

Code to generate:

import { db, pg } from './db';
import type { AnnotatedResult, Request } from 'trackhar';
import { processRequest } from 'trackhar';
import { writeFile } from 'fs/promises';

type DbRequest = {
    appId: string;
    platform: 'android' | 'ios';
    appVersion: string;

    content?: Buffer;
} & Omit<Request, 'content'>;

(async () => {
    const requests = await db
        .manyOrNone<DbRequest>(
            `select requests.id as "requestId", apps.name as "appId", apps.platform as platform, apps.version as "appVersion", requests.start_time as "startTime", requests.method as method, requests.host as host, requests.path as path, content_raw as content, requests.port as port, requests.scheme as scheme, requests.http_version as "httpVersion", regexp_replace(concat(requests.scheme, '://', requests.host, requests.path), '\\?.+$', '') as "endpointUrl"
        from apps
          join runs r on apps.id = r.app
          join requests on r.id = requests.run;`
        )
        .then((reqs) => reqs.map((r) => ({ ...r, content: r.content?.toString('binary') })));

    const processedRequests = requests
        .map((r) => {
            try {
                return processRequest(r);
            } catch (err) {
                console.error(err, r);
                throw err;
            }
        })
        .filter((r): r is AnnotatedResult => !!r);

    const values = processedRequests.reduce<Record<string, Set<any>>>((acc, cur) => {
        for (const r of cur) {
            const key = `${r.adapter}/${r.property}/${r.context}/${r.path}`;
            if (!acc[key]) acc[key] = new Set();

            acc[key].add(['string', 'number'].includes(typeof r.value) ? r.value : JSON.stringify(r.value));
        }

        return acc;
    }, {});

    const json = JSON.stringify(values, (k, v) => (v instanceof Set ? [...v] : v), 4);
    await writeFile('adapter-examples.json', json);

    pg.end();
})();

process.on('unhandledRejection', (err) => {
    console.error('An unhandled promise rejection occurred:', err);
    pg.end();
    process.exit(1);
});

[baltpeter]

Annoying quirk: Hugo force-lowercases the keys of the page params (https://discourse.gohugo.io/t/config-params-should-be-all-lowercase-or-not/5051, https://discourse.gohugo.io/t/why-is-page-params-key-lowercase/15828), so appId becomes appid, for example, which means that the lookup in our examples JSON fails.

[baltpeter]

Updated code that takes this into account:

import { db, pg } from './db';
import type { AnnotatedResult, Request } from 'trackhar';
import { processRequest } from 'trackhar';
import { writeFile } from 'fs/promises';

type DbRequest = {
    appId: string;
    platform: 'android' | 'ios';
    appVersion: string;

    content?: Buffer;
} & Omit<Request, 'content'>;

(async () => {
    const requests = await db
        .manyOrNone<DbRequest>(
            `select requests.id as "requestId", apps.name as "appId", apps.platform as platform, apps.version as "appVersion", requests.start_time as "startTime", requests.method as method, requests.host as host, requests.path as path, content_raw as content, requests.port as port, requests.scheme as scheme, requests.http_version as "httpVersion", regexp_replace(concat(requests.scheme, '://', requests.host, requests.path), '\\?.+$', '') as "endpointUrl"
        from apps
          join runs r on apps.id = r.app
          join requests on r.id = requests.run;`
        )
        .then((reqs) => reqs.map((r) => ({ ...r, content: r.content?.toString('binary') })));

    const processedRequests = requests
        .map((r) => {
            try {
                return processRequest(r);
            } catch (err) {
                console.error(err, r);
                throw err;
            }
        })
        .filter((r): r is AnnotatedResult => !!r);

    const values = processedRequests.reduce<Record<string, Set<any>>>((acc, cur) => {
        for (const r of cur) {
            // https://github.com/tweaselORG/tracker-wiki/issues/3#issuecomment-1505548347
            const key = `${r.adapter}/${r.property.toLowerCase()}/${r.context}/${r.path}`;
            if (!acc[key]) acc[key] = new Set();

            acc[key].add(['string', 'number'].includes(typeof r.value) ? r.value : JSON.stringify(r.value));
        }

        return acc;
    }, {});

    const json = JSON.stringify(values, (k, v) => (v instanceof Set ? [...v] : v), 4);
    await writeFile('adapterExamples.json', json);

    pg.end();
})();

process.on('unhandledRejection', (err) => {
    console.error('An unhandled promise rejection occurred:', err);
    pg.end();
    process.exit(1);
});

[baltpeter]

Fixed with #1.