Introduce simple metrics for payjoin-directory

[spacebear21]

In yesterday's dev call it was mentioned that we "Observed a 10x increase in logs in the Payjoin directory." since Cake Wallet went live.

We should be able to roughly quantify payjoin activity in a payjoin directory by e.g. counting the number of sessions opens/closes.

[zealsham]

I was wondering about the timeframe for maintaining this count, It might be helpful to quantify payjoin activity within a specific time period for more meaningful analysis.

[spacebear21]

I think the most straightforward solution would be to maintain a log of datapoints with a value and a timestamp - these points can then be grouped by whatever timeframe is preferred. Datadog does this: https://docs.datadoghq.com/metrics/

[zealsham]

Thinking about this , We can do this in two ways , maybe others can offer more insight on which way is better . We could do something like

{
  "timestamp": "2025-06-07T12:00:00Z",
  "event": "session.open",
}

Then with the data-dog agent installed on the server , we can collect the logs .

We could also post the data directly to the data-dog api which does not involve installing an agent on the server . Do you have a different approach in mind ?

[spacebear21]

I don't think we actually need to use datadog, that's probably overkill, but was providing it as an example of how event tracking is done in mainstream software. We can write our own simple counter.

[zealsham]

Do you think metrics-exporter-prometheus crate and Metrics crate is enough for this . We definitely need to view this metrics else we wouldn't have a reason to store them . Prometheus seems to be the best way to collect this metrics , as metrics would be stored in memory and someone running a directory server woudn't need to install any third party software . i really want to work on this .

[spacebear21]

Prometheus sounds good to me and was also suggested in #471. Dan was able to produce a chart of sessions per day using our existing logs. @DanGould can you share how you did it and whether it makes sense to pursue this further or close this issue?

[DanGould]

I wrote a python script to regex for timestamps and handle_v2 debug logs that contain shortIDs, then counting only unique shortIDs. This definitely should be baked in further. I want a dashboard where I can look at granular data by the day and soon enough by the hour

[arminsabouri]

I wrote a python script to regex for timestamps and handle_v2 debug logs that contain shortIDs, then counting only unique shortIDs. This definitely should be baked in further. I want a dashboard where I can look at granular data by the day and soon enough by the hour

I found that Grafana works well with Prometheus for creating dashboards. We would need to host it ourselves.

[thebrandonlucas]

Inspired by the discussion here, I played around with creating a simple nix flake which spins up Prometheus and Grafana exposing metrics for payjoin-directory by simply running nix develop.

The rationale as stated here is to make it as simple as possible for people to run a monitor-able payjoin-directory (and eventually ohttp-relay), while also solving our own problem.

Prometheus requires a /metrics endpoint to be exposed which payjoin-directory doesn't yet have, so just for the proof of concept in the meantime and since it's probably also useful to have, I used node_exporter which gives info about the underlying host machine.

https://github.com/thebrandonlucas/payjoin-directory-flake

Some screenshots of Prometheus and the node_exporter dashboard:

[zealsham]

@thebrandonlucas This is really nice! However, we still need to consider integrating metrics into the payjoin-directory crate. Specifically, we want to track how many sessions are opened and closed over time — something we can visualize on a dashboard like this one.

Since you're also interested in this, I’ll briefly outline my intended implementation so I can get your thoughts or suggestions. Tracking session openings is straightforward — a session opens when a sender posts a PSBT to the directory.

The tricky part is session closing. Ideally, a session closes when the sender broadcasts the transaction containing both their inputs and the receiver’s. However, since we may not have that visibility, we can approximate it by using the expiration time of the corresponding channel ID in Redis. Once the session expires there, we can consider it closed.

This is just the starting point — I believe over time, we’ll want to gather more detailed metrics, both beyond session open/close counts and with more granularity around session lifecycles.

[nothingmuch]

The tricky part is session closing. Ideally, a session closes when the sender broadcasts the transaction containing both their inputs and the receiver’s. However, since we may not have that visibility, we can approximate it by using the expiration time of the corresponding channel ID in Redis. Once the session expires there, we can consider it closed.

IMO start with just the basics like counting the # of requests, focusing on the metrics endpoint and how to expose it. not sure if we want these metrics to be public, i think given the nature of the service we might consider making them available, but the metrics endpoint should be collected through a control plane network not a data plane in order to be useful in case of a DoS attack (even if the metrics endpoint is OK to publish on both)

later we can estimate session counts by using hyperloglog estimation of the # of unique shortids that were written to and the shortids that were read from, and just assume a shortid that was read from corresponds to a half session that is "closed", and estimate number of closed sessions by just dividing that in 2. further subdividing these estimates by response code (202 vs. 200 for GET) would also be useful for estimating session stats.

however, by design we don't have any visibility into session expiry or state, only messages with short IDs and payloads that are indistinguishable from randomness.

This is just the starting point — I believe over time, we’ll want to gather more detailed metrics, both beyond session open/close counts and with more granularity around session lifecycles.

see #775 (at the bottom) for additional metrics ideas, but IMO that can be done in followup work (i.e. first get metrics gathering working, both in the code and on the operational side), then iterate on specifically what metrics we want to keep track of)