invoices: expose custom tlv records from the payload

[joostjager]

This PR extracts the received custom records from the hop payload and stores them in the invoice database. Users can retrieve the custom records through the usual invoice queries / subscriptions.

[Roasbeef]

This seems to be really overloading the type beyond its original intended usage. It's only meant to be used in order to determine violations w.r.t known/unknown types. We already return the full set of unparsed types after decoding, so not sure why this is needed.

[Roasbeef]

Also have you benched marked this? As is, it appears we'll end up copying all the values from the main stream twice.

[joostjager]

Yes it is overloading the type, but we can deviate from an original intention.

I haven't benchmarked. If we want to process the raw bytes of unrecognized records, we need to copy anyway. Just in case our intention is to really ignore them, the copy is unnecessary. Afaik, for db tlv that isn't expected. And for the tlv payload it may happen with an unrecognized standard field (the custom ones we probably want to store somewhere). To me it seems premature to optimize for.

I am open to different ways of implementing this. What do you prefer?

[joostjager]

Where do you see the copy twice btw? Afaik we only read bytes from the stream and store them in a new byte slice that is a value of the TypeSet map.

[joostjager]

Renamed to TypeMap and preallocated the buffer.

[Roasbeef]

Feels like there's another way to do this with less copying.

[joostjager]

In line with comment above, I don't think it matters much. If you want the tlv stream to return a map that we can use directly, we need to add more controls to the stream to specify what exactly we expect.

[joostjager]

suggestion from @cfromknecht: delete items from the map

[joostjager]

It seems it isn't possible to cast map[Type][]byte to map[uint64][]byte though

[joostjager]

Talked about this offline with @Roasbeef. We do want to stop tlv.Type from being used anywhere. Copying the map (but not the values) looks to be ok.

[joostjager]

Offline comment from @cfromknecht: not sure if it is a good idea to mix the custom records with the db types here.

[joostjager]

Discussed this further. It may be better to create a single tlv record for which the value is a tlv stream with just the custom records. As a simple insurance against accidental modification of standard fields.

[joostjager]

Discussed offline. There seems to be more support for a flat structure, so leaving it as is.

[cfromknecht]

I am still somewhat concerned about mingling minimally validated user data with known records. I think it is possible to do it safely, but i think it forces us to be stricter in how we handle migrations.

Consider the scenario where we add a field to the invoice body but do not bump the db version. If a user downgrades, this will now appear in the custom record set even tho its type is beneath the cutoff. I think it's possible that data gets written back, but it's difficult imo to ascertain whether this can lead to unforeseen bugs.

We could prevent this by filter the custom records when reading as we do in the hop, then the data is actually dropped when writing back. This would maintain the current behavior in that scenario.

[joostjager]

Question open as to whether we want to bump the db version, to prevent users running master from downgrading and loosing their custom records

[joostjager]

Discussed this in the context of another PR. We don't give this guarantee to users running master generally.

[cfromknecht]

I am still somewhat concerned about mingling minimally validated user data with known records. I think it is possible to do it safely, but i think it forces us to be stricter in how we handle migrations.

Consider the scenario where we add a field to the invoice body but do not bump the db version. If a user downgrades, this will now appear in the custom record set even tho its type is beneath the cutoff. I think it's possible that data gets written back, but it's difficult imo to ascertain whether this can lead to unforeseen bugs.

We could prevent this by filter the custom records when reading as we do in the hop, then the data is actually dropped when writing back. This would maintain the current behavior in that scenario.

[joostjager]

We could prevent this by filter the custom records when reading as we do in the hop, then the data is actually dropped when writing back. This would maintain the current behavior in that scenario.

Yes that is a nice safe guard that doesn't require a different db struct. It should be in place now

[Roasbeef]

LGTM 🧹

Left one comment which is mainly concerned with consistency of the API w.r.t TypeMap.

[Roasbeef]

Why don't we also want to provide the caller w/ the raw bytes of the type just as we do for unknown+odd types?

[joostjager]

In that case we would copy the data twice for the known types. And need another data structure to signal which ones were decoded. We can also rethink that when the need arises to access the raw bytes of known types, this is all in memory.

[cfromknecht]

LGTM 🤘

[cfromknecht]

needs rebase

[joostjager]

rebased