sqldb: add invoice schema and sql queries

[positiveblue]

This is the first PR for #6288

In this PR we define the tables, indexes and queries for our invoice schema.

We aim to support sqlite/postgres out of the box. The sql code in the migration/query files need to work for both engines with the minimum changes and we want to avoid using specific functionalities like postgres SEQUENCEs.

We will use sqlc to generate go code from SQL. The tool uses the config defined in sqlc.yaml. By default we will set the engine to postgres but the generated code also works for sqlite.

The invoice schema can be divided in three parts:

• Bolt11 invoices: All the common tables to keep track of created invoices, their htlcs and their payments.
• AMP invoices: Most of the logic is shared with BOLT11 invoices, but we add a couple of tables to be able to keep track of the different payments for each AMP invoice (identified by the set_id) and the extra data of each htlc.
• Invoice Events: This aims to support a new functionality that is not currently supported by the KV store. We are interested in keep a record of the events that the an invoice went through.

Because the queries are not generated at runtime, we need to write generic ones that allow us to filter/order using different filter parameters that can be set to NULL when we don't want to apply in a specific query.

For filtering, we use the pattern

(
    column = sqlc.narg('param_name') OR
    sqlc.narg('param_name') IS NULL
) AND (
    column = sqlc.narg('param_name_2') OR
    sqlc.narg('param_name_2') IS NULL
)

We also can also use

    CASE
        WHEN sqlc.narg('param_name')=TRUE THEN (custom_logic)
        ELSE TRUE 
    END

Ordering is a bit more complicated, because we cannot add ASC/DESC in a CASE-WHEN-THEN-ELSE so we need to unroll the cases

ORDER BY
    CASE
        WHEN sqlc.narg('ording_param') = FALSE THEN column1  
        ELSE NULL
    END ASC,
    CASE
        WHEN sqlc.narg('ordering_param') = TRUE  THEN column1  
        ELSE NULL
    END DESC

A good example of a real query using this patterns is FilterInvocies.

There are a couple of things that I am considering to add, but I would like to know others opinions.

• Currently, row ids (INTEGER PRIMARY KEY) are mapped to int32. That means that we have a counter valid until 2,147,483,647 which is 1 invoice/second for the next ~68 years. In Go code we use uint64 for them. uint64 is not a supported sql type, but we may want to use the largest integer value (int64) for this.

• Invoice payments are stored in their own table. Some invoice types can have a one invoice -> many peyments so a JOIN
  make sense in this case. However, it could be interesting to add information about the "latest" payment in the invoice table. In that way we would fetch the invoice and its payment information in one trip to the db, whenever it's possible.

• Most of the invoices will have the same feature set, based on their type. Maybe we could add an "features_set_enum" instead of storing them for each invoice. That would also gives us the flexibility of having a different feature set for the same kind of invoice in case we add it only to new invoices.

Here is a diagram of the proposed schema.

NOTE: the structure of this PR has changed so some of the old comments apply now to other PRs. This one is only about the SQL schema.

[ziggie1984]

Hi, impressive work! :)
Did a first pass-through in preparation of the review club next week.

[ziggie1984]

question: Could you explain why we are naming the test files with the prefix test instead of the suffix test?

[positiveblue]

yes, comes from taproot_assets but as you noticed it would be good to change it to match the Go's good practices

This feedback will be added in #7343

[ziggie1984]

question: would it be reasonable to only include this file when building for example with the kvdb_sqlite package, this came to my mind because later in the test files you mark the with specific tags like test_db_postgres or test_db_sqlite so was wondering why in the test code but not here?

[bhandras]

+1

[sputn1ck]

Great work @positiveblue . Added some qs and need to research the SEQUENCES stuff a bit

[sputn1ck]

I'm not sure about the implications of storing with or without the time zone. E.g. assuming I'm running my node in my home in utc+0 and later migrating that to a remote server that is in a different timezone, my gut feeling is that would be a conflict here?

[bhandras]

I think it's ok to use no timezone until we make sure to display all printed timestamps as UTC or alternatively transform to the user's timezone (when printing).

[positiveblue]

@sputn1ck your gut feeling is right here: time/timezone == software engineering headache

As Andras said, the idea is to not use timezone (and everything is sitll consistent) until you want to show it to the user

[sputn1ck]

nit: hodl 😄

[positiveblue]

haha I checked it in our code and we call them hodl invoices

I think it's they real name hold-invoices though...

[bhandras]

Cool PR, great work so far @positiveblue 🚀 🚀

[bhandras]

I think it's ok to use no timezone until we make sure to display all printed timestamps as UTC or alternatively transform to the user's timezone (when printing).

[bhandras]

+1

[bhandras]

Looking really good to me, awesome work @positiveblue 🥇

[lightninglabs-deploy]

@bhandras: review reminder
@Roasbeef: review reminder
@sputn1ck: review reminder
@ziggie1984: review reminder

[ziggie1984]

Great PR 🤓, learned a lot how to cleanly setup sql schemas with sqlc.

Just had some nits and questions.

Regarding your remarks:

1. I would prefer int64 for the primary key, especially because sql dbs can easily deployed on other hosts and compacted on the fly so I consider the additonal space not as critical. On the other hand a migration later one is not a big deal as well?

2. So regarding the payment information when fetching an invoice, do I understand it right that you propose just adding the latest successful payment rather then all when using MPP or AMP for example?

3. Very much in favour of making the feature set space requirement more efficient but also keep the possibility to have it separate for each invoice just in case we want to be more flexible?

[ziggie1984]

Q: For Bolt12 invoices there will be a "cltv_delta" included in the blinded path, however there will be no specific final_hop delta as in bolt11 invoices, there might be still a receiver required delta but it will be handled differently I think. So my question is, maybe we should allow a NULL value then?

[ziggie1984]

so for every update there will be a corresponding invoice_event?

[ziggie1984]

wondering why the set_id is stored as bytes but he htlc_id we go for text ? Is text more efficient that BLOB?

[ziggie1984]

in the next commit you filter for pending_only which only hits for 0 and 3, would it make sense then to define the states in the comments here as well?

[positiveblue]

I think we can leave that for the application layer (Go) given that "pending_only"/"pending" is not a state by itself but an invoice is considered pending when its state is "open"/"accepted"

[ziggie1984]

Q: Could you elaborate why we want to show information from the invoice_payments table here? Will those not be equal to the ones in the amp_invoice_payments table?

[positiveblue]

Yes, sql generates a struct where fields in both tables get a standard name, fore example:

type SelectAMPInvoicePaymentsRow struct {
	SetID          []byte
	State          int16
	CreatedAt      time.Time
	SettledIndex   sql.NullInt32
	InvoiceID      int32
	ID             sql.NullInt32
	SettledAt      sql.NullTime
	AmountPaidMsat sql.NullInt64
	InvoiceID_2    sql.NullInt32
}

We could "avoid" the overlap and specify a .* for one of the tables and only the distinct columns in the other table. The problem is that next time that we add a column to the second table, we need to update this query too. Because it is not much data, I think it's better to leave it as is.

[ziggie1984]

Sounds good, now I understand thanks.

[ziggie1984]

Q: We do not use ENUM types here because we would need to redefine them in the go code which makes things more complicated than having everything in one place ?

[bhandras]

LGTM, pending @ziggie1984's comments.
Great to see this coming together, will be super nice to use SQL finally! 🚀 🚀

[sputn1ck]

Awesome work @positiveblue 🥇

[ziggie1984]

Great Job 🎉

[Roasbeef]

Excellent PR! This series beings a marked improvement for the general persistence situation within lnd.

Left some non-blocking comments that can be followed up on later. Some may end up changing the schema, but we can do so in a breaking manner, as people can't yet use the definitions, etc.

LGTM 🌪️

[Roasbeef]

I think we have a bit of room to further decouple things here. Eg: in the future a logical invoice will have both a BOLT 11 and BOLT 12 invoice format, along w/ w/e might arise in the future. So we could spin this out into another table that then references this main invoice table.

Similarly, not every invoice will have a 1:1 payment hash to pre-image relationship. Eg: AMP invoices only have a payment_addr as the hash+preimage are generated by the sender.

[Roasbeef]

This in theory could be derived via a view that sums over the set of payments to the invoice itself. Another factor here is the set of active HTLCs as well.

[Roasbeef]

So then the general invoice feature table was dropped in favor of this/

[Roasbeef]

👍

In the future we can add another event table here for stuff like: cancel, accept, hodl, etc.

[Roasbeef]

Perhaps also nice to have easy access to the amount here as well? Then can just read this out rather than needing to also fetch+sum over the set of invoice HTLCs.

[Roasbeef]

If you look at the way the queries exist as is, I think what you also want here is based on the payment_addr as well. Then given that, you can fetch them all for a payment addr w/o an intermediate look up for the invoice itself.

[Roasbeef]

In the future if we want very strict typing here, we can make a set "enum values" table, then point to that itself (all items unique, essentially pre-populated).

You can also do stuff like:

type CHECK(type in ('THIS', THAT'))

etc