add cache key debuginfo lookup#6061
Conversation
github-actions
Bot
added
area/testing
area/storage
area/util
area/buildkitd
area/solver
area/source
labels
This allows opt-in to cache key debug database on daemon startup. If enabled, all cache keys generated by builds are saved into this database together with the plaintexts of the original data so a reverse lookup can be performed later to compare two checksums and find out their original difference. If checksum contains other checksums internally then these are saved as well. For storage constraints, the plaintext of file content is not saved but the metadata portion can be still looked up. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
tonistiigi
force-pushed
the
cache-debuginfo-db
branch
from
99ffe37 to
4c9d94f
Compare
These endpoints show the contents of current boltdb cache database together with debug plaintexts if they have been saved. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
|
Fixed a bug where |
| Usage: "list of directories to scan for CDI spec files", | ||
| }, | ||
| cli.BoolFlag{ | ||
| Name: "save-cache-debug", |
There was a problem hiding this comment.
| Name: "save-cache-debug", | |
| Name: "debug-save-cache", |
If more debug-* flags are to come
There was a problem hiding this comment.
--debug-save-cache would mean something about storing the cache itself. Atm it means "save-cache-debuginfo". If we want --debug prefix, then it should be smth like --debug-save-cache-plaintexts
| TypeStringArray Type = "string-array" | ||
| TypeDigestArray Type = "digest-array" | ||
| TypeFileList Type = "file-list" |
There was a problem hiding this comment.
Can we be consistent? Maybe -list instead of array?
| TypeStringArray Type = "string-array" | |
| TypeDigestArray Type = "digest-array" | |
| TypeFileList Type = "file-list" | |
| TypeStringList Type = "string-list" | |
| TypeDigestList Type = "digest-list" | |
| TypeFileList Type = "file-list" |
There was a problem hiding this comment.
I can change this but logically these are different. File-list is a special format for file headers and checksum per file, not delimiter-separated.
| if d.db == nil { | ||
| return "", nil, errors.WithStack(ErrNotFound) | ||
| } |
There was a problem hiding this comment.
ErrNotFound might be misleading if db not set. Maybe ErrNoDB would be more accurate?
There was a problem hiding this comment.
This is modeled so that DB can be called all the time. In the future, there may be other implementations. So the internals of whether the plaintext is not found because it was not saved for a specific record or for the whole database shouldn't matter. Note that because lookup and store happen at different times, it may be that one of them had DB configured and another one did not.
Is it related to https://github.com/moby/buildkit/pull/6061/files#diff-d44a0b2b1649ed2262e65d607ac40667f9e287a289484f2b26e22c96297f9ebaR43 ? |
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
3 participants
This allows opt-in to cache key debug database on
daemon startup.
If enabled, all cache keys generated by builds are saved into this database together with the plaintexts of the original data so a reverse lookup can be performed later to compare two checksums and find out their original difference. If checksum contains other checksums internally then these are saved as well. For storage constraints, the plaintext of file content is not saved but the metadata portion can be still looked up.