Skip to content

Conversation

@johha
Copy link
Contributor

@johha johha commented Jan 30, 2026

PR #22 introduced slog for logging for S3, GCS and Ali. To ensure a consistent logging format this PR adopts slog for S3.
Additionally a new http logging middleware is added for S3 which prints request logs when needed (e.g. with flag -log-level debug). The existing middlewares.go has been renamed to accept_encoding_fix.go and moved to the s3middleware folder for clearer structure.

S3 Debug Log Format

S3 Debug Log Format

Non-Existing Blob

❯ ./storage-cli -c tmp/minio_aws.json -log-level debug -s s3 exists dummy.txt
{"time":"2026-01-30T13:23:08.638516+01:00","level":"DEBUG","msg":"s3 http request","method":"HEAD","url":"http://localhost:9000/storage-cli/dummy.txt","host":"","request_content_length":0,"duration_ms":2,"status_code":404,"response_content_length":0,"request_id":"188F81B3FDDA1418","extended_request_id":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"}
{"time":"2026-01-30T13:23:08.638982+01:00","level":"INFO","msg":"Blob does not exist in bucket","bucket":"storage-cli","blob":"dummy.txt"}
{"time":"2026-01-30T13:23:08.638988+01:00","level":"ERROR","msg":"performing operation","command":"exists","error":"object does not exist"}

Put Blob

❯ ./storage-cli -c tmp/minio_aws.json -log-level debug -s s3 put tmp/dummy.txt dummy.txt
{"time":"2026-01-30T13:23:13.479642+01:00","level":"DEBUG","msg":"s3 http request","method":"PUT","url":"http://localhost:9000/storage-cli/dummy.txt?x-id=PutObject","host":"","request_content_length":6,"duration_ms":5,"status_code":200,"response_content_length":0,"request_id":"188F81B51E3F2260","extended_request_id":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"}
{"time":"2026-01-30T13:23:13.480116+01:00","level":"INFO","msg":"Successfully uploaded file","location":"http://localhost:9000/storage-cli/dummy.txt"}

Existing Blob

❯ ./storage-cli -c tmp/minio_aws.json -log-level debug -s s3 exists dummy.txt
{"time":"2026-01-30T13:23:17.724548+01:00","level":"DEBUG","msg":"s3 http request","method":"HEAD","url":"http://localhost:9000/storage-cli/dummy.txt","host":"","request_content_length":0,"duration_ms":11,"status_code":200,"response_content_length":6,"request_id":"188F81B61AE43E38","extended_request_id":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"}
{"time":"2026-01-30T13:23:17.725054+01:00","level":"INFO","msg":"Blob exists in bucket","bucket":"storage-cli","blob":"dummy.txt"}

Get Blob

❯ ./storage-cli -c tmp/minio_aws.json -log-level debug -s s3 get dummy.txt dummy.txt
{"time":"2026-01-30T13:23:37.724995+01:00","level":"DEBUG","msg":"s3 http request","method":"GET","url":"http://localhost:9000/storage-cli/dummy.txt?x-id=GetObject","host":"","request_content_length":0,"duration_ms":2,"status_code":206,"response_content_length":6,"request_id":"188F81BAC3858188","extended_request_id":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"}

Delete Blob

./storage-cli -c tmp/minio_aws.json -log-level debug -s s3 delete dummy.txt
{"time":"2026-01-30T13:50:05.375201+01:00","level":"DEBUG","msg":"s3 http request","method":"DELETE","url":"http://localhost:9000/storage-cli/dummy.txt?x-id=DeleteObject","host":"","request_content_length":0,"duration_ms":8,"response_content_length":0,"request_id":"188F832C6A9FE630","extended_request_id":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","status_code":204}

@johha johha marked this pull request as ready for review January 30, 2026 13:02
@johha
Copy link
Contributor Author

johha commented Jan 30, 2026

When the log level is not set to debug and an error occurs we still get meaningful error messages from AWS:

{"time":"2026-01-30T14:27:53.504115+01:00","level":"ERROR","msg":"performing operation","command":"exists","error":"failed to check exist: operation error S3: HeadObject, https response error StatusCode: 403, RequestID: 188F853C81E454B8, HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, api error Forbidden: Forbidden"}

@stephanme
Copy link
Member

Non-existing blob:

./storage-cli -c tmp/minio_aws.json -log-level debug -s s3 exists dummy.txt
{"time":"2026-01-30T13:23:08.638516+01:00","level":"DEBUG","msg":"s3 http request","method":"HEAD","url":"http://localhost:9000/storage-cli/dummy.txt","host":"","request_content_length":0,"duration_ms":2,"status_code":404,"response_content_length":0,"request_id":"188F81B3FDDA1418","extended_request_id":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"}
{"time":"2026-01-30T13:23:08.638982+01:00","level":"INFO","msg":"Blob does not exist in bucket","bucket":"storage-cli","blob":"dummy.txt"}
{"time":"2026-01-30T13:23:08.638988+01:00","level":"ERROR","msg":"performing operation","command":"exists","error":"object does not exist"}

The last log-line should be on INFO and not on ERROR level. The "exists" operation returned the desired result and there was no problem with executing the request. Only the client can decide if a non-existing blob is an error or not.

For CAPI, ERROR logs for non-existing blobs would lead to log pollution for resource matching.

Copy link
Contributor

@serdarozerr serdarozerr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, just a minor suggestion added

This log message was redundant was clients already log if a blob does
not exist, e.g. '{"time":"2026-01-30T17:17:05.405448+01:00","level":"INFO","msg":"Blob does not exist in bucket","bucket":"storage-cli","blob":"bommel3"}'
@github-project-automation github-project-automation bot moved this from Inbox to Pending Merge | Prioritized in Foundational Infrastructure Working Group Feb 2, 2026
@johha johha merged commit 301aa4d into main Feb 2, 2026
7 checks passed
@johha johha deleted the s3-logging branch February 2, 2026 09:16
@github-project-automation github-project-automation bot moved this from Pending Merge | Prioritized to Done in Foundational Infrastructure Working Group Feb 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

3 participants