feat(gateway): IPFSBackend metrics and HTTP range support #245

lidel · 2023-03-31T02:19:32Z

This PR is part of ipfs-inactive/bifrost-gateway#61

Moves metric-related code to gateway/metrics.go
Adds opentelemetry span for every IPFSBackend API call
Adds success/failure histogram for each API call ipfs_gw_backend_api_call_duration_seconds

TODO

Get and GetRange are now the same, as suggested in feat: switch gateway GetRange API to match Get #240 (review)
fix tests: check why Kubo sharness fails
- Kubo sharness / CI in feat(gw): new metrics and HTTP range support kubo#9786 must be green

- Adds spans for every IPFSBackend API call - Adds success/failure histogram for each API call ipfs_gw_backend_api_call_duration_seconds

codecov · 2023-03-31T02:24:21Z

Codecov Report

Merging #245 (de9daf5) into main (92407c6) will increase coverage by 0.23%.
The diff coverage is 64.38%.

@@            Coverage Diff             @@
##             main     #245      +/-   ##
==========================================
+ Coverage   47.88%   48.12%   +0.23%     
==========================================
  Files         269      271       +2     
  Lines       32976    33159     +183     
==========================================
+ Hits        15792    15957     +165     
- Misses      15513    15534      +21     
+ Partials     1671     1668       -3

Impacted Files	Coverage Δ
gateway/gateway.go	`88.52% <ø> (ø)`
gateway/metrics.go	`62.50% <62.50%> (ø)`
gateway/handler_defaults.go	`36.66% <70.96%> (+5.15%)`	⬆️
blockservice/blockservice.go	`73.56% <100.00%> (ø)`
gateway/blocks_gateway.go	`46.64% <100.00%> (+2.09%)`	⬆️
gateway/handler.go	`56.04% <100.00%> (-6.99%)`	⬇️
gateway/handler_unixfs_dir.go	`62.89% <100.00%> (ø)`

... and 11 files with indirect coverage changes

- IPFSBackend metrics from ipfs/boxo#245 - GraphGateway metrics from #61 - Version for tracking rollouts

Also fixes serving range requests for /ipfs/bafydir when we know we'll get the index.html or _redirect.

Exposing metrics from ipfs/boxo#245

#240 (review)

Exposing range fixes and new metrics from ipfs/boxo#245

blockstore can be used in contexts where there is no bitswap, and this log message will me EXTREMELY confusing, wasting people's time on debugging invalid side of system

Exposing range fixes and new metrics from ipfs/boxo#245

lidel · 2023-04-01T00:44:08Z

gateway/metrics.go

+	md, rc, errCh, err := b.api.GetCAR(ctx, path)
+
+	// TODO: handle errCh
+	b.updateApiCallMetric(name, err, begin)
+	return md, rc, errCh, err


💭 (late friday wrap-up) @aschmahmann updateApiCallMetric will increase counter for success or failure based on err. Are we missing anything critical if we only check err here and ignore errCh?
(This should not block this PR, we can fix metric later)

lidel

@aschmahmann @hacdias I think this is ready for review.

TLDR: adds metrics, fixes range requests, merges Get and GetRange into one func.

Sharness in Kubo PR is green: ipfs/kubo#9786

lidel · 2023-04-01T00:47:48Z

blockservice/blockservice.go

 		// TODO be careful checking ErrNotFound. If the underlying
 		// implementation changes, this will break.
-		logger.Debug("Blockservice: Searching bitswap")
+		logger.Debug("BlockService: Searching")


ℹ️ there is no bitswap in bifrost-gateway, yet we got super confusing "Searching bitswap" message here ;)

lidel · 2023-04-01T00:49:51Z

gateway/metrics.go

ℹ️ moved metrics code here

lidel · 2023-04-01T01:02:45Z

gateway/gateway.go

+	//   - A range request for a directory currently holds no semantic meaning.
+	//
+	// [HTTP Byte Ranges]: https://httpwg.org/specs/rfc9110.html#rfc.section.14.1.2
+	Get(context.Context, ImmutablePath, ...ByteRange) (ContentPathMetadata, *GetResponse, error)


ℹ️ removed GetRange, we now have Get with optional range requests

hacdias

Thanks for this. I tested it locally and it indeed solves the range requests. It LGTM. Nice call to move the metrics stuff to a separate file. It was quite a clutter!

removed depecated and unused TTFB histograms. after #176 and #245 we now have api_call_duration_seconds histograms for higher resolution info per backend operation

feat(gateway): IPFSBackend tracing and metrics

e49cc43

- Adds spans for every IPFSBackend API call - Adds success/failure histogram for each API call ipfs_gw_backend_api_call_duration_seconds

lidel requested review from aschmahmann and hacdias March 31, 2023 02:19

lidel added a commit to ipfs-inactive/bifrost-gateway that referenced this pull request Mar 31, 2023

feat: IPFSBackend and GraphGateway metrics

69b5a49

- IPFSBackend metrics from ipfs/boxo#245 - GraphGateway metrics from #61 - Version for tracking rollouts

This was referenced Mar 31, 2023

feat: opt-in GRAPH_BACKEND ipfs-inactive/bifrost-gateway#61

Merged

Support Server-Timing header ipfs-inactive/bifrost-gateway#69

Open

guseggert force-pushed the main branch from cb788ab to 5dee8e4 Compare March 31, 2023 14:34

fix: switch gateway GetRange API to match Get

b775259

Also fixes serving range requests for /ipfs/bafydir when we know we'll get the index.html or _redirect.

lidel added a commit to ipfs/kubo that referenced this pull request Mar 31, 2023

feat(gateway): IPFSBackend tracing and metrics

b0684b5

Exposing metrics from ipfs/boxo#245

lidel mentioned this pull request Mar 31, 2023

feat(gw): new metrics and HTTP range support ipfs/kubo#9786

Merged

3 tasks

lidel added a commit to ipfs/kubo that referenced this pull request Mar 31, 2023

feat(gateway): IPFSBackend tracing and metrics

032ff84

Exposing metrics from ipfs/boxo#245

lidel changed the title ~~feat(gateway): IPFSBackend tracing and metrics~~ feat(gateway): IPFSBackend metrics and HTTP range support Mar 31, 2023

refactor(gw): merge IPFSBackend.Get and GetRange

ecf32dc

#240 (review)

lidel mentioned this pull request Mar 31, 2023

feat: switch gateway GetRange API to match Get #240

Closed

lidel added a commit to ipfs/kubo that referenced this pull request Mar 31, 2023

feat(gw): new metrics and HTTP range support

cef961c

Exposing range fixes and new metrics from ipfs/boxo#245

lidel added a commit to ipfs/kubo that referenced this pull request Mar 31, 2023

feat(gw): new metrics and HTTP range support

60558fd

Exposing range fixes and new metrics from ipfs/boxo#245

chore: remove bitswap from debug log

de9daf5

blockstore can be used in contexts where there is no bitswap, and this log message will me EXTREMELY confusing, wasting people's time on debugging invalid side of system

lidel added a commit to ipfs/kubo that referenced this pull request Apr 1, 2023

feat(gw): new metrics and HTTP range support

f1dd311

Exposing range fixes and new metrics from ipfs/boxo#245

lidel commented Apr 1, 2023

View reviewed changes

lidel marked this pull request as ready for review April 1, 2023 01:04

lidel requested a review from a team as a code owner April 1, 2023 01:04

hacdias approved these changes Apr 3, 2023

View reviewed changes

hacdias merged commit 5a2c987 into main Apr 3, 2023

hacdias deleted the feat/backend-api-metrics branch April 3, 2023 07:46

lidel added a commit that referenced this pull request Apr 5, 2023

fix(gw): widen duration histograms and cleanup

b8d766f

removed depecated and unused TTFB histograms. after #176 and #245 we now have api_call_duration_seconds histograms for higher resolution info per backend operation

lidel mentioned this pull request Apr 5, 2023

fix(gw): widen duration histograms and cleanup #265

Merged

lidel added a commit that referenced this pull request Apr 5, 2023

fix(gw): widen duration histograms and cleanup

6e1d7a4

removed depecated and unused TTFB histograms. after #176 and #245 we now have api_call_duration_seconds histograms for higher resolution info per backend operation

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(gateway): IPFSBackend metrics and HTTP range support #245

feat(gateway): IPFSBackend metrics and HTTP range support #245

Uh oh!

lidel commented Mar 31, 2023 •

edited

Loading

Uh oh!

codecov bot commented Mar 31, 2023 •

edited

Loading

Uh oh!

lidel Apr 1, 2023 •

edited

Loading

Uh oh!

lidel left a comment

Uh oh!

lidel Apr 1, 2023

Uh oh!

lidel Apr 1, 2023

Uh oh!

lidel Apr 1, 2023 •

edited

Loading

Uh oh!

hacdias left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat(gateway): IPFSBackend metrics and HTTP range support #245

feat(gateway): IPFSBackend metrics and HTTP range support #245

Uh oh!

Conversation

lidel commented Mar 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

codecov bot commented Mar 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

lidel Apr 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidel left a comment

Choose a reason for hiding this comment

Uh oh!

lidel Apr 1, 2023

Choose a reason for hiding this comment

Uh oh!

lidel Apr 1, 2023

Choose a reason for hiding this comment

Uh oh!

lidel Apr 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hacdias left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lidel commented Mar 31, 2023 •

edited

Loading

codecov bot commented Mar 31, 2023 •

edited

Loading

lidel Apr 1, 2023 •

edited

Loading

lidel Apr 1, 2023 •

edited

Loading