Skip to content

feat(observability): replace metrics-rs with prometheus-client#12787

Merged
bohutang merged 88 commits intodatabendlabs:mainfrom
flaneur2020:add-prometheus-in-compute
Sep 20, 2023
Merged

feat(observability): replace metrics-rs with prometheus-client#12787
bohutang merged 88 commits intodatabendlabs:mainfrom
flaneur2020:add-prometheus-in-compute

Conversation

@flaneur2020
Copy link
Copy Markdown
Contributor

@flaneur2020 flaneur2020 commented Sep 11, 2023 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

This PR migrated all the metrics into prometheus-client, and:

  • change the /metrics endpoint to promtheus-client
  • unit test about the metrics endpoint
  • change the metrics-table to prometheus-client
  • remove metric-rs's recorder
  • support reset metrics (in another PR)

Closes #12688
Closes #12870

This change is Reviewable

@vercel
Copy link
Copy Markdown

vercel Bot commented Sep 11, 2023 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
databend ⬜️ Ignored (Inspect) Visit Preview Sep 20, 2023 7:33am

@flaneur2020 flaneur2020 marked this pull request as ready for review September 14, 2023 09:33
@github-actions github-actions Bot added the pr-refactor this PR changes the code base without new features or bugfix label Sep 14, 2023
@flaneur2020 flaneur2020 changed the title refactor(observability): Add more prometheus metrics refactor(observability): replace metrics-rs with prometheus-client Sep 15, 2023
@flaneur2020 flaneur2020 changed the title refactor(observability): replace metrics-rs with prometheus-client feat(observability): replace metrics-rs with prometheus-client Sep 15, 2023
@flaneur2020 flaneur2020 marked this pull request as ready for review September 15, 2023 13:17
@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Sep 15, 2023
Xuanwo
Xuanwo reviewed Sep 18, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please enable opendal's prometheus layer and disable it's metrics layer to make it work with databend.

@flaneur2020 flaneur2020 mentioned this pull request Sep 19, 2023
Merged
@bohutang bohutang requested a review from Xuanwo September 20, 2023 04:03
@flaneur2020
Copy link
Copy Markdown
Contributor Author

flaneur2020 commented Sep 20, 2023 •
edited
Loading

Please enable opendal's prometheus layer and disable it's metrics layer to make it work with databend.

I tried upgrade opendal but failed due to the api seems changed a lot, I choose to only disable the MetricsLayer in this PR yet, and enable the prometheus client layer later after opendal upgraded.

It seems that currently there's a StorageMetricsAccessor which also covers the metrics on accessing opendal, but the metrics in StorageMetricsAccessor is kind of a subset of the opendal metrics, I suppose that StorageMetricsAccessor could be removed after the opendal upgraded with PrometheusClientLayer enabled.

drmingdrmer
drmingdrmer approved these changes Sep 20, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@drmingdrmer drmingdrmer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 39 of 40 files at r1, 3 of 3 files at r2.
Reviewable status: 40 of 42 files reviewed, all discussions resolved (waiting on @Xuanwo)

@Xuanwo
Copy link
Copy Markdown
Member

Xuanwo commented Sep 20, 2023

Hi, this PR contains breaking changes introduced in OpenDAL v0.40.0, please read our upgrade note carefully to make sure nothing is broken: https://opendal.apache.org/docs/rust/opendal/docs/upgrade/index.html#upgrade-to-v040

@Xuanwo
Copy link
Copy Markdown
Member

Xuanwo commented Sep 20, 2023

Hi, this PR contains breaking changes introduced in OpenDAL v0.40.0, please read our upgrade note carefully to make sure nothing is broken: https://opendal.apache.org/docs/rust/opendal/docs/upgrade/index.html#upgrade-to-v040

Oh, please ignore this comment. Seems opendal not upgraded yet.

@bohutang bohutang merged commit 306b085 into databendlabs:main Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@drmingdrmer drmingdrmer drmingdrmer approved these changes

@Xuanwo Xuanwo Xuanwo approved these changes

Assignees

No one assigned

Labels

pr-feature this PR introduces a new feature to the codebase pr-refactor this PR changes the code base without new features or bugfix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

observability: replace the metrics_table with prometheus-client, and removes metrics-rs observability: switch to prometheus-client in query execution