Skip to content

HDDS-5378 Add APIs to retrieve Namespace Summary from Recon #2417

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
smengcl merged 25 commits into from
Jul 28, 2021
Merged

HDDS-5378 Add APIs to retrieve Namespace Summary from Recon #2417

smengcl merged 25 commits into from
Jul 28, 2021

Conversation

@yuangu002
Copy link
Contributor

@yuangu002 yuangu002 commented Jul 14, 2021

What changes were proposed in this pull request?

Add new API(s) (likely REST APIs) exposed by Recon to be used by CLI and Web UI later.

Other changes from HDDS-5386:

  1. Added a field dirName under NSSummary to store a directory's name, which will be used in NSSummaryEndpoint where we query the next-level path name for the DU endpoint.
  2. Optimized the childDir from List to Set as subdirectories' order doesn't matter for us.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5378

How was this patch tested?

Unit test: org.apache.hadoop.ozone.recon.api.TestNSSummaryEndpoint

@yuangu002
Copy link
Contributor Author

yuangu002 commented Jul 14, 2021

@smengcl @avijayanhwx @vivekratnavel Could you review it?

smengcl
smengcl reviewed Jul 16, 2021
View reviewed changes
Copy link
Contributor

@smengcl smengcl left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks really good overall!

@yuangu002
Copy link
Contributor Author

yuangu002 commented Jul 23, 2021

Testing on clusters:
Configs: 1 volume, 1 bucket, 4 directories, and 8 keys.
shell cmd

Results:
[Volume Level]
volume basic
volume du
volume quota
volume distribution

[Bucket Level]
bucket du

[Directory Level]
directory basic
directory du

I have more complicated tests but there's just a simple demo.

avijayanhwx
avijayanhwx reviewed Jul 26, 2021
View reviewed changes
Copy link
Contributor

@avijayanhwx avijayanhwx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @yuangu002. A few comments inline.

@smengcl
Copy link
Contributor

smengcl commented Jul 27, 2021

Also I noticed a peculiar thing that, before Recon's initial sync with OM, /namespace endpoints will return HTTP 500. But /containers endpoint won't.

500

smengcl
smengcl requested changes Jul 27, 2021
View reviewed changes
Copy link
Contributor

@smengcl smengcl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job on the patch overall. It's a huge one. It deserves some scrutiny. With the rest of the comments addressed I would +1 on this.

yuangu002 and others added 4 commits July 27, 2021 15:04
@yuangu002 @smengcl
Update hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon…
942529a 
…/api/NSSummaryEndpoint.java

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
@yuangu002 @smengcl
Update hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon…
6fbaaca 
…/api/NSSummaryEndpoint.java

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
@yuangu002 @smengcl
Update hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon…
d9ebb20 
…/api/types/DUResponse.java

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
@yuangu002 @smengcl
Update hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon…
580c729 
…/api/types/DUResponse.java

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
@yuangu002
Copy link
Contributor Author

yuangu002 commented Jul 27, 2021

Also I noticed a peculiar thing that, before Recon's initial sync with OM, /namespace endpoints will return HTTP 500. But /containers endpoint won't.

500

I believe it's a NPE on either Recon's OM or the Table isn't initialized.

@yuangu002
Copy link
Contributor Author

yuangu002 commented Jul 28, 2021 •
edited
Loading

Future works:

  1. Notification on non-FSO-enabled system.
  2. Refactoring on /summary and /count
  3. Add an additional parameter &files=true to DU endpoint, which counts key as a subpath.
  4. Server failure error during initialization.
  5. Add path and size on du's current level.

yuangu002 and others added 3 commits July 28, 2021 08:45
@yuangu002 @smengcl
Update hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon…
ce3f02d 
…/api/types/DUResponse.java

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
@yuangu002
retrigger CI
c6adfe0
@yuangu002
Merge branch 'HDDS-5378-2' of https://github.com/yuangu002/ozone into…
85529b1 
… HDDS-5378-2
@smengcl smengcl merged commit cd8287b into apache:master Jul 28, 2021
@smengcl
Copy link
Contributor

smengcl commented Jul 28, 2021

Merged. Thanks @yuangu002 for the contribution! And thanks @avijayanhwx for reviewing this as well.

errose28 added a commit to errose28/ozone that referenced this pull request Jul 30, 2021
@errose28
Merge branch 'master' into HDDS-5432-no-downgrade-action
c0a585d 
* master: (48 commits)
  HDDS-5514. Skip check for UNHEALTHY containers for datanode finalize. (apache#2469)
  HDDS-5279. OFS mkdir -p does not work when Volume is not pre-created (apache#2412)
  HDDS-5328. Remove delete container command from admin CLI (apache#2456)
  HDDS-5382. Increase default container report interval to 60 mins (apache#2363)
  HDDS-5378 Add APIs to retrieve Namespace Summary from Recon (apache#2417)
  HDDS-5466. Refactor BlockOutputStream. (apache#2442)
  HDDS-5465. Delete redundant code when set、add and remove bucket acl (apache#2439)
  HDDS-5184. Use separate DB profile for Datanodes. (apache#2214)
  HDDS-5494. Reduce retry in Kubernetes test (apache#2461)
  HDDS-5414. Data buffers incorrectly filtered for Ozone Insight (apache#2387)
  HDDS-5450. Avoid refresh pipeline for S3 headObject (apache#2431)
  HDDS-5500. New k3s version breaks kubernetes test (apache#2464)
  HDDS-5489. Install OS-specific flekszible (apache#2462)
  Multi-raft style placement with permutations for offline data generator (apache#2434)
  HDDS-5484. Intermittent failure in TestReplicationManager#testMovePrerequisites (apache#2454)
  HDDS-5443 Create and then recreate a bucket with a randomized name (apache#2436)
  HDDS-5492. Disable failing kubernetes test (apache#2459)
  HDDS-4330. Bootstrap new OM node (apache#1494)
  HDDS-5418. Let Recon send reregisterCommand to Datanodes if DatanodeDetails changed (apache#2392)
  HDDS-5479. s3g bucket list failed when there is non-english key name. (apache#2450)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@avijayanhwx avijayanhwx avijayanhwx left review comments

@smengcl smengcl smengcl approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yuangu002 @smengcl @avijayanhwx