Skip to content

HDDS-10431. Merge recent commits from master (7c8160fe) to HDDS-7593 #6286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
smengcl merged 111 commits into from
Mar 7, 2024
Merged

HDDS-10431. Merge recent commits from master (7c8160fe) to HDDS-7593 #6286

smengcl merged 111 commits into from
Mar 7, 2024

Conversation

@smengcl
Copy link
Contributor

@smengcl smengcl commented Feb 27, 2024 •
edited
Loading

What changes were proposed in this pull request?

Merge recent commits from master to HDDS-7593. Resolve code conflicts (mostly imports), and manually fix changes that are not solved by auto merge:

  1. Adapt to HDDS-10226 in TestOmMetadataManager and TestOMRecoverLeaseRequest.
  2. Adapt to HDDS-10410 in FilePerBlockStrategy.
  3. Set modification time in test util createOmKeyInfo to fix the test failure of testLeaseSoftLimitForHsyncRecoverFile: cd7c790 This is properly fixed in master branch in HDDS-10482, which is also included in this merge.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10431

How was this patch tested?

  • Green CI.

kerneltime and others added 30 commits February 8, 2024 06:14
@kerneltime
HDDS-10229. Fixes for Grafana dashboards (apache#6120)
039dea9
@sumitagrawl
HDDS-10296. Orphan blocks during overwrite of key. (apache#6180)
3d8365f
@Galsza
HDDS-10226. Refactor OMRequestTestUtils.createOmKeyInfo (apache#6184)
06399b5
@adoroszlai
HDDS-10291. Set simple properties in MiniOzoneCluster via Configurati…
e93e781 
…on (apache#6186)
@jojochuang
HDDS-10328. Support cross realm Kerberos out of box. (apache#6192)
5715aee
@adoroszlai
HDDS-10322. Make VolumeArgs immutable (apache#6193)
601fd41
@ArafatKhan2198
HDDS-8627. Recon - API for Count of deletePending directories (apache…
c1efa33 
…#5037)
@adoroszlai
HDDS-10333. RocksDB logger not closed (apache#6200)
15b62de
@duongkame @szetszwo
HDDS-9843. Ozone client high memory (heap) utilization (apache#6153)
75df6c1 
Co-authored-by: Tsz-Wo Nicholas Sze <szetszwo@apache.org>
@devmadhuu
HDDS-10319. Also consider bucket layout deciding whether to normalize…
d3e2e59 
… path for listKeys (apache#6195)
@Cyrill
HDDS-10262. Encapsulate SnapshotCache inside OmSnapshotManager (apach…
2f2234c 
…e#6135)
@kerneltime
HDDS-10340. Skip ci on dashboard updates (apache#6203)
af25a48 
Co-authored-by: Ritesh H Shukla <ritesh@apache.org>
@tanvipenumudy
HDDS-10318. Add OM client protocol metrics to Ozone - ListKey Metrics…
47ef84c 
… dashboard (apache#6183)

Co-authored-by: tanvipenumudy <tanvi.penumudy@cloudera.com>
@ashishkumar50 @ashishkr200 @jojochuang
HDDS-10256. Retry block allocation when SCM is in safe mode. (apache#…
7c79246 
…6189)

Co-authored-by: ashishk <ashishk@cloudera.com>
(cherry picked from commit 370b9d7)
@hemantk-12
HDDS-10250. Use SnapshotId as key in SnapshotCache (apache#6139)
c35e99f
@adoroszlai
HDDS-10343. Remove dependency on jsr305 (apache#6208)
ffd8221
@raju-balpande
HDDS-10218. Speed up TestSstFilteringService (apache#6196)
cd00691
@adoroszlai
HDDS-10325. Make BucketArgs immutable (apache#6205)
45c853c
@adoroszlai
HDDS-10344. Schedule dependabot for weekend (apache#6209)
c289c67
@dependabot
HDDS-10347. Bump jacoco to 0.8.11 (apache#6214)
68662a7
@dependabot
HDDS-10356. Bump exec-maven-plugin to 3.1.1 (apache#6215)
bacb184
@vtutrinov
HDDS-9680. Use md5 hash of multipart object part's content as ETag (a…
7370676 
…pache#5668)
@Tejaskriya
HDDS-9738. Display startTime, pipeline and container counts for decom…
3c4683e 
…missioning datanode (apache#6185)
@ashishkumar50
HDDS-10359. Recursively deleting volume with OBS bucket shows error d…
44adf80 
…espite success (apache#6217)
@dependabot
HDDS-10368. Bump guice to 6.0.0 (apache#6212)
3876852
@sadanand48
HDDS-10369. Set Times API doesn't work with linked buckets. (apache#6220
78fc781 
)
@smitajoshi12
HDDS-10301. Recon - Fold the pipeline info for a DN on Datanode page. (…
6194d42 
…apache#6198)
@duongkame
HDDS-10288. Checksum to support direct buffers (apache#6162)
2348784
@kerneltime
HDDS-10339. Add S3 API level dashboard (apache#6202)
cce2f96 
Co-authored-by: Ritesh H Shukla <ritesh@apache.org>
@dependabot
HDDS-10391. Bump joda-time to 2.12.7 (apache#6230)
e0f3ae1
@jojochuang
Copy link
Contributor

jojochuang commented Mar 5, 2024

Cherrypicking HDDS-9843. Ozone client high memory (heap) utilization on top of
HDDS-9751. [hsync] Make Putblock performance acceptable - DataNode side (#5662)
failed that test. Cherrypicking HDDS-9843 on the commit before does not fail. HDFS-9751 is what caused the problem.

@smengcl
Copy link
Contributor Author

smengcl commented Mar 5, 2024 •
edited
Loading

During CI debugging, I've noticed that because unit job depends on basic but not on build (plus the fact that build takes longer than basic), a race can unintentionally cause the unit job to use an old cached ozone build to run the actual test. This caused my change to print extra debugging log in my branch to not take effect.

My question: Should we make build a dependency of unit? Or is there a reason not to do so? Posted PR #6333

cc @adoroszlai

@smengcl smengcl mentioned this pull request Mar 5, 2024
Closed
@jojochuang jojochuang mentioned this pull request Mar 5, 2024
Merged
@jojochuang
Copy link
Contributor

jojochuang commented Mar 6, 2024

Apart from HDDS-10471, another problem is TestOMRecoverLeaseRequest#testLeaseSoftLimitForHsyncRecoverFile fails consistently. It looks like a regression in the master branch where modification time of an open key is always zero.

@smengcl
Copy link
Contributor Author

smengcl commented Mar 6, 2024

Just brought 3 new commits including the Direct ByteBuffer crash fix HDDS-10471 into the dev branch:

be5ffc29e2 HDDS-10471. [hsync] MockDatanodeStorage.writeChunk should make a copy of byte string. (#6334)
f244a67037 HDDS-10252. [hsync] Revisit configuration keys for incremental chunk list after HDDS-9884 (#6289)
1cf9e95eef HDDS-10427. Retry read wait based on policy. (#6292)

jojochuang
jojochuang approved these changes Mar 6, 2024
View reviewed changes
Copy link
Contributor

@jojochuang jojochuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

please make sure the git history is not squashed together. Otherwise we can merge it. Thanks!

@smengcl smengcl changed the title HDDS-10431. Merge recent commits from master (54548aa7) to HDDS-7593 HDDS-10431. Merge recent commits from master (7c8160fe) to HDDS-7593 Mar 7, 2024
@jojochuang
Copy link
Contributor

jojochuang commented Mar 7, 2024

LGTM

@smengcl
Copy link
Contributor Author

smengcl commented Mar 7, 2024

Thanks @jojochuang for reviewing this. I will push from local in a minute.

@smengcl smengcl merged commit 3cadb24 into apache:HDDS-7593 Mar 7, 2024
@smengcl
Copy link
Contributor Author

smengcl commented Mar 7, 2024

Done.

Also thanks @jojochuang for all the help on resolving the test failures in this one!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jojochuang jojochuang jojochuang approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.