Skip to content

HDDS-13335. Implement and adopt lightweight proto for Recon listKeys API #8699

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sumitagrawl merged 4 commits into from
Jul 11, 2025
Merged

HDDS-13335. Implement and adopt lightweight proto for Recon listKeys API #8699

sumitagrawl merged 4 commits into from
Jul 11, 2025

Conversation

@devmadhuu
Copy link
Contributor

@devmadhuu devmadhuu commented Jun 25, 2025 •
edited
Loading

What changes were proposed in this pull request?

Recon's listKeys API (/api/v1/keys/listKeys) uses full KeyInfo proto where lot of other key proto fields are not desired and needed. Some of them are like

KeyLocationList, FileEncryptionInfoProto, OzoneAclInfo, FileChecksumProto etc

In case of large keys or Multi part upload keys, key locations list can be huge and can create performance impact for some clients calling the listKeys API. So suggested and proposed light weight proto is:

message KeyInfoProtoLight {
required string volumeName = 1;
required string bucketName = 2;
required string keyName = 3;
required uint64 dataSize = 4;
required hadoop.hdds.ReplicationType type = 5;
optional hadoop.hdds.ReplicationFactor factor = 6;
required uint64 creationTime = 8;
required uint64 modificationTime = 9;
optional uint64 objectID = 14;
optional uint64 updateID = 15;
optional uint64 parentID = 16;
optional hadoop.hdds.ECReplicationConfig ecReplicationConfig = 17;
optional bool isFile = 19;
}

Above proto is light weight compared to keyinfo proto and skipped not needed fields.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13335

How was this patch tested?

This patch is tested with existing junit and integration test cases. And tested using docker cluster locally.

API response before the PR change:

{
    "status": "OK",
    "path": "/volume1/fso-bucket",
    "replicatedDataSize": 188743680,
    "unReplicatedDataSize": 62914560,
    "lastKey": "/-9223372036854775552/-9223372036854775040/-9223372036854774527/testfile",
    "keys": [
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "file1",
            "dataSize": 10485760,
            "parentId": -9223372036854774525,
            "file": true,
            "creationTime": 1750853431913,
            "modificationTime": 1750853432736,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774525/file1",
            "path": "volume1/fso-bucket/dir1/dir2/dir3/file1",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        },
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "testfile",
            "dataSize": 10485760,
            "parentId": -9223372036854774525,
            "file": true,
            "creationTime": 1750853429474,
            "modificationTime": 1750853430258,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
            "path": "volume1/fso-bucket/dir1/dir2/dir3/testfile",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        },
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "file1",
            "dataSize": 10485760,
            "parentId": -9223372036854774526,
            "file": true,
            "creationTime": 1750853426888,
            "modificationTime": 1750853427810,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774526/file1",
            "path": "volume1/fso-bucket/dir1/dir2/file1",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        },
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "testfile",
            "dataSize": 10485760,
            "parentId": -9223372036854774526,
            "file": true,
            "creationTime": 1750853424294,
            "modificationTime": 1750853425192,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774526/testfile",
            "path": "volume1/fso-bucket/dir1/dir2/testfile",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        },
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "file1",
            "dataSize": 10485760,
            "parentId": -9223372036854774527,
            "file": true,
            "creationTime": 1750853421599,
            "modificationTime": 1750853422538,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774527/file1",
            "path": "volume1/fso-bucket/dir1/file1",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        },
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "testfile",
            "dataSize": 10485760,
            "parentId": -9223372036854774527,
            "file": true,
            "creationTime": 1750853418580,
            "modificationTime": 1750853419743,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774527/testfile",
            "path": "volume1/fso-bucket/dir1/testfile",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        }
    ]
}

API response after the PR change:

{
    "status": "OK",
    "path": "/volume1/fso-bucket",
    "replicatedDataSize": 188743680,
    "unReplicatedDataSize": 62914560,
    "lastKey": "/-9223372036854775552/-9223372036854775040/-9223372036854774527/testfile",
    "keys": [
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "file1",
            "dataSize": 10485760,
            "parentId": -9223372036854774525,
            "file": true,
            "creationTime": 1750853431913,
            "modificationTime": 1750853432736,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774525/file1",
            "path": "volume1/fso-bucket/dir1/dir2/dir3/file1",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        },
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "testfile",
            "dataSize": 10485760,
            "parentId": -9223372036854774525,
            "file": true,
            "creationTime": 1750853429474,
            "modificationTime": 1750853430258,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774525/testfile",
            "path": "volume1/fso-bucket/dir1/dir2/dir3/testfile",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        },
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "file1",
            "dataSize": 10485760,
            "parentId": -9223372036854774526,
            "file": true,
            "creationTime": 1750853426888,
            "modificationTime": 1750853427810,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774526/file1",
            "path": "volume1/fso-bucket/dir1/dir2/file1",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        },
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "testfile",
            "dataSize": 10485760,
            "parentId": -9223372036854774526,
            "file": true,
            "creationTime": 1750853424294,
            "modificationTime": 1750853425192,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774526/testfile",
            "path": "volume1/fso-bucket/dir1/dir2/testfile",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        },
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "file1",
            "dataSize": 10485760,
            "parentId": -9223372036854774527,
            "file": true,
            "creationTime": 1750853421599,
            "modificationTime": 1750853422538,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774527/file1",
            "path": "volume1/fso-bucket/dir1/file1",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        },
        {
            "volumeName": "volume1",
            "bucketName": "fso-bucket",
            "keyName": "testfile",
            "dataSize": 10485760,
            "parentId": -9223372036854774527,
            "file": true,
            "creationTime": 1750853418580,
            "modificationTime": 1750853419743,
            "key": "/-9223372036854775552/-9223372036854775040/-9223372036854774527/testfile",
            "path": "volume1/fso-bucket/dir1/testfile",
            "replicatedSize": 31457280,
            "replicationInfo": {
                "replicationFactor": "THREE",
                "requiredNodes": 3,
                "minimumNodes": 1,
                "replicationType": "RATIS"
            },
            "size": 10485760,
            "isKey": true
        }
    ]
}

@devmadhuu devmadhuu marked this pull request as ready for review June 25, 2025 10:55
@adoroszlai adoroszlai changed the title HDDS-13335. Ozone Recon - Implement and adopt lightweight proto for Recon listKeys API. HDDS-13335. Implement and adopt lightweight proto for Recon listKeys API Jun 25, 2025
sumitagrawl
sumitagrawl reviewed Jul 10, 2025
View reviewed changes
Copy link
Contributor

@sumitagrawl sumitagrawl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@devmadhuu given few comments

sumitagrawl
sumitagrawl approved these changes Jul 11, 2025
View reviewed changes
Copy link
Contributor

@sumitagrawl sumitagrawl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sumitagrawl sumitagrawl merged commit 8d1a0ad into apache:master Jul 11, 2025
42 checks passed
@tanvipenumudy tanvipenumudy mentioned this pull request Jul 15, 2025
Merged
jojochuang pushed a commit to jojochuang/ozone that referenced this pull request Jul 31, 2025
@devmadhuu @jojochuang
HDDS-13335. Implement and adopt lightweight proto for Recon listKeys …
3c25bcf 
…API (apache#8699)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sumitagrawl sumitagrawl sumitagrawl approved these changes

@sodonnel sodonnel Awaiting requested review from sodonnel

@ArafatKhan2198 ArafatKhan2198 Awaiting requested review from ArafatKhan2198

@szetszwo szetszwo Awaiting requested review from szetszwo

Assignees

No one assigned

Labels

recon

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@devmadhuu @sumitagrawl @adoroszlai