HDDS-13345. STS port and endpoint skeleton #9343

len548 · 2025-11-21T20:47:05Z

What changes were proposed in this pull request?

Please describe your PR in detail:

Introduced a new HTTP server on 9880 port
STS endpoint in the new server with mock return values
Modification in signature initialization to make it compatible with STS
The current signature base creation method expects headers to include x-amz-content-sha256 as payload hash for s3. For STS request, it needs actual payload hash.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13345

How was this patch tested?

Unit test and modified existing tests

sodonnel · 2025-12-05T16:46:50Z

@fmorg-git Could you give this a review pass and if you are happy I can check it too?

...zone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/AWSSignatureProcessor.java

...ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java

fm-cdera · 2025-12-06T00:06:08Z

...zone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/AWSSignatureProcessor.java

+    byte[] body = readAllBytes(in);
+    String payloadHash = Hex.encode(MessageDigest.getInstance("SHA-256").digest(body));
+    context.setEntityStream(new ByteArrayInputStream(body));
+    return payloadHash;


it doesn't appear the previous getPayloadHash() method was reading all the bytes into memory - curious what is the reason we are doing that now?

This is needed for STS. While S3 requests require x-amz-content-sha256 header as a payload hash while STS requests require hash of the original request body.

perhaps there should be a reasonable limit set on the request body to protect against huge payloads. For example, would there ever be a valid reason the payload is larger than 8192 bytes? If not, then probably readAllBytes could have a check there, and the limit could likely be lower.

I added a check with the limit set to 32KB. This ensures sufficient buffer for the maximum theoretical size of the request body with required parameters. In the request body, Action, Version, RoleArn, RoleSessionName, and DurationSeconds are required, among which RoleArn can be ~25k bytes in the worse-case scenario based on AWS document. Other parameters add relatively minor overhead. So reasonable limit should be 32KB. But if we have other optional parameters support (e.g. Policy or Tag) , we should consider increasing this default size if needed.

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSConfigKeys.java

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSEnabled.java

...s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSEnabledEndpointRequestFilter.java

fm-cdera · 2025-12-06T00:21:46Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSEndpoint.java

+  }
+
+  private String getExpirationTime(int durationSeconds) {
+    Instant expiration = Instant.now().plusSeconds(durationSeconds);


I think this can use a Clock parameter instead of Instant.now() and we may need to have a unit test that uses TestClock to ensure the expiration on the return XML is set correctly. For example, you can have a private static final field with the value Clock.system(ZoneOffset.UTC); and pass the Clock into getExpirationTime method.

actually, maybe this unit test isn't required here - looks like I have a similar one in TestS3AssumeRoleRequest

I agreed with that. This method can be removed once getting AssumeRole response from OM to this endpoint is implemented.

I left TODO comments in the class for which methods can be removed once OM response is implemented in the class.

fm-cdera · 2025-12-06T00:22:57Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSEndpointBase.java

+  @Inject
+  private SignatureInfo signatureInfo;
+
+  protected static final AuditLogger AUDIT =


nit: this can probably be all on one line. Also, is this AUDIT variable being used?

Actually S3STSEndpoint class that inherits this class should use this to log to write success/failure message, similarly to s3 endpoint classes (e.g. BucketEndpoint). But since it does not affect this PR, a new Jira for this can be open after this is merged. For now, I will remove this variable.

I will use this variable in a different PR to log in the audit.

...e/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/signature/TestStringToSignProducer.java

fm-cdera · 2025-12-06T00:24:42Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSEndpoint.java

+   * @param roleArn The ARN of the role to assume (for AssumeRole)
+   * @param roleSessionName Session name for the role (for AssumeRole)
+   * @param durationSeconds Duration of the token validity in seconds
+   * @param version AWS STS API version (should be "2011-06-15")


does the spec require this to be 2011-06-15? if so, it doesn't appear it is being validated

fmorg-git · 2025-12-07T07:50:50Z

hi @sodonnel I did a first pass review - I was logged in as @fm-cdera at the time.

Tejaskriya

Thanks @len548 mostly looks good to me, just a few minor suggestions below.
@sodonnel would you like to take a look?

Tejaskriya · 2025-12-16T08:40:02Z

hadoop-hdds/common/src/main/resources/ozone-default.xml

+    <tag>OZONE, S3GATEWAY</tag>
+    <description>
+      The bind host for the S3 Gateway STS HTTP server.
+      If not set, the value of ozone.s3g.http-bind-host is used.


We could also mention the override aspect (other similar configs have this uniform description):

Suggested change

If not set, the value of ozone.s3g.http-bind-host is used.

If this optional address is set, it overrides only the hostname portion of

ozone.s3g.sts.http-address.

If not set, the value of ozone.s3g.http-bind-host is used.

Agreed and updated.

Tejaskriya · 2025-12-16T08:41:28Z

hadoop-hdds/common/src/main/resources/ozone-default.xml

+    <tag>OZONE, S3GATEWAY</tag>
+    <description>
+      The bind host for the S3 Gateway STS HTTPS server.
+      If not set, the value of ozone.s3g.https-bind-host is used.


Similarly:

Suggested change

If not set, the value of ozone.s3g.https-bind-host is used.

If this optional address is set, it overrides only the hostname portion of

ozone.s3g.sts.https-address.

If not set, the value of ozone.s3g.https-bind-host is used.

Tejaskriya · 2025-12-16T08:59:32Z

...s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSEnabledEndpointRequestFilter.java

+          "<Code>" + errorCode + "</Code>" +
+          "<Message>" + errorMessage + "</Message>" +


Any specific reason for creating the strings for errorCode and message instead of using Response.Status.NOT_IMPLEMENTED itself?

I removed it.

Tejaskriya · 2025-12-16T09:11:02Z

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSEndpoint.java

+  private static final int DEFAULT_DURATION_SECONDS = 3600;
+  private static final int MAX_DURATION_SECONDS = 43200; // 12 hours
+  private static final int MIN_DURATION_SECONDS = 900;   // 15 minutes


Can we consider making these into configurations instead? Or universal constants (like in OzoneConst). The default can be configurable. The min and max can be fixed to AWS specifics if we want to strictly adhere to their design. (IMO having all of them as configs would be better)

cc: @fm-cdera @fmorg-git (not sure which account you are using)
I think in one of your PRs, I had seen similar constants getting created. To avoid having multiple constants mentioned like this can we find a better place to specify this?

I originally thought the backend would only be doing the validations, so I have constants in S3AssumeRoleRequest. Since the endpoint will be doing some validations as well (per earlier discussion above #9343 (comment)), perhaps those constants in S3AssumeRoleRequest can be made public instead of private and be reused in the endpoint.

fapifta · 2026-01-13T22:25:11Z

Based on what we have discussed with @len548 and @fmorg-git I think we are good to go and we can commit this PR. I reviewed the code and I am +1 to commit it.
Two things are remaining here, one is to add audit logging, for which I will create a new JIRA and we already agreed with @len548 that he will take it on and finish that part.
The minimum and maximum values that are addressed by @Tejaskriya we think they should not be configurable, as the AWS spec is pretty concrete and it defines these as non-configurable boundaries with a non-configurable default value, so we are safe to mimic that and we don't see the use of configurability for those values. @Tejaskriya let me know if you disagree but as this is something you mention as a possibility I assume we are good to go this way.
The unification of these constants is to be done by @fmorg-git as part of the integration taks that integrates this endpoint with the OM itself.

fapifta

I have created HDDS-14420 to track the addition of audit logging.
With that I think this PR is ready to merge once CI is green, as this is agains a feature branch, I am going to merge it to help unblock the integration for @fmorg-git.

@Tejaskriya if you think we should add configuration options for the duration constants, let's discuss this in a separate JIRA before we merge back to master, I hope my assumption is correct that you are ok without configurability if @fmorg-git pull these together while following up with the integration.

ivandika3 added the sts Changes for Ozone's S3 Security Token Service label Nov 22, 2025

fm-cdera reviewed Dec 6, 2025

View reviewed changes

len548 force-pushed the HDDS-13345 branch from 9e48878 to bef905b Compare December 10, 2025 17:12

Tejaskriya reviewed Dec 16, 2025

View reviewed changes

len548 added 4 commits January 13, 2026 14:18

add sts http server and endpoint

b41c7b4

fix signature to be compatible with STS

aa4aba7

fix for CI tests

7615ef6

address review comments

99fbeb6

len548 force-pushed the HDDS-13345 branch from bef905b to 4127669 Compare January 13, 2026 13:26

address some comments

75bd04f

len548 force-pushed the HDDS-13345 branch from 4127669 to 75bd04f Compare January 13, 2026 15:05

fapifta approved these changes Jan 13, 2026

View reviewed changes

fapifta merged commit bf1453d into apache:HDDS-13323-sts Jan 19, 2026
123 of 125 checks passed

len548 mentioned this pull request Jan 26, 2026

HDDS-14150. [STS] Connect STS Endpoint to Backend Processing #9654

Closed

		"<Code>" + errorCode + "</Code>" +
		"<Message>" + errorMessage + "</Message>" +

HDDS-13345. STS port and endpoint skeleton #9343

HDDS-13345. STS port and endpoint skeleton #9343

Uh oh!

Conversation

len548 commented Nov 21, 2025

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

sodonnel commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

fm-cdera Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fm-cdera Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fmorg-git Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

len548 Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fmorg-git commented Dec 7, 2025

Uh oh!

Tejaskriya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fmorg-git Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fapifta commented Jan 13, 2026

Uh oh!

fapifta left a comment

Choose a reason for hiding this comment

fm-cdera Dec 6, 2025 •

edited

Loading

fm-cdera Dec 6, 2025 •

edited

Loading

fmorg-git Dec 11, 2025 •

edited

Loading

len548 Jan 9, 2026 •

edited

Loading

fmorg-git Dec 16, 2025 •

edited

Loading