From 7eef36b90ca46b9ba968424c16efaa8c345b44aa Mon Sep 17 00:00:00 2001 From: Fabian Morgan Date: Thu, 30 Oct 2025 12:59:02 -0700 Subject: [PATCH 01/14] add design doc for STS --- hadoop-hdds/docs/content/design/ozone-sts.md | 132 +++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 hadoop-hdds/docs/content/design/ozone-sts.md diff --git a/hadoop-hdds/docs/content/design/ozone-sts.md b/hadoop-hdds/docs/content/design/ozone-sts.md new file mode 100644 index 000000000000..c76241db7155 --- /dev/null +++ b/hadoop-hdds/docs/content/design/ozone-sts.md @@ -0,0 +1,132 @@ +--- +title: AWS STS Design for Ozone S3 +summary: STS Support in Ozone +date: 2026-10-30 +jira: HDDS-13323 +status: implementing +author: Ren Koike, Fabian Morgan +--- + + +# AWS STS Design for Ozone S3 + +# 1. Introduction + +S3 credentials used to communicate with Ozone S3 APIs are based on a Kerberos identity. + +Historically, the Ozone community has had interest in a REST API capable of programmatically generating +temporary S3 credentials. + +Amazon AWS has the [Security Token Service (STS)](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) which +provides the ability to generate short-lived access to resources. + +The primary scope of this document is to detail the initial implementation of STS within the Ozone ecosystem. + +# 2. Why Use STS Tokens? + +Providing short-lived access to various resources in Ozone is useful in scenarios such as Data Lake +solutions that want to aggregate data across multiple cloud providers. + +# 3. How Ozone STS Works + +The initial implementation of Ozone STS supports only the [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) +API from the AWS specification. A new STS endpoint will be created to service STS requests in the S3 Gateway. + +## 3.1 Capabilities + +The Ozone STS implementation has the following capabilities: + +- Create temporary credentials that last from a minimum of 15 minutes to a maximum of 12 hours. The +return value of the AssumeRole call will be temporary credentials consisting of 3 components: + - accessKeyId - a generated String identifier beginning with the sequence "ASIA" + - secretAccessKey - a generated String password + - sessionToken - an opaque String identifier +- The temporary credentials will have the permissions associated with a role. Furthermore, an +[AWS IAM Session Policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) can +**optionally** be sent in the AssumeRole API call to limit the scope of the permissions further. If +an IAM policy is specified, the temporary credential will have the permissions comprising the intersection of the role permissions +and the IAM policy permissions. **Note:** If the IAM policy is specified and does not grant any permissions, then +the generated temporary credentials won't have any permissions and will essentially be useless. + +## 3.2 Limitations in IAM Session Policy Support + +The AWS IAM policy specification is vast and wide-ranging. The initial Ozone STS supports a limited +subset of its capabilities. The restrictions are outlined below: + +- The only supported prefix in ResourceArn is `arn:aws:s3:::` - all others will be rejected. **Note**: a ResourceArn +of `*` is supported as well. +- The only supported Condition operator is `StringEquals` - all others will be rejected. +- The only supported Condition attribute is `s3:prefix` - all others will be rejected. +- Only one Condition operator per Statement is supported - a Statement with more than one condition will be rejected. +- The only supported Effect is `Allow` - all others will be rejected. +- If a (currently) unsupported S3 action is requested, such as `s3:GetAccelerateConfiguration`, it will be silently ignored. +Similarly, an invalid S3 action will be silently ignored. +- Supported wildcard expansions in Actions are: `s3:*`, `s3:Get*`, `s3:Put*`, `s3:List*`, +`s3:Create*`, and `s3:Delete*`. +- If using OzoneNativeAuthorizer, bucket wildcards (ex. ResourceArn `arn:aws:s3:::*`, `arn:aws:s3:::bucket*` or `*`) will be rejected. +However, certain object wildcards be accepted. For example, ResourceArn `arn:aws:s3:::myBucket/*` and `arn:aws:s3:::myBucket/folder/logs/*` +will be accepted but `arn:aws:s3:::myBucket/file*.txt` +will not be accepted. + +## 3.3 SessionToken Format + +As mentioned above, one of the return values from the AssumeRole call will be the sessionToken. To support not +storing temporary credentials server-side in Ozone, the sessionToken will comprise various components needed to validate +subsequent S3 calls that use the token. The sessionToken will have the following information encoded: + +- The originalAccessKeyId - this is the Kerberos identity of the user that created the sessionToken via the AssumeRole call. +When the temporary credentials are used to make S3 API calls, this Kerberos identity (in conjunction with the role permissions and +optional session policy) will be used to authorize the call. +- The roleArn - the role used in the original AssumeRole call +- The encrypted secretAccessKey - this will be used to validate the AWS signature when the temporary credentials are used +to make S3 API calls +- (Optional) sessionPolicy - when using the RangerOzoneAuthorizer, if Ranger successfully authorizes the AssumeRole call, +it will return a String representing the resources (i.e. buckets, keys, etc.) and permissions (i.e. ACLType) that the token +has been granted access to. This sessionPolicy will be included in the sessionToken sent back to the client so it can be sent to Ranger to +authorize subsequent S3 API calls that use the sessionToken. +- HMAC-SHA256 signature - used to ensure the sessionToken was created by Ozone and was not altered since it was created. +- The expiration time of the token (via `ShortLivedTokenIdentifier#getExpiry()`) +- The UUID of the secret key used to sign the sessionToken and encrypt the secretAccessKey (via `ShortLivedTokenIdentifier#getSecretKeyId()`) + +## 3.5 STS Token Revocation + +In the rare event temporary credentials need to be revoked (ex. for security reasons), a table in RocksDB will be created +to store revoked tokens, and a command-line utility will be created to add tokens to the table. A background cleaner service +will be created to run every 3 hours to delete revoked tokens that have been in the table for more than 12 hours. + +## 3.6 Overall Flow + +The following section outlines the overall flow when using STS in Ozone: + +- An authorized user (having permanent S3 credentials) makes the AssumeRole STS call to Ozone. +- If successful, Ozone responds with the temporary credentials. +- A client makes S3 API calls with the temporary credentials for up to as long as the credentials last. +- When Ozone receives an S3 api call using temporary credentials, it will perform the following checks: + - Ensure the token is not expired + - Ensure the token is not revoked + - Validate the HMAC-SHA256 signature in the sessionToken + - Decrypt the secretAccessKey from the sessionToken and validate the AWS signature + - Authorize the call with either RangerOzoneAuthorizer or OzoneNativeAuthorizer + +Assuming all these checks pass, the S3 API call will be invoked. + +# 4. Prerequisites + +A user must be configured with a Kerberos identity in Ozone and the S3 `getSecret` command +must be called to issue permanent S3 credentials. With these credentials, the AssumeRole API can be made. +Furthermore, when using RangerOzoneAuthorizer, a role must be configured in Ranger UI for each role the AssumeRole API +can be used with. The user making the AssumeRole call must be in this role, and this role must have access +to the (new) `ASSUME_ROLE` permission in Ranger, in addition to the volumes/buckets/keys and other permissions (such as `READ`, +`LIST`, etc.) that are required. From cf8be6c2ce0472bd0e8cf56b92c0db36e81950c9 Mon Sep 17 00:00:00 2001 From: Fabian Morgan Date: Thu, 30 Oct 2025 13:05:52 -0700 Subject: [PATCH 02/14] add sample policy --- hadoop-hdds/docs/content/design/ozone-sts.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/hadoop-hdds/docs/content/design/ozone-sts.md b/hadoop-hdds/docs/content/design/ozone-sts.md index c76241db7155..952867181d16 100644 --- a/hadoop-hdds/docs/content/design/ozone-sts.md +++ b/hadoop-hdds/docs/content/design/ozone-sts.md @@ -80,6 +80,21 @@ However, certain object wildcards be accepted. For example, ResourceArn `arn:a will be accepted but `arn:aws:s3:::myBucket/file*.txt` will not be accepted. +A sample IAM policy that allows read access to all objects in the `example-bucket` bucket is shown below: +```JSON +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "s3:GetObject", + "Resource": "arn:aws:s3:::example-bucket/*" + } + ] +} + +``` + ## 3.3 SessionToken Format As mentioned above, one of the return values from the AssumeRole call will be the sessionToken. To support not From 2a1b532808b2e2ef0cc229bc5e40a201ef5d0066 Mon Sep 17 00:00:00 2001 From: Fabian Morgan Date: Fri, 31 Oct 2025 17:25:32 -0700 Subject: [PATCH 03/14] PR review comments and minor formatting --- hadoop-hdds/docs/content/design/ozone-sts.md | 65 +++++++++++++++----- 1 file changed, 50 insertions(+), 15 deletions(-) diff --git a/hadoop-hdds/docs/content/design/ozone-sts.md b/hadoop-hdds/docs/content/design/ozone-sts.md index 952867181d16..87e3260b89ca 100644 --- a/hadoop-hdds/docs/content/design/ozone-sts.md +++ b/hadoop-hdds/docs/content/design/ozone-sts.md @@ -42,7 +42,7 @@ solutions that want to aggregate data across multiple cloud providers. # 3. How Ozone STS Works The initial implementation of Ozone STS supports only the [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) -API from the AWS specification. A new STS endpoint will be created to service STS requests in the S3 Gateway. +API from the AWS specification. A new STS endpoint `/sts` on port `9880` will be created to service STS requests in the S3 Gateway. ## 3.1 Capabilities @@ -69,16 +69,17 @@ subset of its capabilities. The restrictions are outlined below: of `*` is supported as well. - The only supported Condition operator is `StringEquals` - all others will be rejected. - The only supported Condition attribute is `s3:prefix` - all others will be rejected. -- Only one Condition operator per Statement is supported - a Statement with more than one condition will be rejected. +- Only one Condition operator per Statement is supported - a Statement with more than one Condition will be rejected. - The only supported Effect is `Allow` - all others will be rejected. - If a (currently) unsupported S3 action is requested, such as `s3:GetAccelerateConfiguration`, it will be silently ignored. Similarly, an invalid S3 action will be silently ignored. - Supported wildcard expansions in Actions are: `s3:*`, `s3:Get*`, `s3:Put*`, `s3:List*`, `s3:Create*`, and `s3:Delete*`. -- If using OzoneNativeAuthorizer, bucket wildcards (ex. ResourceArn `arn:aws:s3:::*`, `arn:aws:s3:::bucket*` or `*`) will be rejected. +- If using OzoneNativeAuthorizer, bucket wildcards (ex. ResourceArn `arn:aws:s3:::*`, `arn:aws:s3:::bucket*` or `*`) will be rejected, +as wildcards are not supported for buckets for the OzoneNativeAuthorizer. However, certain object wildcards be accepted. For example, ResourceArn `arn:aws:s3:::myBucket/*` and `arn:aws:s3:::myBucket/folder/logs/*` will be accepted but `arn:aws:s3:::myBucket/file*.txt` -will not be accepted. +will not be accepted. Again, these restrictions correspond to the capabilities of the OzoneNativeAuthorizer. A sample IAM policy that allows read access to all objects in the `example-bucket` bucket is shown below: ```JSON @@ -121,11 +122,54 @@ In the rare event temporary credentials need to be revoked (ex. for security rea to store revoked tokens, and a command-line utility will be created to add tokens to the table. A background cleaner service will be created to run every 3 hours to delete revoked tokens that have been in the table for more than 12 hours. -## 3.6 Overall Flow +## 3.6 Prerequisites + +A user must be configured with a Kerberos identity in Ozone and the S3 `getSecret` command +must be called to issue permanent S3 credentials. With these credentials, the AssumeRole API can be made. +Furthermore, when using RangerOzoneAuthorizer, a role must be configured in Ranger UI for each role the AssumeRole API +can be used with. The user making the AssumeRole call must be in this role, and this role must have access +to the (new) `ASSUME_ROLE` permission in Ranger, in addition to the volumes/buckets/keys and other permissions (such as `READ`, +`LIST`, etc.) that are required. + +### 3.6.1 Additions to RangerOzoneAuthorizer + +The `IAccessAuthorizer` interface that both the RangerOzoneAuthorizer and OzoneNativeAuthorizer implement, will have a +new method: + +```java +default String generateAssumeRoleSessionPolicy(AssumeRoleRequest assumeRoleRequest) throws OMException { + throw new OMException("The generateAssumeRoleSessionPolicy call is not supported", NOT_SUPPORTED_OPERATION); +} +``` + +When using RangerOzoneAuthorizer, the AssumeRole API call must invoke this method to ensure the caller is authorized to create +temporary credentials, given the criteria in the AssumeRoleRequest. The AssumeRoleRequest input parameter will have the +components: +- `String` host - hostname of caller +- `InetAddress` ip - IP address of caller +- `UserGroupInformation` ugi - the user making the call +- `String` targetRoleName - what role is being assumed +- `Set` grants - further limiting the scope of the role according to the grants + +The grants parameter is optional, and would only be present if the AssumeRole API call had an IAM session policy JSON +parameter supplied. A conversion utility, `IamSessionPolicyResolver` will process the IAM policy and convert it to a +`Set`, in effect translating from S3 nomenclature for resources and actions to Ozone nomenclature of +`IOzoneObj` and `ACLType`. Ranger would use all of this information to determine if the AssumeRole call should be +successfully authorized, and if so, it will return a String representation of the granted permissions and paths. + +The format of this String is entirely up to the Ranger team. What is required from the Ozone side is to supply this String to Ranger when any +subsequent S3 API calls are made that use STS tokens. In order to achieve this, the sessionPolicy String from Ranger will +be included in the sessionToken response to the AssumeRole API call (as mentioned above), and Ozone will supply this String +to Ranger whenever STS tokens are used on S3 API calls via a new `RequestContext.sessionPolicy` field in the +`IAccessAuthorizer#checkAccess(IOzoneObj, RequestContext)` call. + +## 3.7 Overall Flow The following section outlines the overall flow when using STS in Ozone: -- An authorized user (having permanent S3 credentials) makes the AssumeRole STS call to Ozone. +- An authorized user for AssumeRole API calls must be configured in Ozone, and if using RangerOzoneAuthorizer, the role +created in Ranger as per the Prerequisites above. +- This authorized user (having permanent S3 credentials) makes the AssumeRole STS call to Ozone. - If successful, Ozone responds with the temporary credentials. - A client makes S3 API calls with the temporary credentials for up to as long as the credentials last. - When Ozone receives an S3 api call using temporary credentials, it will perform the following checks: @@ -136,12 +180,3 @@ The following section outlines the overall flow when using STS in Ozone: - Authorize the call with either RangerOzoneAuthorizer or OzoneNativeAuthorizer Assuming all these checks pass, the S3 API call will be invoked. - -# 4. Prerequisites - -A user must be configured with a Kerberos identity in Ozone and the S3 `getSecret` command -must be called to issue permanent S3 credentials. With these credentials, the AssumeRole API can be made. -Furthermore, when using RangerOzoneAuthorizer, a role must be configured in Ranger UI for each role the AssumeRole API -can be used with. The user making the AssumeRole call must be in this role, and this role must have access -to the (new) `ASSUME_ROLE` permission in Ranger, in addition to the volumes/buckets/keys and other permissions (such as `READ`, -`LIST`, etc.) that are required. From 51f0f0a50b26025831978f3de0ffb9a893f02c79 Mon Sep 17 00:00:00 2001 From: Fabian Morgan Date: Mon, 3 Nov 2025 16:27:44 -0800 Subject: [PATCH 04/14] updates per PR review comments --- hadoop-hdds/docs/content/design/ozone-sts.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/hadoop-hdds/docs/content/design/ozone-sts.md b/hadoop-hdds/docs/content/design/ozone-sts.md index 87e3260b89ca..0d9339656465 100644 --- a/hadoop-hdds/docs/content/design/ozone-sts.md +++ b/hadoop-hdds/docs/content/design/ozone-sts.md @@ -44,6 +44,10 @@ solutions that want to aggregate data across multiple cloud providers. The initial implementation of Ozone STS supports only the [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) API from the AWS specification. A new STS endpoint `/sts` on port `9880` will be created to service STS requests in the S3 Gateway. +Furthermore, the initial implementation of Ozone STS focuses only on Apache Ranger for the first phase, as it aligns more +with IAM policies. Support for the Ozone Native Authorizer may be provided in a future phase. Consideration for the Ozone +Native Authorizer will be given when processing IAM policies as described below. + ## 3.1 Capabilities The Ozone STS implementation has the following capabilities: @@ -125,8 +129,9 @@ will be created to run every 3 hours to delete revoked tokens that have been in ## 3.6 Prerequisites A user must be configured with a Kerberos identity in Ozone and the S3 `getSecret` command -must be called to issue permanent S3 credentials. With these credentials, the AssumeRole API can be made. -Furthermore, when using RangerOzoneAuthorizer, a role must be configured in Ranger UI for each role the AssumeRole API +must be called to issue permanent S3 credentials. With these credentials, the AssumeRole API call can be made, but additional +steps below are needed for the call to be successfully authorized. +When using RangerOzoneAuthorizer, a role must be configured in Ranger UI for each role the AssumeRole API can be used with. The user making the AssumeRole call must be in this role, and this role must have access to the (new) `ASSUME_ROLE` permission in Ranger, in addition to the volumes/buckets/keys and other permissions (such as `READ`, `LIST`, etc.) that are required. @@ -149,11 +154,11 @@ components: - `InetAddress` ip - IP address of caller - `UserGroupInformation` ugi - the user making the call - `String` targetRoleName - what role is being assumed -- `Set` grants - further limiting the scope of the role according to the grants +- `Set` grants - further limiting the scope of the role according to the grants The grants parameter is optional, and would only be present if the AssumeRole API call had an IAM session policy JSON parameter supplied. A conversion utility, `IamSessionPolicyResolver` will process the IAM policy and convert it to a -`Set`, in effect translating from S3 nomenclature for resources and actions to Ozone nomenclature of +`Set`, in effect translating from S3 nomenclature for resources and actions to Ozone nomenclature of `IOzoneObj` and `ACLType`. Ranger would use all of this information to determine if the AssumeRole call should be successfully authorized, and if so, it will return a String representation of the granted permissions and paths. From 2690b6e3358b6314e8396ed74fa3567bfb81d8ff Mon Sep 17 00:00:00 2001 From: Fabian Morgan Date: Thu, 6 Nov 2025 11:29:54 -0800 Subject: [PATCH 05/14] pr review comments for revocation --- hadoop-hdds/docs/content/design/ozone-sts.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hadoop-hdds/docs/content/design/ozone-sts.md b/hadoop-hdds/docs/content/design/ozone-sts.md index 0d9339656465..9c6205374e4b 100644 --- a/hadoop-hdds/docs/content/design/ozone-sts.md +++ b/hadoop-hdds/docs/content/design/ozone-sts.md @@ -124,7 +124,10 @@ authorize subsequent S3 API calls that use the sessionToken. In the rare event temporary credentials need to be revoked (ex. for security reasons), a table in RocksDB will be created to store revoked tokens, and a command-line utility will be created to add tokens to the table. A background cleaner service -will be created to run every 3 hours to delete revoked tokens that have been in the table for more than 12 hours. +will be created to run every 3 hours to delete revoked tokens that have been in the table for more than 12 hours. The +input parameter for the command-line utility will be the accessKeyId of the temporary token - this value is returned in +plain text as a result of the AssumeRole call (mentioned above). In this way, specific STS tokens can be revoked as opposed +to all tokens. Furthermore, AWS doesn't have a standard API to revoke tokens therefore we are creating our own system. ## 3.6 Prerequisites From 0de4d53c64474d7286c17233067610da0166ce19 Mon Sep 17 00:00:00 2001 From: Fabian Morgan Date: Fri, 7 Nov 2025 10:29:15 -0800 Subject: [PATCH 06/14] pr review updates and additional details --- hadoop-hdds/docs/content/design/ozone-sts.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/hadoop-hdds/docs/content/design/ozone-sts.md b/hadoop-hdds/docs/content/design/ozone-sts.md index 9c6205374e4b..4eb951e4d9f0 100644 --- a/hadoop-hdds/docs/content/design/ozone-sts.md +++ b/hadoop-hdds/docs/content/design/ozone-sts.md @@ -1,7 +1,7 @@ --- title: AWS STS Design for Ozone S3 summary: STS Support in Ozone -date: 2026-10-30 +date: 2025-10-30 jira: HDDS-13323 status: implementing author: Ren Koike, Fabian Morgan @@ -44,9 +44,9 @@ solutions that want to aggregate data across multiple cloud providers. The initial implementation of Ozone STS supports only the [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) API from the AWS specification. A new STS endpoint `/sts` on port `9880` will be created to service STS requests in the S3 Gateway. -Furthermore, the initial implementation of Ozone STS focuses only on Apache Ranger for the first phase, as it aligns more -with IAM policies. Support for the Ozone Native Authorizer may be provided in a future phase. Consideration for the Ozone -Native Authorizer will be given when processing IAM policies as described below. +Furthermore, the initial implementation of Ozone STS focuses only on Apache Ranger for authorization in the first phase, +as it aligns more with IAM policies. Support for the Ozone Native Authorizer may be provided in a future phase. +Consideration for the Ozone Native Authorizer will be given when processing IAM policies as described below. ## 3.1 Capabilities @@ -54,9 +54,9 @@ The Ozone STS implementation has the following capabilities: - Create temporary credentials that last from a minimum of 15 minutes to a maximum of 12 hours. The return value of the AssumeRole call will be temporary credentials consisting of 3 components: - - accessKeyId - a generated String identifier beginning with the sequence "ASIA" - - secretAccessKey - a generated String password - - sessionToken - an opaque String identifier + - accessKeyId - a generated String identifier (cryptographically strong using SecureRandom) beginning with the sequence "ASIA" + - secretAccessKey - a generated String password (cryptographically strong using SecureRandom) + - sessionToken - a Base64-encoded opaque String identifier - The temporary credentials will have the permissions associated with a role. Furthermore, an [AWS IAM Session Policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) can **optionally** be sent in the AssumeRole API call to limit the scope of the permissions further. If @@ -181,8 +181,9 @@ created in Ranger as per the Prerequisites above. - If successful, Ozone responds with the temporary credentials. - A client makes S3 API calls with the temporary credentials for up to as long as the credentials last. - When Ozone receives an S3 api call using temporary credentials, it will perform the following checks: - - Ensure the token is not expired - - Ensure the token is not revoked + - Ensure that if the accessKeyId starts with "ASIA", that a sessionToken was included in the `x-amz-security-token` header + - Ensure the sessionToken is not expired + - Ensure the sessionToken is not revoked - Validate the HMAC-SHA256 signature in the sessionToken - Decrypt the secretAccessKey from the sessionToken and validate the AWS signature - Authorize the call with either RangerOzoneAuthorizer or OzoneNativeAuthorizer From d6d3f77e7afb494a9d7b1be9a048a12dd0103f0a Mon Sep 17 00:00:00 2001 From: Fabian Morgan Date: Mon, 10 Nov 2025 11:57:14 -0800 Subject: [PATCH 07/14] pr review updates from ChenSammi --- hadoop-hdds/docs/content/design/ozone-sts.md | 43 +++++++++++++------- 1 file changed, 28 insertions(+), 15 deletions(-) diff --git a/hadoop-hdds/docs/content/design/ozone-sts.md b/hadoop-hdds/docs/content/design/ozone-sts.md index 4eb951e4d9f0..6695783d122b 100644 --- a/hadoop-hdds/docs/content/design/ozone-sts.md +++ b/hadoop-hdds/docs/content/design/ozone-sts.md @@ -64,7 +64,14 @@ an IAM policy is specified, the temporary credential will have the permissions c and the IAM policy permissions. **Note:** If the IAM policy is specified and does not grant any permissions, then the generated temporary credentials won't have any permissions and will essentially be useless. -## 3.2 Limitations in IAM Session Policy Support +## 3.2 Limitations in AssumeRole API Support + +The AWS AssumeRole API has various required and optional fields. We will support the two required fields, i.e. `RoleArn` +and `RoleSessionName`. Additionally, we will support the following optional fields (all others will be rejected): +- `DurationSeconds` +- `Policy` + +## 3.3 Limitations in IAM Session Policy Support The AWS IAM policy specification is vast and wide-ranging. The initial Ozone STS supports a limited subset of its capabilities. The restrictions are outlined below: @@ -100,29 +107,34 @@ A sample IAM policy that allows read access to all objects in the `example-bucke ``` -## 3.3 SessionToken Format +## 3.4 SessionToken Format As mentioned above, one of the return values from the AssumeRole call will be the sessionToken. To support not storing temporary credentials server-side in Ozone, the sessionToken will comprise various components needed to validate subsequent S3 calls that use the token. The sessionToken will have the following information encoded: -- The originalAccessKeyId - this is the Kerberos identity of the user that created the sessionToken via the AssumeRole call. +- originalAccessKeyId - this is the Kerberos identity of the user that created the sessionToken via the AssumeRole call. When the temporary credentials are used to make S3 API calls, this Kerberos identity (in conjunction with the role permissions and -optional session policy) will be used to authorize the call. -- The roleArn - the role used in the original AssumeRole call -- The encrypted secretAccessKey - this will be used to validate the AWS signature when the temporary credentials are used +optional session policy) will be used to authorize the call. This identity is included in the sessionToken because +S3 API calls (such as PutObject) require a Kerberos identity, but the temporary credentials don't have a +Kerberos identity associated to them, therefore the Kerberos identity of the user that created the token will be used in +these cases. +- roleArn - the role used in the original AssumeRole call +- encrypted secretAccessKey - this will be used to validate the AWS signature when the temporary credentials are used to make S3 API calls -- (Optional) sessionPolicy - when using the RangerOzoneAuthorizer, if Ranger successfully authorizes the AssumeRole call, -it will return a String representing the resources (i.e. buckets, keys, etc.) and permissions (i.e. ACLType) that the token -has been granted access to. This sessionPolicy will be included in the sessionToken sent back to the client so it can be sent to Ranger to -authorize subsequent S3 API calls that use the sessionToken. +- sessionPolicy - when using the RangerOzoneAuthorizer, if Ranger successfully authorizes the AssumeRole call, +it will return a String representing the role the token was authorized for. Furthermore, if an AWS IAM Session Policy +was included with the AssumeRole request, the String return value will also include resources (i.e. buckets, keys, etc.) +and permissions (i.e. ACLType) corresponding to the AWS IAM Session Policy. These resources and permissions, if present, +would further limit the scope of the permissions and resources granted by the role in Ranger, such that the temporary +credential will have the permissions comprising the intersection of the role permissions and the sessionPolicy permissions. - HMAC-SHA256 signature - used to ensure the sessionToken was created by Ozone and was not altered since it was created. -- The expiration time of the token (via `ShortLivedTokenIdentifier#getExpiry()`) -- The UUID of the secret key used to sign the sessionToken and encrypt the secretAccessKey (via `ShortLivedTokenIdentifier#getSecretKeyId()`) +- expiration time of the token (via `ShortLivedTokenIdentifier#getExpiry()`) +- UUID of the OzoneManager private key used to sign the sessionToken and encrypt the secretAccessKey (via `ShortLivedTokenIdentifier#getSecretKeyId()`) ## 3.5 STS Token Revocation -In the rare event temporary credentials need to be revoked (ex. for security reasons), a table in RocksDB will be created +In the rare event temporary credentials need to be revoked (ex. for security reasons), a table in the OzoneManager RocksDB will be created to store revoked tokens, and a command-line utility will be created to add tokens to the table. A background cleaner service will be created to run every 3 hours to delete revoked tokens that have been in the table for more than 12 hours. The input parameter for the command-line utility will be the accessKeyId of the temporary token - this value is returned in @@ -180,10 +192,11 @@ created in Ranger as per the Prerequisites above. - This authorized user (having permanent S3 credentials) makes the AssumeRole STS call to Ozone. - If successful, Ozone responds with the temporary credentials. - A client makes S3 API calls with the temporary credentials for up to as long as the credentials last. -- When Ozone receives an S3 api call using temporary credentials, it will perform the following checks: +- When Ozone receives an S3 api call using temporary credentials, it will use the Kerberos identity associated with the +originalAccessKeyId in the session token and perform the following checks: - Ensure that if the accessKeyId starts with "ASIA", that a sessionToken was included in the `x-amz-security-token` header - Ensure the sessionToken is not expired - - Ensure the sessionToken is not revoked + - Ensure the sessionToken is not revoked via a `keyMayExist` check in OzoneManager RocksDB - Validate the HMAC-SHA256 signature in the sessionToken - Decrypt the secretAccessKey from the sessionToken and validate the AWS signature - Authorize the call with either RangerOzoneAuthorizer or OzoneNativeAuthorizer From 885bbd3034eb12df47218ea95a24b80add4ff4a3 Mon Sep 17 00:00:00 2001 From: Fabian Morgan Date: Tue, 11 Nov 2025 08:23:23 -0800 Subject: [PATCH 08/14] pr review updates for ChenSammi --- hadoop-hdds/docs/content/design/ozone-sts.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/hadoop-hdds/docs/content/design/ozone-sts.md b/hadoop-hdds/docs/content/design/ozone-sts.md index 6695783d122b..9b9593be8d80 100644 --- a/hadoop-hdds/docs/content/design/ozone-sts.md +++ b/hadoop-hdds/docs/content/design/ozone-sts.md @@ -73,7 +73,7 @@ and `RoleSessionName`. Additionally, we will support the following optional fie ## 3.3 Limitations in IAM Session Policy Support -The AWS IAM policy specification is vast and wide-ranging. The initial Ozone STS supports a limited +The AWS IAM policy specification is vast and wide-ranging. The initial Ozone STS implementation supports a limited subset of its capabilities. The restrictions are outlined below: - The only supported prefix in ResourceArn is `arn:aws:s3:::` - all others will be rejected. **Note**: a ResourceArn @@ -107,6 +107,18 @@ A sample IAM policy that allows read access to all objects in the `example-bucke ``` +### 3.3.1 Additional Context on Design Behavior + +As mentioned above, some limitations in IAM Session Policy support result in API calls being rejected, while others are +silently ignored. This design behavior came after (much) discussion with external teams. One external team will send +S3 actions that Ozone doesn't support, and they don't have flexibility to change what they are sending, so that's one +reason for the silent ignore. This external team also mentioned that some research indicates AWS also does not fail +the AssumeRole request just because the inline session policy references unknown or unsupported actions, but rather +it will fail when the temporary credentials are used, so this design is accordance with that finding. Another external +team agreed that behavior is fine for actions, but does not work for Conditions, because one can have a Condition to +restrict calls by sourceIp, and if we silently ignore this, the client may incorrectly think the temporary credentials +are restricted for use by that IP address, so the consensus was to reject the request for that scenario. + ## 3.4 SessionToken Format As mentioned above, one of the return values from the AssumeRole call will be the sessionToken. To support not @@ -130,7 +142,7 @@ would further limit the scope of the permissions and resources granted by the ro credential will have the permissions comprising the intersection of the role permissions and the sessionPolicy permissions. - HMAC-SHA256 signature - used to ensure the sessionToken was created by Ozone and was not altered since it was created. - expiration time of the token (via `ShortLivedTokenIdentifier#getExpiry()`) -- UUID of the OzoneManager private key used to sign the sessionToken and encrypt the secretAccessKey (via `ShortLivedTokenIdentifier#getSecretKeyId()`) +- UUID of the OzoneManager secret key used to sign the sessionToken and encrypt the secretAccessKey (via `ShortLivedTokenIdentifier#getSecretKeyId()`) ## 3.5 STS Token Revocation From 2f70ab46a4a26135e0b1aead3f59604bd0c6236c Mon Sep 17 00:00:00 2001 From: Fabian Morgan Date: Wed, 12 Nov 2025 12:01:17 -0800 Subject: [PATCH 09/14] pr review comments per errose28 --- hadoop-hdds/docs/content/design/ozone-sts.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/hadoop-hdds/docs/content/design/ozone-sts.md b/hadoop-hdds/docs/content/design/ozone-sts.md index 9b9593be8d80..d88794298d88 100644 --- a/hadoop-hdds/docs/content/design/ozone-sts.md +++ b/hadoop-hdds/docs/content/design/ozone-sts.md @@ -43,6 +43,10 @@ solutions that want to aggregate data across multiple cloud providers. The initial implementation of Ozone STS supports only the [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) API from the AWS specification. A new STS endpoint `/sts` on port `9880` will be created to service STS requests in the S3 Gateway. +We use a separate port for STS to align with AWS so we don't have conflicts at a later time. This means we have: +- Admin port for Ozone specific S3 admin operations +- STS port for STS APIs, analogous to AWS' separate STS endpoint +- Existing dedicated port/endpoint for S3 object APIs. Furthermore, the initial implementation of Ozone STS focuses only on Apache Ranger for authorization in the first phase, as it aligns more with IAM policies. Support for the Ozone Native Authorizer may be provided in a future phase. From aa147b31e7e488028b654f768685b2575aeef2af Mon Sep 17 00:00:00 2001 From: Fabian Morgan Date: Fri, 14 Nov 2025 04:02:59 -0800 Subject: [PATCH 10/14] update author list --- hadoop-hdds/docs/content/design/ozone-sts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdds/docs/content/design/ozone-sts.md b/hadoop-hdds/docs/content/design/ozone-sts.md index d88794298d88..f53c9b7b9269 100644 --- a/hadoop-hdds/docs/content/design/ozone-sts.md +++ b/hadoop-hdds/docs/content/design/ozone-sts.md @@ -4,7 +4,7 @@ summary: STS Support in Ozone date: 2025-10-30 jira: HDDS-13323 status: implementing -author: Ren Koike, Fabian Morgan +author: Madhan Neethiraj, Ren Koike, Fabian Morgan, Stephen O'Donnell, Istvan Fajth, Uma Maheswara Rao Gangumalla ---