[PoC] [Do Not Merge] Make policy management APIs separate from IRC spec by HonahX · Pull Request #856 · apache/polaris

HonahX · 2025-01-23T02:05:21Z

Not ready for review, depends on #808

[Update] The draft PR to do the separation on current main branch spec is #906

#808 adds new APIs for policy management. These APIs are not IRC Apis (yet) so it is better to separate them into different yaml files according to Jack's proposal (https://lists.apache.org/thread/1fqocs00pno0xfr4ss2p69d6dv5h8qzf)

This PR is a PoC of one possible way to do the separation.

# Conflicts: # service/common/src/main/java/org/apache/polaris/service/catalog/IcebergCatalogAdapter.java

flyrain · 2025-01-28T20:13:50Z

+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/IcebergErrorResponse'


I guess it is OK to reference object from another yaml file. Can you confirm?

Yes it is possible. I did not do that here initially because there are several other places that reference the IcebergErrorResponse. So I define this in this yaml, but the definition is reference from another yaml file:

schemas: Namespace: $ref: '../rest-catalog-open-api.yaml#/components/schemas/Namespace' IcebergErrorResponse: $ref: '../rest-catalog-open-api.yaml#/components/schemas/IcebergErrorResponse'

https://github.com/HonahX/polaris/blob/c86a4c4f3d4001d671ddcf8bbcc09e5e103614d9/spec/polaris-api/policy-management-api.yaml#L293-L298

flyrain · 2025-01-28T20:17:15Z

+---
+openapi: 3.0.3
+info:
+  title: Apache Iceberg REST Catalog API


Nit: may change the tile a bit reflect its scope, which including both Iceberg and Polaris-native REST APIs

flyrain · 2025-01-28T20:20:44Z

+
+
+components:
+  securitySchemes:


I assume we copy it from Iceberg rest spec yaml, as securitySchemes is NOT referenced by any endpoint. Can you confirm?
Is there a way to add a $ref for components?

The 2 securitySchemes are used as global security settings above:

security: - OAuth2: [catalog] - BearerAuth: []

I just realized we can reference them like this.

components: securitySchemes: OAuth2: $ref: './rest-catalog-open-api.yaml#/components/securitySchemes/OAuth2' BearerAuth: $ref: './rest-catalog-open-api.yaml#/components/securitySchemes/BearerAuth'

flyrain · 2025-01-28T20:23:14Z

+servers:
+  - url: "{scheme}://{host}/{basePath}"
+    description: Server URL when the port can be inferred from the scheme
+    variables:
+      scheme:
+        description: The scheme of the URI, either http or https.
+        default: https
+      host:
+        description: The host address for the specified server
+        default: localhost
+      basePath:
+        description: Optional prefix to be appended to all routes
+        default: ""
+  - url: "{scheme}://{host}:{port}/{basePath}"
+    description: Generic base server URL, with all parts configurable
+    variables:
+      scheme:
+        description: The scheme of the URI, either http or https.
+        default: https
+      host:
+        description: The host address for the specified server
+        default: localhost
+      port:
+        description: The port used when addressing the host
+        default: "443"
+      basePath:
+        description: Optional prefix to be appended to all routes
+        default: ""
+# All routes are currently configured using an Authorization header.
+security:
+  - OAuth2: [catalog]
+  - BearerAuth: []


I think it's worth to document any section copied from somewhere else. I assume this section is copied from Iceberg REST spec.

Good point. I will add comments here.

HonahX · 2025-01-28T23:14:28Z

@flyrain Thanks for reviewing this. I will update the PR and do some additional research on how to render a preview of APIs on website

HonahX · 2025-01-29T00:09:43Z

To render the combined spec in website, we can rely on tools like redocly to re-combine them into one spec file, so that the redoc-polaris shortcode can handle that properly

redocly bundle polaris-catalog-open-api.yaml -o merged.yaml

flyrain · 2025-01-29T00:12:36Z

+  /v1/{prefix}/namespaces/{namespace}/tables/{table}/notifications:
+    $ref: './rest-catalog-open-api.yaml#/paths/~1v1~1{prefix}~1namespaces~1{namespace}~1tables~1{table}~1notifications'


Maybe we could extract this endpoint first in this PR, and left out the policy related API to #808. WDYT?

That sounds a good starting point to split spec. If you do not mind, I can open a new PR to do that and leave the current one as a PoC for policy management APIs. This way, we will always have a reference point to see how things look when adding more APIs to the spec.

Sounds good to me!

Created #906, I temporarily left it as draft because we need to figure out a proper way to update the website pointing to new spec file, otherwise it will break the website.

flyrain · 2025-01-29T00:15:31Z

To render the combined spec in website, we can rely on tools like redocly to re-combine them into one spec file, so that the redoc-polaris shortcode can handle that properly
redocly bundle polaris-catalog-open-api.yaml -o merged.yaml

I think we did that before, but it is removed somehow. It makes sense to me to render them together as there are multiple cross-references.

snazy · 2025-01-29T09:42:43Z

I thought we do not want to have two PRs #906 and #856 for the same thing (see #884 and https://lists.apache.org/thread/bcnh1dwgoxd2dvtqk6z935gfzmh4q0jq)

flyrain · 2025-01-29T18:04:05Z

I thought we do not want to have two PRs #906 and #856 for the same thing (see #884 and https://lists.apache.org/thread/bcnh1dwgoxd2dvtqk6z935gfzmh4q0jq)

This PR depends on #808, while #808 also depends on this one, creating a cyclic dependency. To resolve this, we have two options:

Merge Separate Polaris-native API from Iceberg Rest Catalog API spec #906 first, allowing Polaris Catalog Spec: Policy Management Apis #808 to be based on it. In this case, we can close this PR.
Merge Polaris Catalog Spec: Policy Management Apis #808 first, then revisit and proceed with this PR afterward. We can close Separate Polaris-native API from Iceberg Rest Catalog API spec #906 as we don't need it.

@snazy @HonahX which one do you prefer?

snazy · 2025-01-29T18:16:07Z

I thought we do not want to have two PRs #906 and #856 for the same thing (see #884 and https://lists.apache.org/thread/bcnh1dwgoxd2dvtqk6z935gfzmh4q0jq)

This PR depends on #808, while #808 also depends on this one, creating a cyclic dependency. To resolve this, we have two options:
1. Merge [Separate Polaris-native API from Iceberg Rest Catalog API spec #906](https://github.com/apache/polaris/pull/906) first, allowing [Catalog Spec: Policy Management Apis #808](https://github.com/apache/polaris/pull/808) to be based on it. In this case, we can close this PR.

2. Merge [Catalog Spec: Policy Management Apis #808](https://github.com/apache/polaris/pull/808) first, then revisit and proceed with this PR afterward. We can close [Separate Polaris-native API from Iceberg Rest Catalog API spec #906](https://github.com/apache/polaris/pull/906) as we don't need it.
@snazy @HonahX which one do you prefer?

The titles of the PRs are the same so it's the same thing, or isn't it?

flyrain · 2025-01-29T18:42:34Z

No, this one depends on #808, while #906 doesn't.

HonahX · 2025-01-29T18:44:29Z

Apologies for any confusion. I should have provided a clearer title. Allow me to clarify the intent of this PR:

The current one, PR #856 was initially created as a proof of concept to facilitate discussions on this Apache mailing list thread regarding specification separation. It utilizes policy management APIs to demonstrate that $ref can be used across files to modularize the specification into separate YAML files. However, since this PR includes policy management APIs, it depends on PR #808.

PR #906, on the other hand, will specifically focus on separating Polaris-native APIs (currently limited to the notifications API) from the IRC specification. Additionally, it will include documentation and commands for merging YAML files into a single specification for website display.

I will update the title and description accordingly to reflect these distinctions.

HonahX · 2025-01-29T19:12:20Z

which one do you prefer?

How about we do review of #906 and #808 in parallel. If #906 merges first I will follow option 1. If #808 merges first, I can update #906 to also separate policy management APIs from IRC spec. Either way, this one will be closed finally. I leave it as "draft" now since we can have a reference point to see how things look when adding more APIs to the spec.

@flyrain @snazy WDYT?

HonahX · 2025-02-04T07:15:01Z

#906 is merged, so I will close this one and update #808

HonahX added 16 commits January 12, 2025 22:00

Add policy management APIs spec

e792fff

update errors and fix format issues

35df43b

Update spec based on discussion

9e3d993

update rest spec and generate models

d7526ac

add implementation example

7162da6

Merge branch 'main' into honahx_policy_management_apis

9b5d926

# Conflicts: # service/common/src/main/java/org/apache/polaris/service/catalog/IcebergCatalogAdapter.java

update to new branch

35f2c11

update rest spec

608f521

fix import in PolarisCatalogHandlerWrapper

7993dc0

clean up

b87a4de

remove unnecessary change

b8a1959

PoC

97c3803

separate policy management api out

d65742d

add blank line

eda146b

add missing file

bad36cf

add license header

c86a4c4

HonahX mentioned this pull request Jan 23, 2025

Polaris Catalog Spec: Policy Management Apis #808

Merged

HonahX added 2 commits January 22, 2025 20:48

make models auto-generated

50def94

remove unnecessary models

ba05af5

flyrain reviewed Jan 28, 2025

View reviewed changes

flyrain reviewed Jan 29, 2025

View reviewed changes

snazy mentioned this pull request Jan 29, 2025

Separate Polaris-native API from Iceberg Rest Catalog API spec #906

Merged

HonahX changed the title ~~[WIP] PoC separate Polaris' APIs from Iceberg Rest Catalog API~~ [WIP] Make policy management APIs separate from IRC spec Jan 29, 2025

HonahX changed the title ~~[WIP] Make policy management APIs separate from IRC spec~~ [WIP] [PoC] Make policy management APIs separate from IRC spec Jan 29, 2025

HonahX changed the title ~~[WIP] [PoC] Make policy management APIs separate from IRC spec~~ [PoC] Make policy management APIs separate from IRC spec Jan 29, 2025

HonahX changed the title ~~[PoC] Make policy management APIs separate from IRC spec~~ [PoC] [Do Not Merge] Make policy management APIs separate from IRC spec Jan 29, 2025

HonahX closed this Feb 4, 2025

HonahX deleted the honahx_api_split_try branch February 15, 2025 02:55

HonahX restored the honahx_api_split_try branch February 15, 2025 02:55

		/v1/{prefix}/namespaces/{namespace}/tables/{table}/notifications:
		$ref: './rest-catalog-open-api.yaml#/paths/~1v1~1{prefix}~1namespaces~1{namespace}~1tables~1{table}~1notifications'



		components:
		securitySchemes:

Conversation

HonahX commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain Jan 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HonahX commented Jan 28, 2025

Uh oh!

HonahX commented Jan 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HonahX Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain commented Jan 29, 2025

Uh oh!

snazy commented Jan 29, 2025

Uh oh!

flyrain commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snazy commented Jan 29, 2025

Uh oh!

flyrain commented Jan 29, 2025

Uh oh!

HonahX commented Jan 29, 2025

Uh oh!

HonahX commented Jan 29, 2025

Uh oh!

HonahX commented Feb 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HonahX commented Jan 23, 2025 •

edited

Loading

flyrain Jan 28, 2025 •

edited

Loading

HonahX Jan 29, 2025 •

edited

Loading

flyrain commented Jan 29, 2025 •

edited

Loading