feat: Improve documentation

README.md

@@ -36,10 +36,17 @@ difference of factor **100** which is not going to be easy to beat.
 
 ## Trying
 
-Trying Keystone (assuming you have the Rust build environment or you are in the 
+Trying Keystone (assuming you have the Rust build environment or you are in the
 possession of the binary is as easy as `keystone -c etc/keystone.conf -vv`
 
+Alternatively you can try it with `docker compose -f docker-compose.yaml up`.
+
+## Documentation
+
+Comprehensive (as much as it can be at the current stage) is available
+[here](https://gtema.github.io/keystone).
+
 ## Talks
 
-Detailed introduction of the project was given as 
+Detailed introduction of the project was given as
 [ALASCA tech talk](https://www.youtube.com/watch?v=0Hx4Q22ZNFU).

doc/src/SUMMARY.md

@@ -2,14 +2,19 @@
 
 [Introduction](./intro.md)
 [Installation](./install.md)
-[Policy](./policy.md)
 
 ---
 
-- [API](./swagger-ui.html)
+[Architecture](./architecture.md)
+[Policy enforcement](./policy.md)
+
+---
+
 - [Federation](./federation.md)
+  - [Oidc RP mode](./oidc.md)
+  - [JWT](./jwt.md)
 - [Passkeys](./passkey.md)
-
+- [API](./swagger-ui.html)
 
 ---
 

doc/src/architecture.md

@@ -0,0 +1,57 @@
+# Architecture
+
+Keystone requires 2 additional components to run:
+
+- database (the same as the py-keystone uses)
+
+- OpenPolicyAgent, that implements API policy enforcement
+
+```mermaid
+architecture-beta
+
+    service db(database)[Database]
+    service keystone(server)[Keystone]
+    service opa(server)[OpenPolicyAgent]
+
+    db:L -- R:keystone
+    opa:L -- T:keystone
+
+```
+
+## Database
+
+Python keystone uses the sqlalchemy as ORM and the migration tool. It cannot be
+used from Rust efficiently, therefore keystone-ng uses the `sea-orm` which
+provides async support natively and also allows database type abstraction.
+Current development focuses on the PostgreSQL database. The MySQL should be
+supported, but is not currently tested against.
+
+New API and resources are being added. This requires database changes. sea-orm
+also comed with the migration tools. However there is a slight difference
+between sqlalchemy and sea-orm. The later suggests doing database schema first.
+In the next step object types are created out of the database. That means that
+the database migration must be written first and cannot be automatically
+generated from the code (easily, but there is a way). Current migrations do not
+create database schema that is managed by the py-keystone. Therefore in order
+to get a fully populated database schema it is necessary to apply
+`keystone-manage db_sync` and `keystone-db up` independently.
+
+Target of the keystone-ng is to be deployed in pair with the python keystone of
+"any" version. Due to that it is not possible to assume the state of the
+database, nor to apply any changes to the schema manaaged by the py-keystone. A
+federation rework assumes model change. To keep it working with the
+python-keystone artificial table entries may be created (in the example when a
+new identity provider is being created automatically sanitized entries are
+being added for the legacy identity provider and necessary protocols) A
+federation rework assumes model change. To keep it working with the
+python-keystone artificial table entries may be created (in the example when a
+new identity provider is being created automatically sanitized entries are
+being added for the legacy identity provider together with necessary idp
+protocols).
+
+## Fernet
+
+keystone-ng uses the same mechanism for tokens to provide compatibility. The
+fernet-keys repository must be provided in the runtime (i.e. by mounting them
+as a volume into the container). There is no tooling to create or rotate keys
+as the py-keystone does.

doc/src/database.md

@@ -0,0 +1 @@
+# Database

doc/src/federation.md

@@ -19,169 +19,36 @@ pretty big number of limitations (not limited to):
 - Client authentication right now is complex and error prone (every public
   provider has implementation specifics that are often even not cross-compatible)
 
-
-In order to address those challenges and complete reimplementation is being
-done here. This leads to a completely different design opening doors for new
-features.
+In order to address those challenges a complete reimplementation is being done
+with a different design. This allows implementing features not technically
+possible in the py-keystone:
 
 - Federation is controlled on the domain level by the domain managers. This
   means that the domain manager is responsible for the configuration of how users
   should be federated from external IdPs.
 
-- Keystone serves as a relying party in the OIDC authentication flow. This
-  moves the complex logic from client to the the Keystone side. This allows
-  making client applications much simpler and more reliable.
-
-## Authentication using the Authorization Code flow and Keystone serving as RP
-
-```mermaid
-sequenceDiagram
-
-    Actor Human
-    Human ->> Cli: Initiate auth
-    Cli ->> Keystone: Fetch the OP auth url
-    Keystone --> Keystone: Initialize authorization request
-    Keystone ->> Cli: Returns authURL of the IdP with cli as redirect_uri
-    Cli ->> User-Agent: Go to authURL
-    User-Agent -->> IdP: opens authURL
-    IdP -->> User-Agent: Ask for consent
-    Human -->> User-Agent: give consent
-    User-Agent -->> IdP: Proceed
-    IdP ->> Cli: callback with Authorization code
-    Cli ->> Keystone: Exchange Authorization code for Keystone token
-    Keystone ->> IdP: Exchange Authorization code for Access token
-    IdP ->> Keystone: Return Access token
-    Keystone ->> Cli: return Keystone token
-    Cli ->> Human: Authorized
-
-```
-
-## Authenticating with the JWT
-
-It is possible to authenticate with the JWT token issued by the federated IdP.
-More precisely it is possible to exchange a valid JWT for the Keystone token.
-There are few different use scenarios that are covered.
-
-Since the JWT was issued without any knowledge of the Keystone scopes it
-becomes hard to control scope. In the case of real human login the Keystone may
-issue unscoped token allowing user to further rescope it. In the case of the
-workflow federation that introduces a potential security vulnerability. As such
-in this scenario the attribute mapping is responsible to fix the scope.
-
-Login request looks following:
-
-```console
-
-  curl https://keystone/v4/federation/identity_providers/${IDP}/jwt -X POST -H "Authorization: bearer ${JWT}" -H "openstack-mapping: ${MAPPING_NAME}"
-```
-
-### Regular user obtains JWT (ID token) at the IdP and presents it to Keystone
-
-In this scenario a real user (human) is obtaining the valid JWT from the IDP
-using any available method without any communication with Keystone. This may
-use authorization code grant, password grant, device grant or any other enabled
-method. This JWT is then presented to the Keystone and an explicitly requested
-attribute mapping converts the JWT claims to the Keystone internal
-representation after verifying the JWT signature, expiration and further
-restricted bound claims.
-
-### Workflow federation
-
-Automated workflows (Zuul job, GitHub workflows, GitLab CI, etc) are typical
-workloads not being bound to any specific user and are more regularly
-considered being triggered by certain services. Such workflows are usually in
-possession of a JWT token issued by the service owned IdP. Keystone allows
-exchange of such tokens to the regular Keystone token after validating token
-issuer signature, expiration and applying the configured attribute mapping.
-Since in such case there is no real human the mapping also need to be
-configured slightly different.
-
-- It is strongly advised the attribute mapping must fill `token_user_id`,
-  `token_project_id` (and soon `token_role_ids`). This allows strong control of
-  which technical account (soon a concept of service accounts will be introduced
-  in Keystone) is being used and which project such request can access.
-
-- Attribute mapping should use `bound_audiences`, `bound_claims`,
-  `bound_subject`, etc to control the tokens issued by which workflows are
-  allowed to access OpenStack resources.
-
-### GitHub workflow federation
-
-In order for the GitHub workflow to be able to access OpenStack resources it is
-necessary to register GitHub as a federated IdP and establish a corresponding
-attribute mapping of the `jwt` type.
-
-IdP:
-
-```json
-"identity_provider": {
-    "name": "github",
-    "bound_issuer": "https://token.actions.githubusercontent.com",
-    "jwks_url": "https://token.actions.githubusercontent.com/.well-known/jwks"
-}
-```
-
-
-Mapping:
-
-```json
-"mapping": {
-   "type": "jwt",
-   "name": "gtema_keystone_main",
-   "idp_id": <IDP_ID>,
-   "domain_id": <DOMAIN_ID>,
-   "bound_audiences": ["https://github.com"],
-   "bound_subject": "repo:gtema/keystone:pull_request",
-   "bound_claims": {
-       "base_ref": "main"
-   },
-   "user_id_claim": "actor_id",
-   "user_name_claim": "actor",
-   "token_user_id": <UID>
-}
-```
-
-TODO: add more claims according to [docs](https://docs.github.com/en/actions/reference/security/oidc#oidc-token-claims)
-
-A way for the workflow to obtain the JWT [is described here](https://docs.github.com/en/actions/reference/security/oidc#methods-for-requesting-the-oidc-token).
-
-```yaml
-...
-permissions:
-  token: write
-  contents: read
+- Identity providers and/or attribute mappings can be reused by different
+  domains allowing implementing social logins.
 
-job:
-  ...
-  - name: Get GitHub JWT token
-    id: get_token
-    run: |
-      TOKEN_JSON=$(curl -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
-      "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=https://github.com")
+- Keystone serves as a relying party in the OIDC authentication flow. It
+  decreases amount of different flows to the minimum making client applications
+  much simpler and more reliable.
 
-      TOKEN=$(echo $TOKEN_JSON | jq -r .value)
-      echo "token=$TOKEN" >> $GITHUB_OUTPUT
-  ...
-  # TODO: build a proper command for capturing the actual token and/or write a dedicated action for that.
-  - name: Exchange GitHub JWT for Keystone token
-    run: |
-      KEYSTONE_TOKEN=$(curl -H "Authorization: bearer ${{ steps.get_token.outputs.token }}" -H "openstack-mapping: gtmema_keystone_main" https://keystone_url/v4/federation/identity_providers/IDP/jwt)
-
-```
 
 ## API changes
 
 A series of brand new API endpoints have been added to the Keystone API.
 
-- /v3/federation/identity_providers (manage the identity providers)
+- **/v4/federation/identity_providers** (manage the identity providers)
 
-- /v3/federation/mappings (manage the mappings tied to the identity provider)
+- **/v4/federation/mappings** (manage the mappings tied to the identity provider)
 
-- /v3/federation/auth (initiate the authentication and get the IdP url)
+- **/v4/federation/auth** (initiate the authentication and get the IdP url)
 
-- /v3/federation/oidc/callback (exchange the authorization code for the Keystone token)
+- **/v4/federation/oidc/callback** (exchange the authorization code for the Keystone token)
 
-- /v3/federation/identity_providers/{idp_id}/jwt (exchange the JWT token issued by the referred IdP for the Keystone token)
+- **/v4/federation/identity_providers/{idp_id}/jwt**
+  (exchange the JWT token issued by the referred IdP for the Keystone token)
 
 ## DB changes
 
@@ -229,3 +96,5 @@ need to rely on Selenium.
 At the moment following integrations are tested automatically:
 
 - Keycloak (login using browser)
+- Keycloak (login with JWT)
+- GitHub (workload federation with JWT)

doc/src/install.md

@@ -1,18 +1,92 @@
 # Installation
 
-TODO:
+The easiest way to get started with the keystone-ng is using the container
+image. It is also possible to use the compiled version. It can be either
+compiled locally or downloaded from the project artifacts.
 
-- Prepare the binary (download from GH releases, build yourself, use the
-  container image, ...)
+## Using pre-compiled binaries
 
-- Perform the DB migration `keystone-db up`
+As of the moment of writing there were no releases. Due to that there are no
+pre-compiled binaries available yet. Every release of the project would include
+the pre-compiled binaries for a variety of platforms.
 
-- Start the binary as `keystone -c <PATH_TO_THE_KEYSTONE_CONFIG>`
+## Compiling
 
+In order to compile the keystone-ng it is necessary to have the rust compiler
+available. It may be installed from the system packages or using the
+`rustup.rs`
+
+```console
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+```
+
+Afterwards in the root of the project source tree following command may be
+executed to invoke the `cargo`
+
+```console
+
+cargo build --release
+
+```
+
+It produces 2 binaries:
+
+- target/release/keystone (the api server)
+
+- target/release/keystone-db (the database management tool)
+
+Currently keystone depends on the openssl (through one of the dependencies).
+Depending on the environment it may be a statically linked or dynamically.
+There are signals that that may be not necessary anymore once all dependencies
+transition to the use of rustls.
+
+## Using containers
+
+It is possible to run Keystone-ng inside containers. A sample Dockerfile is
+present in the project source tree to build container image with the Keystone
+and the `keystone-db` utility. When no ready image is available it can be build
+like that:
+
+```console
+
+docker build . -t keystone:rust
+
+```
+
+Since keystone itself communicates with the database and OpenPolicyAgent those
+must be provided separately. `docker-compose.yaml` demonstrates how this can be
+done.
 
 ## Database migrations
 
 Rust Keystone is using different ORM and implements migration that co-exist
 together with alembic migrations of the python Keystone. It also ONLY manages
 the database schema additions and does NOT include the original database
 schema. Therefore it is necessary to apply both migrations.
+
+```console
+keystone-db -u <DB_URL>
+```
+
+It is important to also understand that the DB_URL may differ between python
+and rust due to the optional presence of the preferred database driver in the
+url. keystone-ng will ignore the the driver in the application itself, but the
+migration may require user to manually remove it since it is being processed by
+the ORM itself and not by the keystone-ng code.
+
+## OpenPolicyAgent
+
+keystone-ng relies on the OPA for policy enforcement. Default policies are
+provided with the project and can be passed directly to the OPA process or
+compilied into the bundle.
+
+```console
+
+opa run -s policies
+
+```
+
+**NOTE:** by default OPA process listens on the localhost only what lead to
+unavailability to expose it between containers. Please use `-a 0.0.0.0:8181` to
+
+start listening on all interfaces.