OAuth2-Proxy doesn't work when entrusting 'Pinniped' as an OIDC provider (for passing Authorization JWT to kubernetes-dashboard application for auto-login)

[am-i-helpful]

What happened?

I have been working on configuring Pinniped for our cloud Kubernetes clusters (Kubeadm based), and the CLI works just fine. I've been able to figure out this with a little bit of help from the LLMs (Gemini), where I identified that the AzureAD/EntraID (OIDC) integration with pinniped-supervisor doesn't work as it is specified in the official docs, with a minor change needed in .spec.authorizationConfig.additionalScopes = [offline_access, email, openid, profile] and .spec.claims.username = preferred_username, but the document has been very helpful otherwise! I am not fully sure if it was my AzureAD directory which was configured like that - anyway, that was fairly easy to resolve, and I am very happy with the pinniped's OIDC-integration functionality (which allows making portable changes, without making any changes on the K8s API-server side).

I was so excited to see how easy it was to configure pinniped with multiple OIDC providers, and the documentation is quite detailed about it. I went ahead and tried to extend the logic, by trying to use pinniped for providing the Authorization bearer token to kubernetes-dashboard v7.*, using the logic detailed in "configuring auth for webapps" docs by Pinniped team. Unfortunately, I have hit a road-blocker due to the challenges related to cookie_refresh and cookie_expire as applicable to pinniped itself while acting as an OIDC provider for webapps (oauth2-proxy acting as an intermediary in-between, while kubernetes-dashboard application is the upstream).

What did you expect to happen?
I was expecting the communication between "oauth2-proxy <-> pinniped (as an OIDC provider)" to be seamless, but that turned out to be not true. I found that there are certain limitations of pinniped itself acting as an OIDC-provider, which resulted in the approach to not work. I am almost sure that I have followed all possible official docs along with various online references and Google Gemini Pro, but it has resulted in me concluding that my setup won't work for kubernetes-dashboard web-application.

EXPECTATION: I expected that after successfully authenticating to Azure OIDC (via pinniped), oauth2-proxy component would pass the authorization token (known to pinniped-concierge) along with user and group details to the kubernetes-dashboard (which will talk to a pinniped-concierge JWTAuthenticator as its api-server, as the former has been configured to impersonate the K8s API-server with mode: 'enabled'), but that didn't happen.
ACTUAL RESULT: This is what I understood from Google Gemini Pro about my issue, when cookie_refresh is configured in oauth2-proxy:

• I couldn't reach that far, as oauth2-proxy kept throwing errors upon attempting to refresh the token with pinniped, by sending the client_id and client_secret inside the POST Body (this is called client_secret_post).
• Pinniped is most likely configured to only accept the client_id and client_secret in the HTTP Basic Auth Header (this is called client_secret_basic).
• Pinniped rejects the request. oauth2-proxy marks the session as dead. The browser gets a 401.

What is the simplest way to reproduce this behavior?

0. For CredentialIssuer 'pinniped-concierge-config', set the impersonationProxy.mode to 'enabled', and impersonationProxy.service.type to 'ClusterIP', so that pinniped-concierge is ready to impersonate the K8s API-server. I performed this manually in my lab cluster, via kubectl edit CredentialIssuer .... Also, create a JWTAuthenticator for kubernetes-dashboard application, as listed below:

apiVersion: authentication.concierge.pinniped.dev/v1alpha1
kind: JWTAuthenticator
metadata:
   name: kubernetes-dashboard-jwt-authenticator
   namespace: pinniped-concierge
spec:
   issuer: "https://pinniped.k8s-auth.example.com"
   audience: client.oauth.pinniped.dev-oauth2-proxy-client # name of the oidc-client to be created in step-1
   claims:
     username: username
     groups: groups

1. Roll out an OIDC client, and create a new application-secret, as detailed in the corresponding official docs.

apiVersion: config.supervisor.pinniped.dev/v1alpha1
kind: OIDCClient
metadata:
  # name must have client.oauth.pinniped.dev- prefix
  name: client.oauth.pinniped.dev-oauth2-proxy-client
  namespace: pinniped-supervisor
spec:
  allowedRedirectURIs:
    - https://k8s-ui-oauth.example.com/oauth2/callback
  allowedGrantTypes:
    - authorization_code
    - refresh_token
    - urn:ietf:params:oauth:grant-type:token-exchange
  allowedScopes:
    - openid
    - offline_access
    - pinniped:request-audience
    - username
    - groups

cat <<EOF | kubectl create -o yaml -f -
apiVersion: clientsecret.supervisor.pinniped.dev/v1alpha1
kind: OIDCClientSecretRequest
metadata:
  name: client.oauth.pinniped.dev-oauth2-proxy-client # the name of the OIDCClient
  namespace: pinniped-supervisor # the namespace of the OIDCClient
spec:
  generateNewSecret: true
EOF

2. Get the CA from Pinniped using the command kubectl get secret -n pinniped-concierge pinniped-concierge-impersonation-proxy-ca-certificate -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/pinniped-concierge-ca.crt.
3. Create a config-map with CA-certificate details from above secret in the kubernetes-dashboard namespace using the command kubectl create configmap pinniped-concierge-ca -n kubernetes-dashboard --from-file=/tmp/pinniped-concierge-ca.crt.
4. Next, roll out kubernetes-dashboard application (helm-chart v7.*) with below helm-chart values.yaml content:

    auth:
      containers:
        ports:
          - name: auth
            containerPort: 8000
            protocol: TCP
        args:
          - --apiserver-host=https://pinniped-concierge-proxy.pinniped-concierge.svc.cluster.local:443
          - --apiserver-skip-tls-verify=true
        volumeMounts:
          - name: pinniped-concierge-ca
            mountPath: /etc/ssl/certs/pinniped-concierge-ca.crt
            subPath: pinniped-concierge-ca.crt
      volumes:
        - name: pinniped-concierge-ca
          configmap:
            name: pinniped-concierge-ca
    api:
      containers:
        ports:
          - name: api
            containerPort: 8000
            protocol: TCP
        args:
          - --apiserver-host=https://pinniped-concierge-proxy.pinniped-concierge.svc.cluster.local:443
          - --apiserver-skip-tls-verify=true
        volumeMounts:
          - name: pinniped-concierge-ca
            mountPath: /etc/ssl/certs/pinniped-concierge-ca.crt
            subPath: pinniped-concierge-ca.crt
      volumes:
        - name: pinniped-concierge-ca
          configmap:
            name: pinniped-concierge-ca

5. Create an ingress for kubernetes-dashboard separately, and not via the helm-chart (it was easier for me to troubleshoot with a dedicated ingress YAML manifest).

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/use-proxy-protocol: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-nginx-prod"
    nginx.ingress.kubernetes.io/whitelist-source-range: 1.2.3.4/32
    nginx.ingress.kubernetes.io/auth-url: "https:/k8s-ui-oauth.example.com/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://k8s-ui-oauth.example.com/oauth2/start?rd=https://$host$escaped_request_uri"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      auth_request_set $auth_cookie $upstream_http_authorization;
      proxy_set_header Authorization $auth_cookie;
      proxy_set_header X-Forwarded-Proto "https";
      proxy_set_header X-Forwarded-Host $host;
    # ID/Access tokens + claims are pushed into headers or cookies, and might exceed NGINX’s default limits
    nginx.ingress.kubernetes.io/client-header-buffer-size: "16k"
    nginx.ingress.kubernetes.io/large-client-header-buffers: "4 16k"
    # Increase buffer size for the response headers from the upstream app
    nginx.ingress.kubernetes.io/auth-proxy-buffer-size: "256k"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "1m"
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
    nginx.ingress.kubernetes.io/proxy-buffering: 'on'
    nginx.ingress.kubernetes.io/proxy-body-size: 2m
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - k8s-ui.example.com
      secretName: k8s-ui-cert
  rules:
  - host: k8s-ui.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard-kong-proxy
            port:
              number: 443

6. Lastly, create a secret 'vault-oauth2-proxy-secret' as described here in oauth2-proxy guide.

client-id = client.oauth.pinniped.dev-oauth2-proxy-client
client-secret = secret value from Step-1: OIDCClientSecretRequest
cookie-secret = retrieve using the command: python -c "import secrets; print(secrets.token_urlsafe(24)[:32])"

• Don't forget to add a Redirect-UI for "https://k8s-ui-oauth.example.com/oauth2/callback" in your AzureAD app-registration, which you might have performed for pinniped-supervisor, while registering pinniped application in AzureAD.
• Afterwards, roll out the oauth2-proxy (helm-chart v8.5.x) with below helm-chart values.yaml manifest, using a separate ingress for oauth2-proxy (because, it is easier to manage multiple ingresses, each intended for different sources). I felt that it was important to do this as per my approach.

    revisionHistoryLimit: 3
    config:
      existingSecret: vault-oauth2-proxy-secret # name of the secret created above
      # configuration reference: https://oauth2-proxy.github.io/oauth2-proxy/configuration/overview/
      configFile: |-
        provider = "oidc"
        provider_display_name = "Pinniped-Supervisor (Azure EntraID)"
        oidc_issuer_url = "https://pinniped.k8s-auth.example.com"
        code_challenge_method = "S256"
        scope = "openid offline_access username groups"
        upstreams = [ "https://kubernetes-dashboard-kong-proxy.kubernetes-dashboard:443" ]
        redirect_url = "https://k8s-ui-oauth.example.com/oauth2/callback"
        oidc_email_claim = "username"
        oidc_groups_claim = "groups"
        reverse_proxy = true
        email_domains = [ "*" ]
        whitelist_domains = "*.example.com"
        cookie_domains = ".example.com"
        cookie_samesite = "lax"
        cookie_refresh = 600 # throws error oauth2: "invalid_client" "Client authentication failed ... The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested.
#        cookie_expire = 86400 # can't set 'cookie_expire' with pinniped as OIDC currently, otherwise throws error "CSRF cookie with name '_oauth2_proxy_csrf' was not found"
        set_authorization_header = true
        pass_access_token = true
        pass_basic_auth = true
        encode_state = true
        set_xauthrequest = true
        cookie_secure = true
        http_address = "0.0.0.0:4180"
        silence_ping_logging = true
    sessionStorage:
      type: cookie
    redis:
      enabled: false
    service:
      type: ClusterIP
      portNumber: 4180
      appProtocol: http
    ingress:
      enabled: true
      className: nginx
      path: /
      pathType: ImplementationSpecific
      hosts:
        - k8s-ui-oauth.example.com
      annotations:
        nginx.ingress.kubernetes.io/backend-protocol: HTTP
        nginx.ingress.kubernetes.io/use-proxy-protocol: "true"
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
        cert-manager.io/cluster-issuer: "letsencrypt-nginx-prod"
        nginx.ingress.kubernetes.io/whitelist-source-range: 1.2.3.4/32
      tls:
        - secretName: k8s-ui-oauth-tls-cert
          hosts:
            - k8s-ui-oauth.example.com

7. Now, when I browse the URL/ingress of kubernetes-dashboard k8s-ui.example.com from Firefox web-browser in an incognito session (with no existing cookies saved for the domain .example.com): I am able to login using the pinniped-supervisor and then to AzureAD, but I receive the error-page with 500-error without auto-login to the kubernetes-dashboard, I receive error while in the oauth2-proxy pod log, when I enable either cookie_refresh = 600 ("oauth2: "invalid_client" "Client authentication failed...The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested") or cookie_expire = 86400 ("Invalid authentication via OAuth2: unable to obtain CSRF cookie: %s (state=%s) CSRF cookie with name '_oauth2_proxy_csrf' was not found").

Below is the error reference when I only have cookie_refresh = 600 enabled (and cookie-expire commented out) in the oauth2-proxy configuration, when the '_oauth2_proxy_csrf' token gets created successfully. Whereas, for cookie_expire = 86400 enabled (and cookie-refresh commented out), it complains that the required CSRF-token doesn't exist, which is also an error. In both cases, I am not able to get to the kubernetes-dashboard auto-login view.

In what environment did you see this bug?

• Pinniped server version: 0.42.0 (installed via YAML manifest)
• Pinniped client version: v0.42.0
• Pinniped container image (if using a public container image):
• Pinniped configuration (what IDP(s) are you using? what downstream credential minting mechanisms are you using?): AzureAD as an OIDC provider for pinniped, but oauth2-proxy indirectly authenticates to AzureAD directory, via pinniped working as an OIDC-provider by itself
• Kubernetes version (use kubectl version): 1.32
• Kubernetes installer & version (e.g., kubeadm version): 1.32
• Cloud provider or hardware configuration: Kubeadm (on Debian)
• OS (e.g: cat /etc/os-release): N/A
• Kernel (e.g. uname -a): N/A
• Others: N/A

What else is there to know about this bug?
Though I am not an expert in the field of identity management, I think that it is a limitation of Pinniped such that it can't be used as an OIDC-provider by itself for other upstream applications (like kubernetes-dashboard). From what I understood after searching online and asking Google Gemini Pro, I think the limitations are being hit due to the limited functionality of pinniped (current version v0.42) to act as an OIDC-provider by itself. Below are the logs showing in detail about the error for both cases:

• enabling 'cookie_expire = 86400' -- could be any value - as that doesn't matter

2025-12-27T20:51:05.865747658Z [2025/12/27 20:51:05] [provider.go:55] Performing OIDC Discovery...
2025-12-27T20:51:06.190207998Z [2025/12/27 20:51:06] [proxy.go:89] mapping path "/" => upstream "..."
2025-12-27T20:52:08.654475735Z [2025/12/27 20:52:08] [oauthproxy.go:1328] &{GET /oauth2/callback?code=pin_ac_t__mc4
...
0xc0002dd980 [] map[]} AuthFailure CSRF cookie %s was found in OAuth callback. _oauth2_proxy_csrf
2025-12-27T20:52:08.654553764Z
2025-12-27T20:52:08.654613187Z [2025/12/27 20:52:08] [oauthproxy.go:891] &{GET /oauth2/callback?code=pin_a
...
z73RbVN93mnLjo4bAubDay0libEP0xKOxqOVKs0fk|1766574430|u0q8EWFaIXQNRA-nBHegG8npJSvCUL84HWkD8aEGp7w=; _oauth2_proxy_csrf=kDpH12345kRohBjP4tGKj1d6-yY_IRDo51ovgM=] Dnt:[1] Sec-Ch-Ua:["Google Chrome";v="143", "Chromium";v="143", "Not A(Brand";v="24"] Sec-Ch-Ua-Mobile:[?0] Sec-Ch-Ua-Platform:["macOS"] Sec-Fetch-Dest:[document] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[cross-site] Sec-Fetch-User:[?1] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36] X-Forwarded-For:[1.2.3.4] X-Forwarded-For-Proxy-Protocol:[1.2.3.4] X-Forwarded-Host:[k8s-ui-oauth.example.com] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Scheme:[https] X-Real-Ip:[1.2.3.4] X-Request-Id:[4aca67125eff4] X-Scheme:[https]] {} 0 [] false k8s-ui-oauth.example.com map[code:[pin_ac_t__mc4kgEg3EJ1t4cjL7w3j2MKV76432kZfp3sbs] scope:[openid offline_access username groups] state:[M0NGMEtYdWttcUc0MbC8]] map[] map[] 192.16812rgyg232gEg3EJ1t4cjL7w3j2MKVM19Ya7lRV1baGKTk.Noxz9Mr6wXqcCT5gZddwWNX9Hg1I5hgw8IokZfp3sbs&scope=openid+offline_access+username+groups&state=M0N123VZ3BZNUwwWXh0cjZXTTpodHRwczovL3BsYXlncm91bmQuazhD5428 0xc0002dd980 [] map[]} AuthFailure Invalid authentication via OAuth2: unable to obtain CSRF cookie: %s (state=%s) CSRF cookie with name '_oauth2_proxy_csrf' was not found 3CF0KX6WM
2025-12-27T20:52:08.654672651Z
2025-12-27T20:52:08.654989175Z 192.168.2.87:57452 - 4aca67a75f0ec30dd562e439b295eff4 - - [2025/12/27 20:52:08] k8s-ui-oauth.example.com GET - "/oauth2/callback?code=pin_ac_t__mc4kgEg3EJ1t4cj12546sdffp3sbs&scope=openid+offline_access+username+groups&state=M0LXVpLmdlY2tvYXBwcy5ubC8" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36" 403 2727 0.001
2025-12-27T20:52:08.769392156Z 192.168.2.87:57452 - bdd470cbfbccfcd79185b325281fd95f - - [2025/12/27 20:52:08] k8s-ui-oauth.example.com GET - "/oauth2/static/css/bulma.min.css" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36" 200 207302 0.033
2025-12-27T20:52:08.807676332Z 192.168.2.87:57452 - 5e8286e9eb547e8158d8128ba35ef4bb - - [2025/12/27 20:52:08] k8s-ui-oauth.example.com GET - "/oauth2/static/css/all.min.css" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36" 200 102025 0.000
2025-12-27T20:52:08.940135118Z 192.168.2.87:57452 - 4728bb5f93e716bb3f427edf307970c5 - - [2025/12/27 20:52:08] k8s-ui-oauth.example.com GET - "/oauth2/static/webfonts/fa-solid-900.woff2" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36" 200 149908 0.000
2025-12-27T20:52:09.090101364Z [2025/12/27 20:52:09] [oauthproxy.go:1034] No valid authentication in request. Initiating login.
2025-12-27T20:52:09.090111474Z [2025/12/27 20:52:09] [stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session
2025-12-27T20:52:09.090924625Z 192.168.2.87:57452 - 3480ecd05eecf69b2ce6a8d5f7aa7e87 - - [2025/12/27 20:52:09] k8s-ui-oauth.example.com GET - "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36" 403 8526 0.001

• enabling 'cookie_refresh = 600' -- could be any value - as that doesn't matter (below, same error repeated 3 times, for different URI paths)

2025-12-28T22:33:20.978638182Z 1.2.3.4 - 6263358f276b00fe11ce0944b220fb27 - my-email@company.com [2025/12/28 22:33:20] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 0.000
2025-12-28T22:33:21.909594222Z [2025/12/28 22:33:21] [stored_session.go:189] Refreshing session - User: https://login.microsoftonline.com/e871219-3123-4123-a312-90f431de12/v2.0?idpName=Microsoft+EntraID+Authentication&sub=IQgiv_Y4Wx28kdulrNdhtk3KTAq4yY4egV3ONB88YbU; SessionAge: 417.408896ms
2025-12-28T22:33:22.436959037Z 1.2.3.4 - 08c616b72583d1e660d338c186ddfcbd - my-email@company.com [2025/12/28 22:33:21] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 0.528
2025-12-28T22:33:23.269297664Z [2025/12/28 22:33:23] [stored_session.go:189] Refreshing session - User: https://login.microsoftonline.com/e871219-3123-4123-a312-90f431de12/v2.0?idpName=Microsoft+EntraID+Authentication&sub=IQgiv_Y4Wx28kdulrNdhtk3KTAq4yY4egV3ONB88YbU; SessionAge: 2.417408896s
2025-12-28T22:33:23.276016398Z [2025/12/28 22:33:23] [stored_session.go:189] Refreshing session - User: https://login.microsoftonline.com/e871219-3123-4123-a312-90f431de12/v2.0?idpName=Microsoft+EntraID+Authentication&sub=IQgiv_Y4Wx28kdulrNdhtk3KTAq4yY4egV3ONB88YbU; SessionAge: 2.417408896s
2025-12-28T22:33:23.279775834Z [2025/12/28 22:33:23] [stored_session.go:189] Refreshing session - User: https://login.microsoftonline.com/e871219-3123-4123-a312-90f431de12/v2.0?idpName=Microsoft+EntraID+Authentication&sub=IQgiv_Y4Wx28kdulrNdhtk3KTAq4yY4egV3ONB88YbU; SessionAge: 2.417408896s
2025-12-28T22:33:23.322587866Z [2025/12/28 22:33:23] [stored_session.go:189] Refreshing session - User: https://login.microsoftonline.com/e871219-3123-4123-a312-90f431de12/v2.0?idpName=Microsoft+EntraID+Authentication&sub=IQgiv_Y4Wx28kdulrNdhtk3KTAq4yY4egV3ONB88YbU; SessionAge: 2.417408896s
2025-12-28T22:33:24.212159899Z [2025/12/28 22:33:24] [stored_session.go:193] Unable to refresh session: error refreshing tokens: unable to redeem refresh token: failed to get token: oauth2: "invalid_client" "Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method). The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested. You must configure the OAuth 2.0 client's 'token_endpoint_auth_method' value to accept 'client_secret_post'."
2025-12-28T22:33:24.212758361Z 1.2.3.4 - 43e4fd36bcf6e453690219f33662a2a4 - my-email@company.com [2025/12/28 22:33:23] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 0.890
2025-12-28T22:33:24.273239157Z [2025/12/28 22:33:24] [stored_session.go:193] Unable to refresh session: error refreshing tokens: unable to redeem refresh token: failed to get token: oauth2: "invalid_client" "Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method). The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested. You must configure the OAuth 2.0 client's 'token_endpoint_auth_method' value to accept 'client_secret_post'."
2025-12-28T22:33:24.273875192Z 1.2.3.4 - 043d8b1711a6efe3501854421681bfc3 - my-email@company.com [2025/12/28 22:33:23] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 0.998
2025-12-28T22:33:24.308442848Z [2025/12/28 22:33:24] [stored_session.go:193] Unable to refresh session: error refreshing tokens: unable to redeem refresh token: failed to get token: oauth2: "invalid_client" "Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method). The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested. You must configure the OAuth 2.0 client's 'token_endpoint_auth_method' value to accept 'client_secret_post'."
2025-12-28T22:33:24.308790502Z [2025/12/28 22:33:24] [stored_session.go:193] Unable to refresh session: error refreshing tokens: unable to redeem refresh token: failed to get token: oauth2: "invalid_client" "Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method). The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested. You must configure the OAuth 2.0 client's 'token_endpoint_auth_method' value to accept 'client_secret_post'."
2025-12-28T22:33:24.309229260Z 1.2.3.4 - 46ce00a33b22adcd40ac2b690d8dfa26 - my-email@company.com [2025/12/28 22:33:23] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 1.040
2025-12-28T22:33:24.309256301Z 1.2.3.4 - caf125d70dfcf95cd2534150168ada36 - my-email@company.com [2025/12/28 22:33:23] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 1.030
2025-12-28T22:33:24.377168039Z [2025/12/28 22:33:24] [stored_session.go:189] Refreshing session - User: https://login.microsoftonline.com/e871219-3123-4123-a312-90f431de12/v2.0?idpName=Microsoft+EntraID+Authentication&sub=IQgiv_Y4Wx28kdulrNdhtk3KTAq4yY4egV3ONB88YbU; SessionAge: 3.417408896s
2025-12-28T22:33:24.716464881Z [2025/12/28 22:33:24] [stored_session.go:193] Unable to refresh session: error refreshing tokens: unable to redeem refresh token: failed to get token: oauth2: "invalid_client" "Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method). The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested. You must configure the OAuth 2.0 client's 'token_endpoint_auth_method' value to accept 'client_secret_post'."
2025-12-28T22:33:24.717123308Z 1.2.3.4 - fafa10c76e8b67c790d300a2bae59ba4 - my-email@company.com [2025/12/28 22:33:24] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 0.340
2025-12-28T22:33:24.880143173Z [2025/12/28 22:33:24] [stored_session.go:189] Refreshing session - User: https://login.microsoftonline.com/e871219-3123-4123-a312-90f431de12/v2.0?idpName=Microsoft+EntraID+Authentication&sub=IQgiv_Y4Wx28kdulrNdhtk3KTAq4yY4egV3ONB88YbU; SessionAge: 3.417408896s
2025-12-28T22:33:24.884319915Z [2025/12/28 22:33:24] [stored_session.go:189] Refreshing session - User: https://login.microsoftonline.com/e871219-3123-4123-a312-90f431de12/v2.0?idpName=Microsoft+EntraID+Authentication&sub=IQgiv_Y4Wx28kdulrNdhtk3KTAq4yY4egV3ONB88YbU; SessionAge: 3.417408896s
2025-12-28T22:33:25.217368711Z [2025/12/28 22:33:25] [stored_session.go:193] Unable to refresh session: error refreshing tokens: unable to redeem refresh token: failed to get token: oauth2: "invalid_client" "Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method). The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested. You must configure the OAuth 2.0 client's 'token_endpoint_auth_method' value to accept 'client_secret_post'."
2025-12-28T22:33:25.217923641Z 1.2.3.4 - e327711b3c11928dd55ccbc48dca05b8 - my-email@company.com [2025/12/28 22:33:24] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 0.338
2025-12-28T22:33:25.222126712Z [2025/12/28 22:33:25] [stored_session.go:193] Unable to refresh session: error refreshing tokens: unable to redeem refresh token: failed to get token: oauth2: "invalid_client" "Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method). The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested. You must configure the OAuth 2.0 client's 'token_endpoint_auth_method' value to accept 'client_secret_post'."
2025-12-28T22:33:25.222639532Z 1.2.3.4 - 0329d7daa08d65eb5b15e0d2a44e575e - my-email@company.com [2025/12/28 22:33:24] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 0.339
2025-12-28T22:33:25.349101531Z [2025/12/28 22:33:25] [stored_session.go:189] Refreshing session - User: https://login.microsoftonline.com/e871219-3123-4123-a312-90f431de12/v2.0?idpName=Microsoft+EntraID+Authentication&sub=IQgiv_Y4Wx28kdulrNdhtk3KTAq4yY4egV3ONB88YbU; SessionAge: 4.417408896s
2025-12-28T22:33:25.351101668Z [2025/12/28 22:33:25] [stored_session.go:189] Refreshing session - User: https://login.microsoftonline.com/e871219-3123-4123-a312-90f431de12/v2.0?idpName=Microsoft+EntraID+Authentication&sub=IQgiv_Y4Wx28kdulrNdhtk3KTAq4yY4egV3ONB88YbU; SessionAge: 4.417408896s
2025-12-28T22:33:25.675006523Z [2025/12/28 22:33:25] [stored_session.go:193] Unable to refresh session: error refreshing tokens: unable to redeem refresh token: failed to get token: oauth2: "invalid_client" "Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method). The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested. You must configure the OAuth 2.0 client's 'token_endpoint_auth_method' value to accept 'client_secret_post'."
2025-12-28T22:33:25.675248915Z 1.2.3.4 - 8fad23e95b893e83a8b77c77790d41e1 - my-email@company.com [2025/12/28 22:33:25] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 0.326
2025-12-28T22:33:25.703590990Z [2025/12/28 22:33:25] [stored_session.go:193] Unable to refresh session: error refreshing tokens: unable to redeem refresh token: failed to get token: oauth2: "invalid_client" "Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method). The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested. You must configure the OAuth 2.0 client's 'token_endpoint_auth_method' value to accept 'client_secret_post'."
2025-12-28T22:33:25.703806691Z 1.2.3.4 - 9092e02bec0b9cc7fda66a67c2bb69b5 - my-email@company.com [2025/12/28 22:33:25] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 0.354
2025-12-28T22:33:53.482568004Z [2025/12/28 22:33:53] [stored_session.go:189] Refreshing session - User: https://login.microsoftonline.com/e871219-3123-4123-a312-90f431de12/v2.0?idpName=Microsoft+EntraID+Authentication&sub=IQgiv_Y4Wx28kdulrNdhtk3KTAq4yY4egV3ONB88YbU; SessionAge: 32.417408896s
2025-12-28T22:33:53.854379382Z [2025/12/28 22:33:53] [stored_session.go:193] Unable to refresh session: error refreshing tokens: unable to redeem refresh token: failed to get token: oauth2: "invalid_client" "Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method). The OAuth 2.0 Client supports client authentication method 'client_secret_basic', but method 'client_secret_post' was requested. You must configure the OAuth 2.0 client's 'token_endpoint_auth_method' value to accept 'client_secret_post'."
2025-12-28T22:33:53.854674746Z 1.2.3.4 - 94425b342917af5667b874661351e588 - my-email@company.com [2025/12/28 22:33:53] k8s-ui-oauth.example.com GET - "/oauth2/auth" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0" 202 0 0.372

I spent quite some days to troubleshoot this thoroughly, and I feel disappointed that I was so close to making this work, but was out of luck due to current limitations of Pinniped (from what I understand from the oauth2-proxy pod logs, IMHO). I would really appreciate any helpful pointers or suggestions, or if anyone can point if there has been any mistake. I've provided this post in quite detailed manner, such that it might be of some help to future visitors. Thanks in advance!

[am-i-helpful]

If it could be of any help, my .well-known/openid-configuration looks like this:

{
  "issuer": "https://k8s-auth.example.com",
  "authorization_endpoint": "https://k8s-auth.example.com/oauth2/authorize",
  "token_endpoint": "https://k8s-auth.example.com/oauth2/token",
  "jwks_uri": "https://k8s-auth.example.com/jwks.json",
  "response_types_supported": [
    "code"
  ],
  "response_modes_supported": [
    "query",
    "form_post"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "ES256"
  ],
  "token_endpoint_auth_methods_supported": [
    "client_secret_basic"
  ],
  "scopes_supported": [
    "openid",
    "offline_access",
    "pinniped:request-audience",
    "username",
    "groups"
  ],
  "claims_supported": [
    "username",
    "groups",
    "additionalClaims"
  ],
  "code_challenge_methods_supported": [
    "S256"
  ],
  "discovery.supervisor.pinniped.dev/v1alpha1": {
    "pinniped_identity_providers_endpoint": "https://k8s-auth.example.com/v1alpha1/pinniped_identity_providers"
  }
}

[cfryanr]

Hi @am-i-helpful. Thanks for providing all the details in your question.

Did you have a chance to review this document? https://pinniped.dev/docs/howto/configure-auth-for-webapps. As mentioned in that doc, the Pinniped Supervisor does not support client authentication via client_secret_post. Perhaps there is a way to ask oauth2-proxy to use client_secret_basic?

Also note that while you can use the Pinniped Supervisor as an OIDC provider for login to a web app, it does not allow the web app to automatically make calls to the k8s APIs using the identity of the logged in user, unless the web app is also willing to make some additional custom APIs calls as described in the section "How a web application can perform actions as the authenticated user on Kubernetes clusters" in the doc https://pinniped.dev/docs/howto/configure-auth-for-webapps.

[am-i-helpful]

Hi @cfryanr,
Thanks for your response!

I have checked that document thoroughly while working on the integration, and I also found that currently there is an open issue with the oauth2-proxy: oauth2-proxy/oauth2-proxy#2909. Unfortunately, there is not much I can do at this moment. I don't want to create an own patch, or add extra reverse-proxy layer before oauth2-proxy to make it work (though, that might address the concern).

Thankfully, I have one pinniped supervisor+concierge setup for my TEST K8s-cluster (not rolled out the whole setup in other environments yet), so I think the auth-configuration section is quite simplified with respect to my scenario. I just want to keep the authentication process simple for the end users:
end-user (login request) -> accesses kubernetes-dashboard ingress URL -> oauth2-proxy redirect -> pinniped-supervisor (as OIDC provider) -> AzureAD authentication -> pinniped -> oauth2-proxy (pass Authorization Bearer token)-> kubernetes-dashboard app -> the dashboard app talks to K8s API server via impersonation proxy mode enabled in pinniped-concierge (via helm configuration of kubernetes-dashboard app, which is set to point the K8s API-server value to the pinniped-concierge-proxy service).

I am not sure if kubernetes-dashboard application would work as expected afterwards though, but unfortunately, I can't proceed further due to the current state. I guess I will live with it for now, and suggest end-users needing GUI support to use applications like 'Headlamp' (which works flawlessly with the Pinniped-CLI setup), instead of the official kubernetes-dashboard app. Appreciate your helpful response!