mlx ui pod: "EACCES: permission denied, mkdir '/workspace/build'"

[jbusche]

Describe the bug

Trying the MLX install on my openshift cluster this am, so the steps are pretty easy

1. Have an existing cluster and be logged in, etc.
2. make sure kustomize is 3.2.0, looks good
3. clone the manifests and switch to the mlx-single-user branch
4. Run:
while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

I suspect my cluster needs more configuration though, after step 1, because I've got what looks like pvc permission issues, for example:

oc get pods -n kubeflow |grep ui
ml-pipeline-ui-9d7d4855d-9m78b                    1/1     Running             0          18m
mlx-ui-c47bd9f9c-qgvf6                            0/1     CrashLoopBackOff    7          18m

oc logs -f mlx-ui-c47bd9f9c-qgvf6
> aiops-frontend@0.1.0 build /workspace
> react-scripts build

Failed to compile.

EACCES: permission denied, mkdir '/workspace/build'

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! aiops-frontend@0.1.0 build: `react-scripts build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the aiops-frontend@0.1.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

To Reproduce

See above

Expected behavior

Was expecting the mlx-ui pod to start successfully.

Screenshots

n/a

Environment

• OS: Deploying from a Mac while oc logged into the OpenShift cluster
• Browser: chrome
• MLX Version: If the UI was up, maybe I could see the version...
• OpenShift Version 4.8.39

Additional context

None so far, thank you!

[ckadner]

Thank you @jbusche

I think @srmeier had come across this a few weeks ago and overcame the issue by adding the following to the mlx-ui Dockerfile:

RUN chgrp -R 0 /workspace && chmod -R g=u /workspace

Just doing a quick Google check on this. It seems the permissions for users running in Docker containers are at the core of this issue along with some changes in npm. There may need to be more thought given to a proper solution, like running as root or unprivileged user etc, … i.e. https://stackoverflow.com/a/45505787

[jbusche]

Noticing a problem with non-root user when deploying mlx:

oc logs -f mlx-ui-b58bb8d4c-s659j -p

> aiops-frontend@0.1.0 build /workspace
> react-scripts build

Creating an optimized production build...
Compiled successfully.

File sizes after gzip:

  346.48 KB  build/static/js/2.29dd04b2.chunk.js
  56.15 KB   build/static/js/main.16e6609d.chunk.js
  3.84 KB    build/static/css/main.265e3da4.chunk.css
  2.72 KB    build/static/css/2.21151bd0.chunk.css
  780 B      build/static/js/runtime-main.b8b6e910.js

The project was built assuming it is hosted at ./.
You can control this with the homepage field in your package.json.

The build folder is ready to be deployed.

Find out more about deployment here:

  bit.ly/CRA-deploy


┌──────────────────────────────────────────────┐
│           npm update check failed            │
│     Try running with sudo or get access      │
│     to the local update config store via     │
│ sudo chown -R $USER:$(id -gn $USER) /.config │
└──────────────────────────────────────────────┘
/workspace/server/node_modules/fs-extra/lib/mkdirs/mkdirs-sync.js:45
        throw err0
        ^

Error: EACCES: permission denied, mkdir '/workspace/sessions'
    at Object.mkdirSync (fs.js:1014:3)
    at Object.mkdirsSync (/workspace/server/node_modules/fs-extra/lib/mkdirs/mkdirs-sync.js:31:9)
    at new FileStore (/workspace/server/node_modules/session-file-store/lib/session-file-store.js:29:8)
    at initLogin (/workspace/server/dist/server.js:110:16)
    at Object.<anonymous> (/workspace/server/dist/server.js:34:5)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12) {
  errno: -13,
  syscall: 'mkdir',
  code: 'EACCES',
  path: '/workspace/sessions'
}

# oc rsh mlx-ui-b58bb8d4c-s659j
$ whoami
1000770000

oc get pods
NAME                                               READY   STATUS              RESTARTS   AGE
cache-deployer-deployment-798dc7d98b-9qtgm         1/1     Running             0          5m37s
cache-server-86f59c8696-44glh                      1/1     Running             0          5m37s
kfp-csi-s3-5btss                                   0/2     ContainerCreating   0          5m36s
kfp-csi-s3-gtw4t                                   0/2     ContainerCreating   0          5m36s
kfp-csi-s3-npj4x                                   0/2     ContainerCreating   0          5m36s
metadata-envoy-deployment-5b4856dd5-tn6jg          1/1     Running             0          5m37s
metadata-grpc-deployment-6b5685488-42zc4           1/1     Running             2          5m37s
metadata-writer-9f698fdcb-fz24w                    1/1     Running             0          5m37s
minio-5b65df66c9-7gkng                             1/1     Running             0          5m37s
ml-pipeline-77b7b79565-8jvxr                       1/1     Running             1          5m37s
ml-pipeline-persistenceagent-684f664fb7-cjw24      1/1     Running             0          5m37s
ml-pipeline-scheduledworkflow-5dfcf96788-2tzr5     1/1     Running             0          5m37s
ml-pipeline-ui-6dfcc5c664-sr6lb                    1/1     Running             0          5m37s
ml-pipeline-viewer-crd-5878c6454f-7r58q            1/1     Running             0          5m37s
ml-pipeline-visualizationserver-6876996cdd-sl5fq   1/1     Running             0          5m37s
mlx-api-7f46b6df4f-48ccr                           1/1     Running             0          5m37s
mlx-ui-b58bb8d4c-s659j                             0/1     CrashLoopBackOff    4          5m37s
mysql-f7b9b7dd4-v2dz9                              1/1     Running             0          5m37s

And then when I deleted the pod, it restarted as root user:

oc delete pod mlx-ui-b58bb8d4c-s659j
pod "mlx-ui-b58bb8d4c-s659j" deleted

oc get pods
NAME                                               READY   STATUS              RESTARTS   AGE
cache-deployer-deployment-798dc7d98b-9qtgm         1/1     Running             0          5m59s
cache-server-86f59c8696-44glh                      1/1     Running             0          5m59s
kfp-csi-s3-5btss                                   0/2     ContainerCreating   0          5m58s
kfp-csi-s3-gtw4t                                   0/2     ContainerCreating   0          5m58s
kfp-csi-s3-npj4x                                   0/2     ContainerCreating   0          5m58s
metadata-envoy-deployment-5b4856dd5-tn6jg          1/1     Running             0          5m59s
metadata-grpc-deployment-6b5685488-42zc4           1/1     Running             2          5m59s
metadata-writer-9f698fdcb-fz24w                    1/1     Running             0          5m59s
minio-5b65df66c9-7gkng                             1/1     Running             0          5m59s
ml-pipeline-77b7b79565-8jvxr                       1/1     Running             1          5m59s
ml-pipeline-persistenceagent-684f664fb7-cjw24      1/1     Running             0          5m59s
ml-pipeline-scheduledworkflow-5dfcf96788-2tzr5     1/1     Running             0          5m59s
ml-pipeline-ui-6dfcc5c664-sr6lb                    1/1     Running             0          5m59s
ml-pipeline-viewer-crd-5878c6454f-7r58q            1/1     Running             0          5m59s
ml-pipeline-visualizationserver-6876996cdd-sl5fq   1/1     Running             0          5m59s
mlx-api-7f46b6df4f-48ccr                           1/1     Running             0          5m59s
mlx-ui-b58bb8d4c-fdw8l                             1/1     Running             0          13s
mysql-f7b9b7dd4-v2dz9                              1/1     Running             0          5m59s

 oc rsh mlx-ui-b58bb8d4c-fdw8l
# exit
#### NOTE that it's root user above. (Not a "$" prompt). #####

# oc logs -f mlx-ui-b58bb8d4c-fdw8l
> aiops-frontend@0.1.0 build /workspace
> react-scripts build

Creating an optimized production build...
Compiled successfully.

File sizes after gzip:

  346.48 KB  build/static/js/2.29dd04b2.chunk.js
  56.15 KB   build/static/js/main.16e6609d.chunk.js
  3.84 KB    build/static/css/main.265e3da4.chunk.css
  2.72 KB    build/static/css/2.21151bd0.chunk.css
  780 B      build/static/js/runtime-main.b8b6e910.js

The project was built assuming it is hosted at ./.
You can control this with the homepage field in your package.json.

The build folder is ready to be deployed.

Find out more about deployment here:

  bit.ly/CRA-deploy

[HPM] Proxy created: /  ->  http://mlx-api
Server listening at http://localhost:3000

[ckadner]

There was some good advice on resolving this issue here: nodejs/docker-node#740 (comment)