capture 403 error in ada --stage

[natalieda]

Capture this error: 403 Error: Forbidden
Either you don't have access to this directory or the operation for recursive staging in subdirectories is not permitted.

[hailihu]

We cannot determine source of error, but give this as info to the user and suggest possible troubleshoot steps (see above)

[hailihu]

I cannot reproduce this error. Will ask @natalieda an example command that triggers this error.

[onnozweers]

I understand it now.

In our dCache API, we have allowed directory expansion:

bulk.allowed-directory-expansion=ALL

This allows recursive staging.

The default value is TARGETS, which means you can stage the files in a directory, but not in its subdirectories (with ada --recursive).

#  ---- Global setting for directory expansion.
#
#           NONE:       do not allow any processing of targets inside a directory (depth 0)
#           TARGETS:    allow only the immediate children of directories to be processed (depth 1)
#           ALL:        allow full recursive processing of directories (depth N)
#
(one-of?NONE|TARGETS|ALL)bulk.allowed-directory-expansion=TARGETS

For now, I commented out this setting on our test server, to have the default of TARGET.

Then staging a dir without recursion still succeeds:

[onno@mylaptop ~/git/SpiderScripts] % ada/ada --stage /users/onno/tape/test2
["//users/onno/tape/test2"]  
Date: Fri, 05 Sep 2025 07:30:00 GMT
request-url: https://dcachetest.grid.surfsara.nl:20443/api/v1/bulk-requests/a8b7bcce-5ac5-4fc8-88d7-ad0360c94d61

But staging a dir recursively fails:

[onno@mylaptop ~/git/SpiderScripts] % ada/ada --stage /users/onno/tape/test2 --recursive
["//users/onno/tape/test2"]  
curl: (22) The requested URL returned error: 403
Date: Fri, 05 Sep 2025 07:31:27 GMT

As this is the default setting in dCache, it will be very helpful if Ada warns a user about it. The message to the user could be something like:

"Operation failed. You may not have permission to the directory, or recursive staging may be prohibited. Try the same command without --recursive. If that works, the dCache system does not allow you to stage recursively. If you need this, ask your dCache admins to set 'bulk.allowed-directory-expansion=ALL'."

It would be nice if dCache differentiated between "permission denied" and "recursion not allowed". I looked for another 4xx HTTP status code than 403 at https://en.wikipedia.org/wiki/List_of_HTTP_status_codes . The best I could find is this:

422 Unprocessable Content
The request was well-formed (i.e., syntactically correct) but could not be processed.[1]: §15.5.21 

[onnozweers]

I suggested to return 422 when recursion is prohibited.
dCache/dcache#7892

Meanwhile, we can catch this situation:

• Ada request with --recursion resulting in a JSON with "expand_directories": "ALL"
• Curl fails with a 403 or 422 (assuming that one day a 422 may be returned)

[hailihu]

Interestingly, recursive staging is allowed on the dCache production instance but not on the test instance. That's why I could not reproduce the error at first when I was trying on production. Is this a change in the settings between production and test, or in dCache version?

[hailihu]

Interestingly, recursive staging is allowed on the dCache production instance but not on the test instance. That's why I could not reproduce the error at first when I was trying on production. Is this a change in the settings between production and test, or in dCache version?

Reading the above better now, I indeed see recursive staging is allowed on production and not on test, good to know!