Support running jobs in jupyterhub namespace

[Adam-D-Lewis]

@TomAugspurger Currently all jobs are created in the jupyter user's username namespace. I assume kbatch is meant to be used only when enable_user_namespaces is enabled. Are there any thoughts on allowing jobs to be created in the same namespace as jupyterhub for those disabling enable_user_namespaces?

Also, I'd be happy to work on the PR for this, and I can propose implementation details based on your initial feedback. As mentioned in the Pangeo meeting, we're hoping to also add the ability to automatically mount the user pod's volumes to the job pod, but this becomes more difficult if the user's pod and the job pod are deployed in different namespaces.

[TomAugspurger]

I assume kbatch is meant to be used only when enable_user_namespaces is enabled.

On our hub at https://planetarycomputer.microsoft.com/compute, we actually do have kbatch deployed, but aren't using user namespaces (yet).

As you've noticed, running jobs in a separate namespace precludes the ability to mount the user's home directory.

---

As for whether to allow this, I'll defer to you / @yuvipanda / other folks more knowledgeable about Kubernetes for whether that's feasible. Thus far, I've operated under the assumption that we can achieve some level of security and convenience by allowing the user to do anything within their Kubernetes namespace, but nothing outside of it. This has kept kbatch relatively simple, and we're hoping to simplify it further by making the kbatch-proxy component a true proxy and minimizing the amount of work it does (#14).

If we allow spawning Job pods within the "main" Hub namespace, we'll need some more complicated authentication / authorization mechanism (I think).

[Adam-D-Lewis]

Thanks for the response @TomAugspurger. I'd love to hear @yuvipanda 's thoughts as well. 👍

[dharhas]

ping @yuvipanda

Any thoughts on this. Longer term, we would love to start using user namespaces but do we have any options in the interim.