This is a note we sat on for too long. The approach has been in production since 2019, but it only ever lived in our heads. Since then the component versions moved and the config shifted a little, so here it is, cleaned up.
The problem: who gets the GPU
On a small team, handing GPUs to ML engineers is a very concrete problem. The big shops have mature k8s plus GPU virtualization, but that is a whole system: user auth, scratch storage, persistent storage, logging, quotas, usage accounting, a job queue. Only once all of that is in place does the k8s scheduler have anything to schedule.
In 2019 our situation was simple: few machines, few people, one kind of project. There was no resource contention to coordinate, so we set up LDAP auth on the host and ran the GPUs on bare metal.
By 2020 both the machines and the people had multiplied, and different engineers wanted different tasks and environments. Bare metal stopped working. GPU utilization was low, there was no boundary around the data, and environments collided — Anaconda and virtualenv could only paper over so much.
So we moved GPU delivery and isolation into Docker containers. That meant each container had to run an SSH service (most of our engineers like to attach VS Code directly to the server’s SSH), and the notebook login had to stay on the existing LDAP auth. We were not going to grow a second identity system just because the delivery mechanism changed.
The plan
- Deliver the GPU into the container with
nvidia-container-toolkiton top of Docker. - Stand up an internal image registry covering the common distros and CUDA versions.
- Build each engineer’s container image from that internal registry.
- An engineer’s project directory is their LDAP account’s home directory, usually under
/home/username. - The data an engineer may touch is passed through by bind-mounting host directories.
- Time, auth, and privileges (sudoers) inside the container are managed centrally.
The last two points are the whole trick: both the data and the identity come from the host, and the container is just an isolated runtime around them. An engineer can switch images or rebuild the container and still land in the same home directory under the same login. That is what we mean by reproducible.
Doing it
At image build time, install three things: openssh-server (and generate the host key files the SSH service needs); jupyterhub with jupyterhub-ldapauthenticator, plus Node.js and the npm package configurable-http-proxy; and the LDAP auth module nss-pam-ldapd.
Then create the container:
docker run -d --rm=false --net=host --restart=unless-stopped --name username --shm-size=8g (cat /etc/hosts|awk -F ' ' '{if(NR>2){print "--add-host "$2":"$1}}') -it -v /data-directory:/data-directory -v /home/username:/home/username -v /var/log/username:/var/log/username -v /etc/jupyterhub/username_config.py:/etc/jupyterhub/username_config.py -e TZ=Asia/Shanghai -v /etc/pam.d:/etc/pam.d:ro -v /etc/nslcd.conf:/etc/nslcd.conf:ro -v /etc/nsswitch.conf:/etc/nsswitch.conf:ro -v /var/run/nslcd:/var/run/nslcd -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v /etc/shadow:/etc/shadow:ro -v /etc/gshadow:/etc/gshadow:ro -v /etc/sudoers:/etc/sudoers:ro --runtime nvidia --privileged image:tag /home/username/start.sh
A few of these mounts and flags are worth spelling out, because they decide whether the networking and the identity line up.
--net=host puts the container on the host network. If you switch to bridge instead, you have to manage the container’s hosts file yourself and explicitly publish the SSH and notebook ports (configured in /home/username/start.sh and /etc/jupyterhub/username_config.py) with -p.
--name username uses the LDAP username as the container name so it is easy to recognize; multiple containers for the same person can carry a trailing number.
(cat /etc/hosts|awk -F ' ' '{if(NR>2){print "--add-host "$2":"$1}}') injects the host’s hosts entries into the container’s hosts file under host networking. Without it, JupyterHub’s HTTP proxy fails to reverse-proxy the container’s published ports. You do not need this under bridge networking, but it is worth using --add-host there to make sure localhost=127.0.0.1.
-v /etc/jupyterhub/username_config.py:/etc/jupyterhub/username_config.py passes this user’s JupyterHub config into the container.
-e TZ=Asia/Shanghai keeps the clock the same inside and out, otherwise log timestamps drift apart.
The most important part is this long run of read-only mounts:
-v /etc/pam.d:/etc/pam.d:ro -v /etc/nslcd.conf:/etc/nslcd.conf:ro -v /etc/nsswitch.conf:/etc/nsswitch.conf:ro -v /var/run/nslcd:/var/run/nslcd -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v /etc/shadow:/etc/shadow:ro -v /etc/gshadow:/etc/gshadow:ro -v /etc/sudoers:/etc/sudoers:ro
This mounts the host’s auth-related files into the container read-only. With them, PAM and nss inside the container reuse the LDAP integration the host already has — no need to configure an LDAP client in every container, and identity is consistent for free. /var/run/nslcd has to come in too, so the container can reach the host’s nslcd socket.
The contents of start.sh:
#!/bin/bash
set -e
ldconfig
#python3 auto_restart.py >> ./nohup.out #for troubleshooting on failure
/usr/sbin/sshd -p PORT &&
jupyterhub --config=/etc/jupyterhub/username_config.py --no-ssl >> /var/log/username/username.log
The JupyterHub config:
c.JupyterHub.ip = 'IP: under host networking use the host IP; under bridge use 127.0.0.1 or leave unset'
c.JupyterHub.port = PORT
c.Spawner.ip = 'IP: under host networking use the host IP; under bridge use 127.0.0.1 or leave unset'
c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator'
c.LDAPAuthenticator.server_address = "ldap://LDAP server IP or hostname:port, default 389"
c.LDAPAuthenticator.bind_dn_template = 'uid={username},ou=people,dc=test,dc=com'
authenticator_class is ldapauthenticator.LDAPAuthenticator, and bind_dn_template has to match your LDAP DN layout. With that, an engineer logs into the notebook with the exact same account and password they use for the server’s SSH.
Before starting sshd, generate the SSH host key files:
ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key
Then bring it up in start.sh:
/usr/sbin/sshd -p PORT #### path on CentOS
What this buys us
The engineer gets an isolated environment with a GPU, VS Code remote over SSH, and a notebook — with the same credentials and the same home directory as the bare-metal days, and nothing new to learn. On the ops side we drop the chore of maintaining a separate auth stack on every machine and in every container: identity is managed in exactly one place, LDAP, so adding a person, revoking access, or changing a password all take effect in one spot. And GPU utilization and the data boundary stop being a matter of good behavior — the container makes them real.