Running models locally can improve privacy, experimentation speed, and cost control. It does not make an AI workload private by default. Prompts can still reach external integrations, models and containers still form a software supply chain, and GPU access changes the isolation boundary of the host.
I built a Proxmox LXC environment around a used RTX 3090 with 24 GB of VRAM and older workstation hardware. The result is a useful local inference platform without dedicating the GPU to one full virtual machine. The design works because LXC shares the host kernel and can expose the NVIDIA device nodes to more than one container.
That efficiency is also the central tradeoff: LXC provides less isolation than a VM with PCI passthrough. For untrusted code, hostile multi-tenancy, or strong workload separation, I would choose the VM boundary and accept the additional overhead.
The architecture
Proxmox host ├── NVIDIA kernel driver and physical GPU └── unprivileged LXC ├── matching NVIDIA user-space libraries ├── NVIDIA Container Toolkit └── Docker AI workloadsThe host owns the kernel module. The LXC receives selected /dev/nvidia* devices and matching user-space libraries. Docker then uses the NVIDIA runtime inside the LXC.
The exact driver, Proxmox kernel, device nodes, and CUDA image tags change over time. Treat the commands below as a design pattern and pin versions that are mutually compatible.
1. Establish the host driver first
Install the Proxmox kernel headers and the dependencies required by the driver method you select:
apt updateapt install --yes pve-headers dkms pkg-config libglvnd-dev nvtopUse NVIDIA’s packaged or official installer guidance for a driver supported by the GPU and kernel. If nouveau, nova_core, or another module binds the card first, resolve that conflict deliberately and rebuild the initramfs before continuing.
Validate the host before touching the container:
nvidia-smilspci -nnk -d 10de:ls -al /dev/nvidia*Record the driver version and the device major numbers. A working host is the prerequisite; container changes cannot repair a broken host driver.
2. Set host-wide power policy
Power limits apply to the physical card, so set them on the Proxmox host. On my RTX 3090, 280 W was a useful starting point for reducing heat and power draw while retaining most inference performance. That value is an observation from this hardware, not a universal optimum.
Check the supported range before setting it:
nvidia-smi \ --query-gpu=power.min_limit,power.default_limit,power.max_limit \ --format=csvnvidia-smi --persistence-mode=1nvidia-smi --power-limit=280If the setting should survive reboot, manage it as host configuration—for example, with a small systemd unit—and monitor whether driver upgrades change the supported range.
3. Create a constrained LXC
Use an unprivileged container unless a tested requirement proves otherwise. Avoid piping an internet-hosted helper script directly into a root shell; review and pin any provisioning code before execution.
Install Docker through a maintained repository, patch the base OS, and keep the container off untrusted networks while the GPU configuration is incomplete.
4. Map only the required NVIDIA devices
Inspect the host’s current devices:
ls -al /dev/nvidia*Then add the corresponding device permissions and bind mounts to /etc/pve/lxc/<id>.conf. A single-GPU example may look like this:
lxc.cgroup2.devices.allow: c 195:* rwmlxc.cgroup2.devices.allow: c 235:* rwmlxc.cgroup2.devices.allow: c 238:* rwmlxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=filelxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=filelxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=filelxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=filelxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=filelxc.mount.entry: /dev/nvidia-caps/nvidia-cap1 dev/nvidia-caps/nvidia-cap1 none bind,optional,create=filelxc.mount.entry: /dev/nvidia-caps/nvidia-cap2 dev/nvidia-caps/nvidia-cap2 none bind,optional,create=fileThe major numbers and available device nodes are host-specific. Do not copy them blindly. Restart the LXC and confirm that only the intended devices appear inside it.
5. Match user-space libraries inside the LXC
Install NVIDIA user-space libraries that match the host driver, but do not build another kernel module inside the container. With NVIDIA’s standalone installer, that generally means using --no-kernel-modules and omitting unnecessary graphics libraries for a headless workload.
After installation, restart the LXC and validate:
nvidia-smiA driver/library mismatch or No devices were found usually means the host driver changed, the container libraries no longer match, or the LXC has not been restarted since device bindings changed.
6. Configure Docker’s NVIDIA runtime
Install the NVIDIA Container Toolkit from NVIDIA’s repository, then configure Docker:
nvidia-ctk runtime configure --runtime=dockersystemctl restart dockerDepending on the LXC configuration, the NVIDIA runtime may require no-cgroups = true. Treat that as an environment-specific compatibility setting, not a default to apply everywhere.
Validate the entire chain with a pinned CUDA image compatible with the selected driver:
docker run --rm --gpus all <reviewed-cuda-image> nvidia-smiOnly after that test passes should an inference service be introduced.
The security controls around local AI
GPU access is only one layer. I also treat the following as part of the design:
- Do not expose model APIs or management interfaces directly to the internet.
- Authenticate internal clients and separate administrative from inference access.
- Keep secrets out of images, compose files, prompts, and model logs.
- Restrict outbound access when the privacy objective depends on local processing.
- Review model, container, and extension sources before execution.
- Monitor resource use; a runaway context or model can exhaust shared memory and GPU capacity.
- Back up configuration and data required to rebuild the service, then test restoration.
Local AI is attractive because it gives the operator more control. It also gives the operator more responsibility. The design is successful when privacy, isolation, recoverability, and cost are explicit decisions—not assumptions inherited from the word “local.”
Lessons learned
On this build, sharing the host driver gave me useful inference capacity without reserving the GPU for one VM. It also removed any ambiguity about who owned the risk. I still had to manage device exposure, the shared kernel, container privileges, model provenance, outbound traffic, and recovery.
“Local or cloud” is too coarse a design question. I need to know which workloads can safely share a kernel and accelerator, what data they may process, which dependencies I trust, and which failures the environment must survive. When stronger isolation matters more than GPU utilization, I use a VM or a separate system. LXC belongs where its narrower boundary is understood and tested.
I validate that choice at every layer: compatible host and container drivers, only the intended device nodes, enforced network policy, authenticated access, observed egress, bounded resource exhaustion, pinned dependencies, and a clean rebuild or restore. Recovery should sit outside the workload’s failure domain; the n8n backup failure shows what happens when a service is also responsible for protecting itself.
Where has your organization drawn the isolation boundary for local AI workloads, and what evidence would justify moving that boundary closer to—or farther from—the host?
Sources and disclosures
- Proxmox VE documentation: Linux containers
- NVIDIA Container Toolkit installation guide
- NVIDIA Container Toolkit architecture overview
This analysis is based on the author’s home-lab implementation using an RTX 3090. The 280 W setting is a hardware-specific observation, not a performance or efficiency benchmark. The article is not affiliated with or endorsed by Proxmox or NVIDIA.
