In partnership with
The NVIDIA DGX Spark is sold as a personal AI supercomputer you can run from a desk. What NVIDIA does not tell you is that turning two of them into a functional distributed inference cluster requires resolving a silent ARM64 incompatibility that will break your entire Ray stack, navigating k3s networking configurations that can lock you out of both nodes via SSH, and tuning a tensor parallelism topology that only works once you understand the interaction between NCCL, KubeRay, and the ConnectX-7 RDMA interconnect.
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | July 21, 2026
What It Actually Does
"From Box to Cluster" (mohnishbasha.github.io/dgx-spark-bundle) is an 8-chapter technical ebook by Mohinish and Sanwi (July 2026, CC BY 4.0) covering the complete path from two unboxed NVIDIA DGX Spark units to a production AI inference cluster running Qwen2.5, Llama 3.3 70B, and other large models via vLLM, KubeRay, k3s, and AIBrix.
The hardware: two NVIDIA GB10 Blackwell GPUs, 128GB unified memory each, 256GB combined, connected via ConnectX-7 RDMA over QSFP. The software stack: DGX OS (Ubuntu 24.04.4 LTS, Kernel 6.17, CUDA 13.0), k3s v1.35.5, GPU Operator (Helm), KubeRay with Ray 2.49.2, vLLM 0.10.1.1, AIBrix v0.6.0, and Prometheus/Grafana monitoring.
Scope covered: every chapter in full: hardware setup, CUDA update via DGX Dashboard, k3s master/worker provisioning, GPU Operator installation, KubeRay deployment with the critical ARM64 image fix, RayCluster YAML, vLLM tensor parallelism configuration, AIBrix gateway setup, and Grafana/DCGM monitoring. Excluded: model fine-tuning, multi-user access control beyond AIBrix multi-tenancy, and hardware failure recovery beyond the troubleshooting appendix.
The Architecture, Unpacked
The two-node cluster has a clear topology: Spark 1 (192.168.86.30, hostname spark-720e) runs the k3s master, Ray head pod, vLLM API server on port 8000, KubeRay operator, AIBrix control plane, and monitoring stack. Spark 2 (192.168.86.26, hostname spark-7229) runs the k3s worker, Ray worker pod, and serves as tensor parallel rank 1. NCCL all-reduce operations cross the ConnectX-7 RDMA link.
Caption: The ConnectX-7 RDMA link is the critical path for tensor parallelism. Every model weight shard that does not fit on one GPU crosses this link on every forward pass. The NCCL all-reduce latency over RDMA determines the floor on inference throughput.
The Code, Annotated
Snippet One: k3s Install with the Flags That Matter
# ─── Spark 1 (Master): k3s install with two non-obvious flags ──────────
curl -sfL https://get.k3s.io | \
INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 --disable=traefik" \
sh -
# --write-kubeconfig-mode 644 allows non-root kubectl without sudo.
# Without this, every kubectl call needs sudo, which breaks most tooling.
# --disable=traefik removes the default k3s ingress controller.
# ← THIS is the trick: k3s ships Traefik by default; leaving it enabled
# creates port conflicts with AIBrix's own HTTP routing on port 80/443.
# Traefik + AIBrix fighting for the same ports is a failure mode the
# docs do not warn about.
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> ~/.bashrc
# Get the join token for Spark 2
sudo cat /var/lib/rancher/k3s/server/node-token
# ─── Spark 2 (Worker): join the cluster ────────────────────────────────
curl -sfL https://get.k3s.io | \
K3S_URL=https://192.168.86.30:6443 \
K3S_TOKEN=<TOKEN_FROM_SPARK1> \
sh -
# Assign the worker role label (k3s doesn't do this automatically)
kubectl label node spark-7229 node-role.kubernetes.io/worker=worker
# Verify both nodes ready with correct roles
kubectl get nodes
# Expected output:
# NAME STATUS ROLES VERSION
# spark-720e Ready control-plane,master v1.35.5+k3s1
# spark-7229 Ready worker v1.35.5+k3s1
Caption: --disable=traefik is the install flag that prevents a port conflict with AIBrix that appears hours later as an unexplained 502 error. The failure mode is not obvious because AIBrix installs cleanly and only fails at first HTTP request.
Snippet Two: The ARM64 Fix and RayCluster Deployment
# RayCluster YAML with the critical ARM64 image and tensor parallelism config
# Source: https://mohnishbasha.github.io/dgx-spark-bundle/books/from-box-to-cluster/chapter-05.html
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: vllm-cluster
namespace: core-services
spec:
rayVersion: '2.49.2'
headGroupSpec:
template:
spec:
nodeSelector:
kubernetes.io/hostname: spark-720e # ← Pin to Spark 1 explicitly
containers:
- name: ray-head
# ← THIS is the critical fix: rayproject/ray is x86-only.
# On ARM64 (DGX Spark Grace CPU), it fails with "exec format error"
# at container startup — a silent crash that looks like a pull failure.
# nvcr.io/nvidia/vllm:25.09-py3 is the only image with:
# - ARM64 native build
# - Ray 2.49.2 + vLLM 0.10.1.1 + CUDA 13.0 pre-installed
image: nvcr.io/nvidia/vllm:25.09-py3
command: ["/bin/bash", "-c"]
args:
- |
ray start --head \
--dashboard-host=0.0.0.0 \
--num-gpus=1 \
--block &
sleep 30 # Wait for Ray head to initialize before vLLM starts
python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-7B-Instruct \
--tensor-parallel-size 2 \ # ← Splits model across 2 GPUs
--distributed-executor-backend ray \ # ← Uses Ray for cross-node comms
--host 0.0.0.0 \
--port 8000 \
--gpu-memory-utilization 0.85 \ # ← 85% leaves headroom for OS/Ray
--max-num-seqs 4
env:
- name: HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token
key: token
resources:
limits:
nvidia.com/gpu: "1" # ← One GPU per pod; Ray handles the other
memory: "100Gi"
workerGroupSpecs:
- replicas: 1
groupName: worker-group
template:
spec:
nodeSelector:
kubernetes.io/hostname: spark-7229 # ← Pin to Spark 2
containers:
- name: ray-worker
image: nvcr.io/nvidia/vllm:25.09-py3 # ← Same ARM64 image
command: ["/bin/bash", "-c"]
args:
- |
ray start \
--address=vllm-cluster-head-svc.core-services.svc.cluster.local:6379 \
--num-gpus=1 \
--block
resources:
limits:
nvidia.com/gpu: "1"
memory: "100Gi"
Caption: --distributed-executor-backend ray is what makes vLLM use Ray for cross-node tensor parallel communication instead of NCCL direct. Without this flag, vLLM defaults to a local multi-process backend that cannot reach the second node.
It In Action: End-to-End Worked Example
Starting point: Two unboxed DGX Sparks on a desk, connected to each other via QSFP cable and to the same LAN switch. Total estimated setup time: 3-5 hours.
Step 1: Hardware and OS (Chapter 2-3, 60-150 min)
Physical: plug in power, QSFP RDMA cable, ethernet (both nodes)
First boot: complete setup wizard on each Spark (hostname, user, password)
Network: set static IPs via Settings → Network
Spark 1: 192.168.86.30
Spark 2: 192.168.86.26
SSH verify: ssh [email protected] from Spark 1
CUDA update: DGX Dashboard → System → Check for Updates → upgrade to CUDA 13.0
Step 2: k3s Kubernetes (Chapter 4, 20-30 min)
Spark 1: install k3s master (--disable=traefik)
Spark 2: install k3s agent using join token
GPU Operator: helm install → auto-deploys device plugin + DCGM exporter
Verify: kubectl get nodes → both Ready
kubectl get nodes -o json | grep '"nvidia.com/gpu":' → "1" per node
Step 3: KubeRay + vLLM (Chapter 5-6, 45-80 min)
NGC login: docker login nvcr.io (both nodes)
HF secret: kubectl create secret generic hf-token --from-literal=token=$HF_TOKEN
Cross-node ping test: deploy busybox pods, verify pod-to-pod connectivity
Apply RayCluster YAML: kubectl apply -f raycluster.yaml
First model download: Qwen/Qwen2.5-7B-Instruct, ~14GB, ~25 min
Verify Ray: {'GPU': 2.0, 'CPU': 40.0, 'accelerator_type:GB10': 2.0}
Step 4: Test inference
HEAD=$(kubectl get pods -n core-services -l ray.io/node-type=head -o name | head -1)
kubectl exec -n core-services $HEAD -- \
curl -s http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [{"role": "user", "content": "What is tensor parallelism?"}],
"max_tokens": 100
}'
# Expected output:
# {"choices": [{"message": {"content": "Tensor parallelism splits..."}}]}
# First response after cold start: ~3-5 minutes (model loading)
# Subsequent responses: <1s for short completions
Step 5: AIBrix + Monitoring (Chapter 7-8, 30-45 min)
AIBrix: helm install aibrix → gateway for routing, multi-tenancy, ModelAdapter
Grafana: kubectl port-forward -n monitoring svc/grafana 3000:3000
GPU metrics: DCGM Exporter → per-GPU utilization, memory, temperature, NVLink
End state: A production inference cluster serving Qwen2.5-7B-Instruct with OpenAI-compatible API, full Kubernetes observability, and GPU telemetry on Grafana. Total hardware cost: two DGX Sparks (approximately $3,000 each), one QSFP cable, one LAN switch.
Why This Design Works (and What It Trades Away)
The k3s choice is correct for this use case. Standard Kubernetes (kubeadm) adds etcd, a dedicated control plane node, and substantial system overhead. k3s packages the full Kubernetes API into a single binary under 100MB, leaving the bulk of both nodes' CPU and memory for GPU workloads. For a two-node cluster where both nodes run workloads, k3s's embedded SQLite (not etcd) is an acceptable tradeoff: you lose HA control plane redundancy, but a two-node home lab does not have HA to lose.
The tensor parallelism setup (TP=2 with Ray backend) is the most operationally sensitive component. --tensor-parallel-size 2 tells vLLM to split every layer's weight matrix across two GPUs using NCCL collective operations. The Ray backend coordinates which GPU handles which shard. The NCCL all-reduce over the ConnectX-7 RDMA link is what makes this fast: RDMA bypasses the CPU entirely, transferring data directly between GPU memory on both nodes.
The tradeoff is cold start latency. vLLM with tensor parallelism must synchronize both Ray workers before the first request is accepted. A model reload or Ray cluster restart takes 3-5 minutes for a cached model, 25 minutes for a fresh download. Teams expecting rapid model switching will find this frustrating. The book recommends keeping the cluster running continuously rather than cycling pods.
The AIBrix gateway (Chapter 7) sits in front of the vLLM API to handle routing, multi-tenancy with per-tenant API keys, and model lifecycle management via the ModelAdapter CRD. This is the component that enables multiple teams to share the cluster without cross-tenant request leakage. AIBrix's role is not inference acceleration; it is the access control and observability layer above vLLM.
Technical Moats
The ARM64 ecosystem gap is real and not closing quickly. The rayproject/ray image is x86-only at the time of writing. The nvcr.io/nvidia/vllm:25.09-py3 image from NVIDIA NGC is the only ARM64-native combination of Ray + vLLM + CUDA 13.0 that works on DGX Spark. Any upgrade to Ray, vLLM, or CUDA independently risks breaking the ARM64 build chain. Teams that pin to this specific image and test every upgrade path before applying it to production are the ones that maintain a working cluster. Teams that helm upgrade on a schedule will hit "exec format error" at the worst possible time.
The ConnectX-7 RDMA link is load-bearing and invisible. k3s and Kubernetes do not automatically configure the RDMA network for GPU-to-GPU communication. NCCL discovers available communication backends at runtime and prefers RDMA when the ConnectX interface is reachable. If static IPs are misconfigured, if the QSFP cable is in the wrong port, or if the ConnectX interface is not on the same subnet, NCCL silently falls back to TCP over the LAN, with a 5-10x throughput penalty that looks like slow inference rather than a networking failure. The book's cross-node ping test in Chapter 5 is not optional: it is the only way to confirm the data plane is correct before committing to the full stack.
GPU Operator handles a multi-component problem that is painful to do manually. Without GPU Operator, you need to manually install the NVIDIA device plugin (for nvidia.com/gpu resource scheduling), the DCGM Exporter (for GPU metrics), and the container toolkit (for GPU access from containers), and keep them synchronized with CUDA version. GPU Operator automates all three via a single Helm chart and a DaemonSet that self-manages on each node. Reproducing this manually is feasible but introduces version skew risk on every CUDA upgrade.
Contrarian Insights
Insight One: k3s is the right choice for this cluster today, but it will be outgrown by any team that adds a third node or needs production-grade HA. k3s uses SQLite as its backing store, not etcd. SQLite is fine for a two-node cluster where the control plane losing a node is an acceptable failure. Add a third node with GPU workloads, and the SQLite-backed k3s control plane becomes a bottleneck: it cannot handle the concurrent API request volume that three busy GPU nodes generate. The book correctly recommends k3s for getting started. Teams who use this cluster as a blueprint for a larger on-premises deployment should plan a migration to RKE2 (which uses etcd and supports HA control plane) before they need it. Migrating k3s to RKE2 after workloads are running is significantly harder than starting with RKE2.
Insight Two: The 256GB unified memory headline understates the real capacity constraint. DGX Spark's "unified memory" means the Grace CPU and the Blackwell GPU share the same physical memory pool: 128GB per node, 256GB combined. In practice, vLLM's --gpu-memory-utilization 0.85 means you are reserving 85% of each node's memory for the KV cache, leaving 15% (19.2GB per node) for the OS, k3s, KubeRay, and monitoring overhead. Monitoring alone (Prometheus + Grafana + DCGM Exporter) uses approximately 2-4GB. Ray's head node process uses 2-8GB depending on cluster state. Running Llama 3.3 70B with --gpu-memory-utilization 0.85 is feasible on 256GB combined but tight: the model weights alone are approximately 140GB at BF16, leaving ~116GB for the KV cache across both nodes. At --max-num-seqs 4, this is sufficient. At --max-num-seqs 32, you will hit OOM. The book provides the right defaults but teams need to understand the memory budget explicitly before increasing concurrency.
Surprising Takeaway
The standard Ray image silently fails without any readable error message in the Kubernetes pod logs. When you use rayproject/ray on DGX Spark instead of the correct nvcr.io/nvidia/vllm:25.09-py3 image, the container starts, produces no output, and terminates with exit code 1. The actual error, "exec format error," appears in the containerd runtime logs at the node level, not in kubectl logs. A developer checking only kubectl describe pod and kubectl logs will see "CrashLoopBackOff" with no explanation. The book calls this out explicitly and provides the correct image. But the broader point is that ARM64 architecture failures on DGX Spark are not caught at image pull time: the image pulls successfully from NGC because NGC stores multi-arch manifests, then fails silently at execution because the ARM64 layer was never built. Container image architecture validation is not a feature of kubectl or standard Kubernetes tooling.
TL;DR For Engineers
Two NVIDIA DGX Sparks (2× GB10 Blackwell, 256GB combined unified memory, ConnectX-7 RDMA) can form a production k3s + KubeRay + vLLM + AIBrix cluster in 3-5 hours using the book's guide at mohnishbasha.github.io/dgx-spark-bundle.
The single most important compatibility fact:
rayproject/rayis x86-only. The only working ARM64 image isnvcr.io/nvidia/vllm:25.09-py3(Ray 2.49.2 + vLLM 0.10.1.1 + CUDA 13.0). Using the wrong image produces a silent crash with no readable Kubernetes log.vLLM flags that matter:
--tensor-parallel-size 2,--distributed-executor-backend ray,--gpu-memory-utilization 0.85,--max-num-seqs 4. First model load (Qwen2.5-7B, ~14GB) takes ~25 minutes. Subsequent restarts: 3-5 minutes.k3s install requires
--disable=traefikto avoid port conflicts with AIBrix. Cross-node networking validation (kubectl exec busybox → ping) is mandatory before deploying the RayCluster, not optional.GPU Operator handles NVIDIA device plugin + DCGM Exporter + container toolkit in one Helm chart. Installing these manually introduces version skew risk on every CUDA upgrade.
Explain It Like I'm New
Running a large AI model on a laptop is straightforward: the model fits in memory, the software runs, and you get responses. The challenge starts when the model is too large to fit on one machine's memory. A 70-billion-parameter model at standard precision requires roughly 140GB of memory. Most machines don't have that much.
The solution is to split the model across multiple machines, the same way a heavy workload might be split across multiple workers in a warehouse. Each machine holds a portion of the model's parameters, processes its portion of every request, and communicates the result to the other machines so they can combine everything into a final answer. This is called tensor parallelism.
The NVIDIA DGX Spark Bundle is two desktop-sized AI computers (each about the size of a Mac mini) that are designed to be linked together for exactly this purpose. They connect via a special high-speed cable called RDMA (Remote Direct Memory Access), which allows them to share data at speeds comparable to what you'd get within a single machine.
What this book documents is the software layer needed to make that hardware work: Kubernetes for managing the workloads, Ray for coordinating the two machines, and vLLM for actually serving the AI model. Getting all of these to work together on the DGX Spark's ARM64 processor, which is different from the x86 chips most AI software is built for, requires specific configuration choices that are not obvious from the documentation of any individual component.
This matters because it makes high-capacity AI inference practical for small teams and individuals, not just large data centers.
See It In Action
"From Box to Cluster" ebook (online edition) Source: Serverless Ventures / mohnishbasha.github.io | https://mohnishbasha.github.io/dgx-spark-bundle/books/from-box-to-cluster/ The full 8-chapter guide with every command, YAML, and configuration table. Chapter 5's RayCluster YAML and Chapter 6's vLLM flag table are the most referenced sections.
PDF download Source: Serverless Ventures | https://mohnishbasha.github.io/dgx-spark-bundle/books/from-box-to-cluster/dist/Ebook_From_Box_to_Cluster.pdf The complete guide in a single downloadable PDF for offline reference during hardware setup.
NVIDIA DGX Spark Product Page Source: NVIDIA | https://www.nvidia.com/en-us/products/workstations/dgx-spark/ Hardware specifications, GB10 Blackwell architecture details, and the official DGX OS documentation that complements the cluster setup guide.
vLLM Documentation (Tensor Parallelism) Source: vLLM Project | https://docs.vllm.ai/en/latest/serving/distributed_serving.html The canonical reference for
--tensor-parallel-sizeand--distributed-executor-backend rayflags. The DGX Spark guide uses vLLM 0.10.1.1; flag behavior may differ in newer versions.KubeRay Documentation Source: Ray Project | https://docs.ray.io/en/latest/cluster/kubernetes/getting-started.html Official KubeRay operator documentation, including RayCluster CRD reference. Useful for customizing the YAML beyond the defaults in the ebook.
Community Conversation
Mohinish Shaikh (@mohinishbasha on X) https://x.com/mohinishbasha The primary author's account, where the book was announced and where ARM64 compatibility issues are discussed. The announcement thread generated the most useful community questions about NCCL backend fallback behavior and AIBrix multi-tenancy configuration.
NVIDIA DGX Spark Community Forums https://forums.developer.nvidia.com/c/dgx/ Active discussion of DGX Spark first-boot issues, CUDA update ordering, and GPU Operator compatibility on ARM64. The most commonly reported issues (CUDA driver mismatch, GPU Operator version pinning) are exactly the failure modes the book's Chapter 3 pre-empts.
vLLM GitHub Issues (ARM64) https://github.com/vllm-project/vllm/issues?q=arm64 The open issue tracker for ARM64 vLLM compatibility. Useful for tracking when (or whether) the standard vLLM image gains ARM64 support, which would simplify the NGC image dependency.
KubeRay GitHub Discussions https://github.com/ray-project/kuberay/discussions Community discussion on RayCluster YAML patterns, nodeSelector usage, and the cross-node networking validation approach. Several threads directly relevant to the multi-node vLLM configuration used in the ebook.
The ARM64 Documentation Gap Is the Real Product Problem, Not the Hardware
The DGX Spark is a well-engineered piece of hardware. The GB10 Blackwell GPU, the unified memory architecture, and the ConnectX-7 RDMA interconnect are all correct choices for a personal AI supercomputer cluster. The product's real gap is documentation: turning two DGX Sparks into a working cluster required days of trial and error that should not have been necessary.
"From Box to Cluster" is the documentation NVIDIA should have shipped with the DGX Spark Bundle. It covers ARM64 image compatibility, k3s configuration specifics, NCCL backend fallback behavior, and vLLM tensor parallelism flags from direct operational experience, not from assembling documentation written for different hardware.
The interesting forward-looking question is whether the ARM64 AI software ecosystem catches up to the hardware. Every x86-only container image that gets discovered on DGX Spark is a signal that the community built its tooling assuming Intel/AMD architecture. The DGX Spark, Apple Silicon, and Qualcomm AI developer platforms are collectively forcing the ecosystem to treat ARM64 as a first-class target. That transition is still incomplete in July 2026. Teams building on ARM64 AI hardware today are ahead of the ecosystem, not behind it.
References
"From Box to Cluster" (Mohinish and Sanwi, July 2026, CC BY 4.0) is an 8-chapter operational guide that documents every command and configuration decision required to turn two NVIDIA DGX Spark units (2× GB10 Blackwell, 256GB combined unified memory, ConnectX-7 RDMA) into a production inference cluster running vLLM with tensor parallelism on k3s, KubeRay, and AIBrix. The guide's most critical contribution is calling out the ARM64 incompatibility that silently breaks the standard Ray container image and providing the exact NGC image (nvcr.io/nvidia/vllm:25.09-py3) and vLLM flags required for a working multi-node deployment.
Sponsored Ad
If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad — it helps us keep building and delivering value 🚀
AI help, without the trust tax.
Most AI tools ask you to trade your data for intelligence. Norton Neo doesn't. It's the first safe AI-native browser built by Norton, and it gives you powerful built-in AI without handing your privacy over to get it. Search, summarize, and write with AI built directly into your browser. Your data stays yours. Your context stays private.
Built-in VPN, anti-fingerprinting, and ad blocking come standard. No add-ons. No setup. No compromises.
Fast. Safe. Intelligent. That's Neo.