Beside the window in my small office sits an NVIDIA DGX Spark with 128GB of unified memory. Nearby is a desktop containing an RTX 4090.

For models that fit within its 24GB of memory, the 4090 is considerably faster. Under sustained load, however, it can sound like an aircraft preparing for take-off. The Spark is slower, but quiet enough to leave running while I work.

Keeping substantial compute at home runs against the cloud-first paradigm that has shaped modern software engineering. For exploratory work, however, local hardware can be more practical than provisioning cloud resources for every experiment.

The Psychological Tax of the Cloud Meter

When a GPU instance starts on AWS or Google Cloud, so does the billing meter.

That is fine for predictable production workloads. It is less appealing when testing an unfamiliar model, vibe coding a hackathon app or trying the latest agentic toolkit. A dependency problem, CUDA mismatch or out-of-memory error can burn through a paid session before the real experiment has even begun.

The running meter can also make experimentation feel less open-ended. Local hardware still carries electricity and depreciation costs, but I can leave an inference server running overnight, test memory limits and fail repeatedly without watching an hourly charge.

Bandwidth Versus Capacity

The two machines expose a fundamental trade-off in AI hardware: bandwidth versus capacity.

The RTX 4090 has 24GB of GDDR6X memory and around 1 TB/s of memory bandwidth. When a model fits comfortably within that limit, inference is exceptionally fast.

But 24GB is a hard boundary. Once the model weights, context and KV cache exceed the available memory, performance is no longer the issue. The workload simply does not fit.

The DGX Spark takes a different approach. Its GB10 platform provides 128GB of coherent unified memory, making larger quantised models and longer contexts possible without spreading the workload across several discrete GPUs.

The compromise is bandwidth. The Spark provides 273GB/s, substantially less than the 4090. A model may fit comfortably in memory while generating tokens more slowly.

The 4090 is the machine I use when the model fits and speed matters. The Spark is the machine I use when the experiment needs room—or when I would rather not spend the day listening to cooling fans.

A Serving Stack I Do Not Have to Babysit

Capacity only matters if the software can use it reliably.

I previously spent more time experimenting with individual vLLM flags, quantisation settings and container builds. These days, I use the community-maintained spark-vllm-docker project. I pull the latest image, choose an appropriate recipe and quantised model, and get on with the experiment.

In my experience, it has been reliable enough that maintaining the serving environment no longer becomes a project of its own.

Quantisation remains important. It reduces the memory occupied by model weights, leaving more room for context and cache. Larger models can fit on the Spark, although fitting a model into memory should never be confused with running it quickly.

For repeatable benchmarks, I record or pin the container and model versions. Pulling latest is convenient, but it is also a moving target.

Keeping Sensitive Data Local

Privacy is another reason local inference matters.

There are situations where confidential documents, internal source code or other sensitive material should not be sent to an external model provider. Cloud platforms can offer strong contractual and technical protections, but those safeguards still need to be assessed rather than assumed.

Running models within a controlled local environment provides another option. Where governance permits, data can remain inside that environment while retrieval systems, extraction pipelines and agent workflows are tested. This can be particularly useful during early prototyping, when logging, retention and data flows may still be evolving.

Local does not automatically mean private. Containers may include telemetry, applications may retain prompts, and downloaded models and dependencies still require scrutiny. But with networking, storage and logging configured appropriately, local inference can keep sensitive data out of third-party services entirely.

A Machine That Can Still Evolve

The hardware may be fixed, but its capabilities are not.

Serving engines continue to improve, quantisation becomes more efficient, and increasingly capable open models are released at sizes that fit within 128GB of memory. A software update or better model can therefore make the same machine noticeably more useful without changing the hardware.

The Spark cannot approach frontier systems backed by vast multi-GPU clusters. That is not its purpose. Its value is that it provides enough memory to explore many of the same ideas locally: long-context inference, retrieval, tool use, quantisation and agent workflows.

It is less a miniature frontier cluster than a practical laboratory for learning how modern AI systems work.

Local Compute Is Not a Rejection of the Cloud

None of this means local hardware should replace cloud infrastructure.

The cloud remains the obvious choice when demand is bursty, workloads must scale across many GPUs or a service needs broad availability. Hosted APIs are also effective when the objective is to integrate a capable model rather than examine the infrastructure beneath it.

Local compute serves a different purpose. It provides a stable experimental baseline.

I can measure memory consumption, compare quantisation formats and observe where inference actually slows down. I can test emerging tools without repeatedly provisioning infrastructure and, where appropriate, keep data within a controlled environment.

Cloud prices, rate limits, model availability and product terms can change. A system built entirely around an external API inherits those uncertainties. Local hardware provides another option—and a clearer view of what the underlying infrastructure is doing.

Hardware as an Instrument

I do not keep the DGX Spark and RTX 4090 because I expect them to replace the cloud or win a simple cost comparison.

They are instruments.

The 4090 gives me bandwidth and raw speed. The Spark gives me capacity, quieter operation and a platform that continues to improve as the open-model ecosystem matures.

Together, they let me learn, experiment and keep some workloads local—without turning my small office into an airport runway.

Local-llm

The Local Silicon Equation: Why I Keep 128GB of AI Compute in My Office