Qualify Local AI Models Under Load

Self-host a local AI model only after it meets your task, concurrency, and restart requirements on the machine that will serve it. Compare Ollama and LM Studio on the same workload, then decide which model to keep and which requests need a different route.

Specify what has to run locally

Write down where model inference, agent orchestration, tool calls, and stored inputs must live. These can run in different places. Cursor documents that Self-Hosted Machines run tool calls on hardware you control while inference and the agent loop remain in Cursor. That arrangement is a candidate if a repository and tool execution must stay inside your perimeter and your policy permits remote inference.

For a local inference requirement, shortlist a model runtime on your own hardware and test it through the client that will use it. Use synthetic data to check the boundary. With external access blocked and any remote fallback disabled, send a fresh request and verify a correct response through the expected local endpoint. A job that requires an external inference route fails that requirement even when its files stay local.

Record an identical prompt set at each concurrency level and label cold starts separately. Mixing those trials makes the tail latency difficult to interpret.

Choose where Cloud Agents run | Cursor Docs

Compare runtimes through the intended client

If the client is NanoClaw, its repository documents a setup route for local open-weight models via Ollama and allows backend selection per agent group. Start with a dedicated test group and a single model version. For LM Studio, verify how the intended client can connect, which endpoint and authentication the installed versions support, and whether the same model and quantization are available. OpenClaw's provider FAQ points to LM Studio for local models, so it is also a candidate for an OpenClaw trial.

Run both candidates through the same client where possible. Match the model, quantization, context limit, and requests, and record any setting you cannot match. Compare whether each client can complete the intended task, preserve the required output format, and reconnect after a runtime restart. A successful prompt in the runtime's own interface leaves the client integration untested.

Reserve memory for the operating system and other services before setting the available capacity. A model fitting once is a different observation from completing a sustained workload.

GitHub - nanocoai/nanoclaw: Your own always-on AI assistant: self-hosted, open source, lightweight. Each agent runs in its own container on your machine, on the harness you choose (Claude Code, Codex, Cursor, OpenCode). Talk to it from Slack, WhatsApp, Telegram, Discord and more. · GitHub

Measure correct answers at peak demand

For an illustrative acceptance test, suppose two operators submit requests at once and the longest task includes 12,000 input tokens. Build a fixture with a required fact near the beginning and another near the end. Give the runtime the same pair of requests when it is cold, when it is warm, and after a restart. Suppose your acceptance rule is that both answers include both facts and each finishes within 30 seconds. Those are hypothetical requirements you would replace with your own.

OpenClaw's FAQ flags context truncation on small cards. Log whether each answer kept the required facts, how long it took, and whether the host showed memory pressure or a failed request. If a shorter prompt succeeds while the long one drops a required fact, inspect the effective context limit and repeat the case before changing the prompt or promoting the model.

FAQ: providers, hardware, and hosting - OpenClaw

Keep recovery and fallback in the trial

A representative test needs the hardware the model will actually use. Docker documents that local sandboxes can use supported host integrations such as a GPU, while cloud sandboxes have no access to the host's hardware. If you use a sandboxed client, verify the client can reach the intended inference runtime and check device use on the machine serving the model. Repeat the longest accepted request after restarting that machine or runtime and measure how long it takes to serve a correct answer again.

If peak concurrency fails but single requests pass, test a queue or reduce concurrency and measure the added wait. For work that may leave the machine, evaluate a separately approved hosted route with the same fixtures. For work that must stay local, return a clear failure when the local runtime is unavailable and check that no external request was sent.

Compare local and cloud sandboxes | Docker Docs

Sources