Running my homelab on Qwen

I've been building a persistent agent that administers my homelab. I call it NetAdmin.

It's a Pi coding agent running on its own Proxmox VM, as an unprivileged user. I can talk to it in a terminal session that systemd keeps alive, or through a small web app with conversations, streaming replies, and approvals right in the thread.

The interesting part for me isn't just that it can check a host or read some logs. It's that I was able to take a small local model, put it in a harness with real questions and injected scenarios, and tune the setup until I was comfortable letting it do the work.

Below I'm going to cover how the system is put together, how that eval loop worked, and what this has taught me about local inference.

How is it put together?

I put the agent on its own VM so a compromised app in the lab can't reach it or its credentials. It doesn't get a root shell. Its reach comes from the tools I give it, and from a guard that decides every tool call before anything runs.

Pi is pinned and kept running in tmux. The web app is a small React and Vite frontend with a Node server, and it starts a separate Pi process for each open conversation. I can switch the model for one conversation without touching the others, and Settings has a default for new ones. The app has its own login, it works on a phone, and it's accessible on my tailnet, so I can reach it from anywhere.

The page shows streaming replies and groups the agent's tool calls into steps. When the guard wants a decision, the approval shows up inline, with the same choices I get in the terminal. Each conversation has a permission mode: Ask before changes, which is the default, Ask for everything, or Permissions off. Even with permissions off, the hard denials still apply.

The deployed web app is owned by root. The agent can edit the docs it works from, but it cannot change the program that relays its own approvals.

What can it actually do?

By default there is no shell. The agent can read, edit and search files in its working tree, and it can use git in that repository. Anything that leaves the VM goes through Swamp, a set of behaviors I wrote for the machines and services in the lab: host health, VMs, containers, logs, the reverse proxy, HTTP checks, backups, and a workflow for moving an app.

Reads can run. Writes ask, unless I've declared a specific one safe. If the behavior code itself has changed since I last approved it, the next Swamp command asks again and shows me what changed.

Secrets come from 1Password and go straight into the behaviors. The values never reach the model, and they never end up in the conversation.

There's also a daily health check that collects read only facts and mails me a summary, an audit log of every tool call that gets copied off the VM so the agent can't erase it, and a kill switch if I need to shut the whole thing down.

The eval loop

I didn't start by pointing a local model at the live lab and hoping for the best. I built a mock mode and a replay harness first.

In mock mode the model still uses the same tools, but nothing reaches another machine. I can give it a scenario, like a disk showing read errors, or last night's backups missing for one app, and the tools answer as if that were true. File edits in the working tree are still real, so I have to watch that, but the rest of the lab stays untouched. In the web app this is Test mode: it turns mock mode on for every conversation, with an optional scenario, and puts a banner on the page so I can't forget it's on.

The evals themselves are real requests I've actually made. Each one has a scenario for the mocker, the calls a good agent has to make, and facts the answer has to contain. The runner starts a fresh session, turns mock mode on, asks the question, and scores the result. Anything that would need approval is blocked, so a run like this can't change anything real.

Claude could drive that eval loop. Qwen could drive the tools, based on those questions, and try to perform the work. And the harness could inject different scenarios, which helped me fine-tune how the model actually worked.

The first pass was rough. Qwen would skip the actual behavior run, or pass arguments in the wrong shape, or occasionally try a write nobody asked for, which the guard blocked. Some of that was the model. Some of it was the setup.

One of the more embarrassing gaps wasn't Qwen's fault. For a reasoning model at an address Pi doesn't recognize, Pi sends its system prompt as a developer message, and Ollama was silently dropping those. Qwen was working with no instructions at all. Once that was fixed, the same kind of question started going to the right tool first.

I also tightened the instructions and the guard, including having the guard reject a malformed command and show the form to use, instead of just asking me about it. After a few harness changes, the same set of tasks went from almost all failing to almost all passing on the first pass. There's still a flaky case around restarting a VM, where sometimes it asks first, which is what I want, and sometimes it reaches for the wrong kind of write and the guard holds it. The mock data can also contradict the scenario, and the scorer checks for the right names, not every fact.

The local model is Qwen 3.5 9B, running on Ollama. I had to give it a much larger context window than the default, because the default was too small for Pi's prompt.

Now I just run Qwen

That loop is what got me here. I now just run Qwen. It's not quite as good as something like Codex, but it gives me a lot of assurances that I'm not going to run out of usage.

I've fully switched to it. NetAdmin runs on Qwen now, and that's where I'm taking it from here.

Wrapping up

Local inference is kind of in a weird state. It's pretty good at a lot of things, but it hasn't quite worked its way into becoming the backbone of anything.

I think this is a nice proof of concept for me, for how to make the most of local hardware, but also how to tune up a small model with a harness and a set of replay evals until it can do real work in a tight box.

If I had to boil it down to the things that mattered most, it would be these:

• Put the model in a box before you let it talk to real machines. A guard that decides every tool call is more useful than hoping the model behaves.
• Keep a shell out of its hands unless you turn that on yourself.
• Don't start by arguing about the model. Build a way to inject scenarios and score real tasks, then change the instructions and the tools until the scores move.
• A smaller local model will miss things a frontier model wouldn't. The point of the harness is that you can see those misses, and that the guard still catches the dangerous ones.

Thanks so much for reading, see you next time!