Rupert: my voice assistant buddy in the lab

I'm a software engineer with over forty years of writing code by hand, and I've spent weeks teaching a machine to talk to me out loud. His name is Rupert. He lives on the home lab, hears me from across the room, and answers back in plain English over the speakers. He is not a cloud assistant on a subscription — he's a stack of open models and hand-written Python running on my own machines, tuned the hard way, one Merl ticket at a time. This is the story of how he got here.

Rupert is the butler-style voice assistant I've been building to watch the market watchtower: when Jimi's squeeze detector fires, Rupert tells me out loud, in plain English, the moment it happens. But he does far more: he answers questions, runs tools, keeps timers, reads audiobooks, joins meetings, and files his own tickets. Most of all, he had to learn the difference between a command and a cough.

Every milestone below is a real ticket in Merl, our project tracker. You can trace Rupert's entire journey by number.

Hear him

Before the technical weeds, the point of all of it. These are Rupert's actual voice responses, generated by the same speech-and-model pipeline he uses live. He's dry, self-owning, and only gently absurd, which is exactly the calibration Ted asked for.

What Rupert is made of

Rupert is va_daemon.py, about four thousand lines of Python that plumb the room's microphone and speakers into a decision pipeline. Everything runs on the lab; the only thing that ever leaves the machine is the transcribed command text, and only when he deliberately escalates it. The pipeline, top to bottom:

1. Always-on listening. The mic streams 16 kHz frames into Vosk, a small on-device keyword spotter. Vosk's whole job is to hear the wake word and arm the system. It is deliberately tiny — it is not the speech recogniser, it's the doorman.

2. Speech-to-text. On a wake, the following utterance is handed to faster-whisper (openai's Whisper, optimised by the CTranslate2 runtime, running as a local model). Configuration matters: with condition_on_previous_text=False, Whisper transcribes each burst on its own and won't carry a wrong guess from one burst into the next.

3. The filtering gate. Before anything reaches a model, the decoded text must survive a gauntlet of rules: RMS audio floor, junk-sound detection, structure-and-fragment checks, social-nicety rejection. Rupert does not answer everything he hears; answering nothing is a feature.

4. The brain, in four tiers. Instant no-model answers (clock, system status, open a site), then a fast local brain, then a full agent on a local voice model, then an escalation to the big edge1 model. More on this below.

5. Speech out. The reply becomes audio via Kokoro, a local ONNX neural TTS, with a fallback to the older Piper engine if Kokoro fails. You're hearing the exact output above: same code path, same voice bm_lewis (Lewis, Rupert's current pick).

And every step (every model, every tier, every filter, every voice, every urgent fix) is logged as a Merl ticket with a running timer. That's the discipline that got him from "hello" to this.

The brain: four tiers, and knowing which one to use

The single most instructive design decision is that Rupert doesn't have one brain, he has four, and a routing rule that says a small fast model usually beats a big slow one.

Tier 0: no model at all. Some questions should never touch an LLM. "What time is it?" "What's the disk usage?" "Open the web site." These are answered by plain Python: spoken-time formatting, a system-query tier that reads memory/disk/CPU/uptime directly, and direct actions. Zero latency, zero hallucination, because there's no model to hallucinate.

Tier 1: the fast local brain. The default brain for most requests is a small local instruct model, NVIDIA Nemotron Lightning on the :10101), answered in about one and a half seconds. It carries Rupert's persona, today's date, and a bounded slice of recent conversation so follow-ups resolve their referents. When it says it doesn't know, it enriches with Bing snippets and re-asks before giving up.

Tier 2: the full local agent. For real work — anything that means using tools, searching the web, running files — a full Hermes agent session runs on the local voice model. This gives typed parity with the CLI: the same agent, the same tools, but voice-driven.

Tier 3: escalation to edge1. When the local brain declines ("I don't have your location," "couldn't find that one," or the model flat-out suggests it) Rupert auto-escalates to edge1 — the escalation tier, a two-node NVIDIA GB10 cluster on the lab's high-speed mesh (edge1 and edge2, serving :8000 over vLLM with tensor parallelism across the two nodes) running the larger DeepSeek V4 Flash 0731, the tougher-responses brain. Declines are never spoken as if final; they become a warm, pre-spawned escalation. Ted asked for exactly this: "go get the answer using edge1 without me having to say anything."

The models, named. Rupert is built and run on a small family of models he can route between. To build and run him we've leaned on Qwen 3.8 Flash Next and DeepSeek V4 Flash 0731. In live operation it comes down to two: Nemotron Lightning as the quick local brain for fast responses (~1.5s). It runs on a dedicated workstation, a Corsair AI Workstation 300 with 128GB of memory and a 40-core Strix Halo APU, on :10101, served by a dedicated standalone copy of llama.cpp (not the same serving path as the big models) and DeepSeek V4 Flash 0731 for the tougher responses that need the big gradient on edge1.

The rule of thumb: fast and local when it's easy, slow and powerful when it's hard, and a routing layer in between that stops embarrassments before they're spoken. That routing is the part that looks trivial and took the most tickets.

Filtering junk: the war against the dog bark

The hardest technical problem on this project was not speech recognition, and it was not the models. It was the sheer volume of junk that an always-on microphone collects, and the risk that the assistant will confidently answer it. This is where hundreds of commits went.

Whisper hallucinates on near-silence. In a silent room with just the fan humming, Whisper will confidently produce speech: "Thank you." "Thanks for watching!" Half a line of trading-sounding prattle. It's not hearing anything — at high mic gain, fan and radio hum satisfy the model's priors just enough to confabulate. Left unchecked, these fakes flooded the single worker and real commands queued behind them for minutes. Measured wake-to-handle went from fine to 09:22:47 → 09:23:49, a sixty-two second delay, felt as "it is not responding."

The fixes, all before Whisper ever runs:

• RMS floor at -45 dBFS — the fan's bursts (-60 to -90) are dropped instantly as "dismissed burst"; close-mic voice (-38 to -15) passes.
• Vosk-fire windows — a wake must fall inside a small time window around the burst, and the timestamp refreshes on every fire (a stale timestamp was silently dropping real wakes).
• Newest-burst-only — outside meetings, the worker keeps only the newest burst.
• Tightened wake matcher — edit-distance ≤1 with a leading 'h', plus a mishear whitelist (permise/harmes/herms/hernies/and a 'kay' prefix).
• Integer overflow fix — np.dot(int16, int16) wraps int32 at ~5×10¹¹ and turned negative, driving a sqrt domain error that ate whole bursts. Fixed with int64 accumulation and a clamp.

Then the semantic layer: a bare "thank you" in dialogue mode used to trigger a full agent "You're welcome, Ted" with a timer. A cough decoded to a vocalization is dropped. A short structureless fragment (a clipped "by music clubs" standing in for "what time does Dave Boyd Music close") must never reach a model, because the model will confidently answer the fragment as if it were a whole question. The _FRAGMENT_HAS_STRUCTURE check, the social-nicety rejection, and the structureless-fragment detection all exist so Rupert asks for a repeat instead of inventing a fact. One classic: the fragment above once produced a confident wrong answer about a 1995 film, because the model treated "by music clubs" as its own topic.

Rupert today gets most of his value out of politely saying the right nothing.

The journey, in Merl tickets

None of this shipped whole. It shipped as a running list of numbered failures and corrections, each one a ticket. What follows is the real timeline, condensed, but faithful, with ticket numbers you could look up.

September 1: birth. #1246 "Voice assistant: always-on mic wake-word listener." Rupert begins as a wake-word experiment. Within the evening (#1247–#1261) the shape appears: connect the voice loop to the CLI (#1247), and a long string of choices about the voice model: piper to neuTTS, then to a dedicated voice-model.service on :10101 (#1250), the model going from an 8B instruct (#1251) to a heavier Qwen3.6-35B-A3B with a 65k context (#1254). This is the fast-local-vs-big idea being discovered in real time: #1259 settles it as "local Qwen by default, the big DeepSeek only on an explicit ask."

September 2: audio, meetings, quality. Meetings arrive (#1263, #1265) with voiceprint speaker-ID (after Ted wisely rejected mid-meeting identification). The first real-mic test exposes natural-phrase bugs (#1271), a blocked end-of-meeting (#1274), and a "system status" question that hit the LLM tier, timed out at 180s, and hallucinated a battery reading (#1275). That single embarrassing failure drove both the no-model system-query tier and the brutal timeout shrink. Latency was a theme: #1276 cuts the catastrophic 180s agent timeout. Then Kokoro replaces Piper for TTS (#1281) because words ran together and felt too fast; warm ":10101" serving via prefix KV-cache reuse (#1283) makes the fast brain fast.

September 3: self-hearing and overload. The daemon starts hearing its own TTS and executing it (#1295). It overloads and false-fires because Whisper ran on every burst and a vosk-only path could start without a real wake (#1296). It goes deaf under a threaded-ml CPU spin (#1299). Each one is diagnosed and fixed, and each fix is a ticket.

September 4: he gets a name and a persona. Ted renames the wake word from "hermes" to Rupert to cut misfires (#1322), and opens personality calibration (#1325): the dry-butler tone, the self-deprecation, the humor that's funny without being a clown. Long-term memory appears (#1321): days and weeks of goals and preferences. And the quality-over-speed stability pass (#1335) makes clear the thing is a marathon.

September 5–8: hardening. Rupert stops answering bare niceties (#1353), stops nagging with uncapped "still working" reminders (#1354), learns to route action requests to the real tool agent instead of a question-only fast model (#1355), and gains the ability to create his own skills (#1366). A pipewire-pulse restart silently kills his microphone and he goes deaf (#1385), fixed. A _swap_stream that never re-inits PortAudio after a global terminate leaves him with a zombie mic overnight (#1403), fixed.

September 9–10: escalation behavior. The routing gains a keyword and an instinct: "edge1" escalated by name (#1398), and auto-escalation of every decline class to edge1 (#1399, #1406): nudges and "couldn't find that one" all promoted to fetching the real answer from the big gradient. The fast-brain decline never stands as a final answer anymore; it becomes a hand-off.

September 12–13: the why. We finally route the analysis tier through the right provider (#1427) and I write this article (#1442), the current ticket, running timer and all. Sixty-odd tickets, three model tiers, two TTS engines, one dog-bark war.

What the journey taught me

Revised 24 Oct 2026: a verbal tic, edited out. Revised 24 Oct 2026: punctuation and filler trim.A voice assistant is 20% model and 80% judgment. The speech recogniser and the LLM are the parts everyone can name; the parts that made Rupert usable are the RMS floors, the fragment checks, the decline-escalation, the timer hygiene, the refusal to answer a dog.

Small-and-fast beats big-and-slow until it doesn't. The local Qwen or Nemotron brain answers most things in a second and a half. Letting it decline plainly, and escalating those declines to edge1, gives you the speed of a small model and the power of a big one — and, crucially, you never hear a confident wrong answer spoken out loud. The instant no-model tier means the most-common questions cost nothing at all.

Write it down, time it, own it. Every one of those tickets has a timer. Every lesson — the overflow that ate bursts, the stale wake timestamp, the zombie mic, the KV-cache warm-up — is written down and searchable. Rupert's real intelligence is partly that he never has to rediscover the hard way, because the hard way was already a closed Merl ticket.

Rupert is a work in progress by construction — there are open tickets right now (#1396, #1412, #1413) for responsiveness, barge-in, latency and reminders. But he wakes when called, hears a command over a cough, judges which brain to use, and talks back in a voice Ted actually likes. For a hear-a-word, sort-the-nonsense, answer-the-question machine on a desk in the lab, that is most of the battle — and it was won one ticket at a time.

Written Sunday, from a desk in the lab, with a running timer on ticket #1442. The audio you heard was produced by Rupert himself.

— T.D. Brace & Rupert