How I Built My JARVIS

Every morning my computer greets me before I say a word. It reads my sleep score, my calendar, the news headline and the markets while I'm still holding my first coffee. Then it reminds me how many projects I've left unfinished. I didn't love that part either.

People assume this needs a lab and a budget. It needs neither. This is the full build: every part, every number I had to tune, and everything that broke on the way. It's long because I wanted the next person to have the guide I didn't.

What he actually does

• Wakes on "Hey Jarvis", then stays in conversation. You only say his name once; after that you just talk, and he decides whether a sentence was meant for him.
• Delivers a morning brief without being asked: sleep, calendar, the day's plan, headlines, markets.
• Opens any app or website by voice. "What's on my calendar" doesn't read a list, it opens the calendar.
• Answers common questions instantly and does real work (browsing, running commands, reading files) when the question needs hands.
• Speaks first when something matters. Any script on the machine can make him talk, so failing services, finished builds and reminders all arrive as a voice in the room.
• Knows the state of my work. The "twenty one unfinished projects" line comes from him actually counting repositories with uncommitted changes.

The architecture

Four parts. Ears, brain, voice, face. One rule connects them: every component is replaceable, and the glue between them is a text file and a named pipe. No message bus, no framework, no cloud.

Ears

A wake word model (openwakeword, which ships with a ready made "hey jarvis" model) scores every chunk of mic audio all day. Above the threshold, he wakes.

The threshold is where I lost my first evening. The default is 0.5. My best, clearest attempt scored 0.48, so he ignored me all night while I shouted at a screen like a madman. The fix was measurement, not faith: I logged every score, saw that background noise never crossed 0.02, and dropped the bar to 0.12. It has not false fired since, and it catches me from across the room.

Once awake, a voice activity detector (silero VAD) finds where my sentence starts and stops. Three details matter more than any tutorial admits:

• Keep a rolling three second buffer of audio from before the wake word fired. People start talking as they say his name, and without the buffer you lose the first half of every command.
• Require a minimum amount of actual speech (I use about a quarter second) before treating a turn as real. Doors, chairs and coffee cups trip the detector constantly.
• End the turn after about three seconds of silence, not one. People pause mid sentence to think.

Transcription is faster-whisper running the small English model on CPU. One hard learned rule: turn OFF whisper's own built in voice filter. It silently discarded quiet speech that the VAD had already accepted, and I spent hours hunting a bug that was a default setting.

The gate: knowing when he's being spoken to

After he answers, he stays awake for about a minute. Anything said in that window gets transcribed and judged: if it contains no question and no request, it wasn't for him, and he stays quiet. Three ignored sentences in a row, or a minute of nothing, and he goes back to sleep.

This one heuristic is the difference between an assistant and an appliance. I can watch videos, talk on the phone, mumble to myself. He ignores all of it and still catches "actually, open that link" without me saying his name again.

Brain: two speeds

Everything transcribed goes down one of two paths.

The fast path, under a millisecond. Time, date, calendar, the morning brief. These answer from plain local code with several phrasings per answer so he never sounds like a ringtone. No AI call at all. The trick that keeps this safe: the fast path only fires on pure questions. Anything containing an action verb ("open", "close", "play", "stop") falls through to the agent, otherwise "open my calendar" gets you a reading instead of a window.

The agent path, about five seconds. The transcript goes to an AI coding agent running in headless mode (I use Claude Code; any of the big agent CLIs works) with a system prompt that makes him a butler, and real permissions: open apps, browse, run commands, read files. Two things make this feel alive rather than laggy:

• The reply streams sentence by sentence, and he starts speaking the first sentence while the rest is still being generated.
• While he works, he narrates what he's actually doing ("Searching your files, sir") instead of playing a spinner sound. I built this after realizing I use him mostly with the screen off. Assume the display is off and every state must be audible: that single rule shaped more of the design than any other decision.

He also keeps a persistent session, so "and tomorrow?" works as a follow up to "what's on my calendar today".

Voice

Kokoro, an open text to speech model, running locally. The voice is called bm_george: dry, British, unimpressed. He calls me sir. Non negotiable.

The pipeline renders ahead: while sentence one plays, sentence two is already being synthesized. Without that, every reply has a dead gap in the middle and the illusion dies.

Face

The orb is one HTML file. About 1500 particles arranged on a sphere with the golden angle, drawn on a canvas, rotating as one body with depth based size and brightness. Cyan while listening and speaking, gold while thinking, dim while asleep. On boot the particles fly in from scatter and assemble, which is the single most Iron Man thing on the whole machine.

It has no connection to the assistant at all. It polls a tiny JSON file:

{"state": "thinking", "text": "The news while you slept: ..."}

The assistant writes that file whenever its state changes. The page reads it four times a second. That's the entire integration, and it means the face survives any redesign of the brain.

Glue: the announcement pipe

There's a named pipe on disk. Anything written to it, he speaks:

echo "Sir, the build finished. Two tests failing." > ~/code/jarvis/announce

That one line turned him from a toy into infrastructure. Cron jobs, deploy scripts and failing system services all talk now. And after he announces something, he opens his listening window, because announcing and then going deaf would be absurd in a person and it's absurd in software.

The morning brief

The brief is boring engineering and that's why it works. A small script reads the local calendar cache directly (milliseconds, no network), another counts repositories with uncommitted work, one more pulls the headline and index moves. The words get assembled with numbers spelled out ("twenty one", not "21", synthesizers read digits badly) and pushed into the announcement pipe. A timer fires it in the morning. He speaks first; that's the whole feature.

Keeping it alive

• It runs as a user service under systemd, restarts on failure, and starts with the desktop session.
• A watchdog counts seconds since the last audio block from the mic. Past twenty, the process kills itself so the service manager brings it back with a fresh stream. Deaf but alive is the worst failure mode a voice assistant has, because it looks exactly like working.
• The restart greeting doubles as the alert. If he suddenly says hello in the afternoon, I know the mic died.

The failure museum

Everything above sounds tidy. It wasn't.

• The threshold night. Covered above. Log your scores before you tune anything.
• Echo cancellation mangled speech. With the fancy echo cancel pipeline on, "Hey Jarvis" transcribed as "Use your wrist" and an enrollment clip came back as "Thank you very much." I pointed capture at the raw mic and never looked back.
• "Go back to sleep" killed him. I'd wired the sleep phrase to the exit routine, so he took the instruction literally and shut down the whole program. Sleep should pause the loop, not end the process.
• The 13 minute coma. The mic stream died silently after an audio device change and he sat there looking alive. That bug is why the watchdog exists.
• Movie audio kept him awake forever. Background speech kept renewing his listening window all night. Fix: the window has a hard age limit no matter how noisy the room is.
• Short commands got dropped. My minimum speech length was tuned for sentences, so "yes" didn't qualify as speech. Every threshold cuts both ways; test the short case.

What it costs

• Wake word, VAD, transcription, voice, orb: free, open source, running on the CPU I already own
• The agent: the subscription I already had for coding
• Total new spend: zero

If you build one this week

1. Day one, ears only. Mic to wake word to transcription, printing to a terminal. No intelligence yet. Tune your thresholds against your real room with your real voice; my numbers are mine, not yours.
2. Day two, the brain. Pipe transcripts to an agent CLI in headless mode and print replies as text. Get the permission model and the personality prompt right before audio ever enters the picture.
3. Day three, the voice. Add TTS with render ahead streaming. This is the day it starts feeling like a person.
4. Day four, the glue. State file, announcement pipe, systemd service, watchdog. This is the day it becomes furniture instead of a demo.
5. Day five, the orb. Contributes nothing. Build it anyway. You'll know why the first time it assembles out of the dark.

He still tells me I have twenty one projects unfinished. Building him made it twenty two.