JARVIS, assembled at home.
A phased, no-cloud plan for a voice-controlled house — lights, temperature, watering, background music — with a local language model as the brain, running on hardware I already own.
The machine, measured first
Before planning anything, I benchmarked my local LLM server. The numbers decided the architecture.
Why short conversations
Time to first token as the conversation grows. Every tool call re-sends the whole context.
The rule falls out of the data: voice turns stay tiny (a rolling ten-message window), the prompt prefix stays stable so the block cache keeps working, and the model only ever handles one stream at a time.
Architecture
Always-on box — Mac mini
- Home Assistant OS (VM)
- Zigbee radio → lights, sensors, water valve
- MQTT broker · ESPHome · Music Assistant
- Speech-to-text (Whisper)
- Voice satellites (wake word) in each room
- Local intents answer when the brain sleeps
The brain — MacBook Pro M3 Max, 128 GB
- Local LLM server, OpenAI-compatible
- Free-form conversation & Q&A
- Text-to-speech voice
- Future: portfolio & personal-data tools
The house never depends on the brain box being awake — only conversation does.
Build phases
0 · Prep one evening, $0
Identify hardware, never-sleep settings, static IPs, fix model config.
1 · Nervous system weekend 1
Home Assistant OS, Zigbee coordinator, MQTT, backups, remote access.
✓ add your first smart plug from your phone2 · Lights weekend 2
One pilot room: dimmable fixtures, room groups, scenes.
✓ "living room to 40%" works3 · Climate weekend 2, half-day
Temperature/humidity sensors per room, history graphs, proactive alerts.
✓ "is the study too dry?" answers with real data4 · Water weekend 3
Soil sensors + motorized garden valve; leak sensors under every wet fixture.
✓ scheduled watering runs 3 days unattended5 · Music weekend 3
Music library per zone; scenes like "dinner" = lights + playlist + volume.
✓ one sentence sets the whole room6 · Voice — the JARVIS moment weekends 4–5
Wake-word satellites, Whisper STT, local LLM, spoken answers. Budget: ≤3 s for commands, ≤5 s for chat.
✓ "lights off, is the garden watered? some jazz." — from the couch7 · Personality ongoing
Morning briefings, presence-based scenes, personal-data integrations, per-voice ID.
Shopping list — one-room pilot
| Item | Qty | ≈ Cost |
|---|---|---|
| Zigbee USB coordinator | 1 | $30 |
| Smart bulbs or Zigbee relay inserts | 3 | $45 |
| Temperature / humidity sensors | 3 | $40 |
| Soil-moisture sensors | 2 | $30 |
| Motorized ball valve + controller | 1 | $60 |
| Leak sensors | 3 | $45 |
| Voice satellite (ESP32-S3 or HA Voice PE) | 1 | $30–60 |
| Used zone speaker | 1 | $30–100 |
| Small UPS | 1 | $50 |
| Pilot total | $310–410 |
Rules carried from the benchmark
- One model stream. The server collapses past two concurrent requests — everything queues politely.
- Short contexts. A rolling ten-turn window keeps first-token latency under a second.
- Stable prompt prefix. The block cache only works if the session start never changes.
- The always-on box never reboots during occupied hours. It is the house.