Building Jarvis as a trustworthy home agent: calendar briefings, device control, and lessons from false alerts
# A field report on separating command transmission, software-known state, and physical verification across calendar, voice, and household-device workflows.
The goal for Jarvis is simple to describe: speak at home, have the system read the calendar, operate a device, and send a notification when needed.
That description makes a home agent sound like a contest in conversational intelligence.
Building one taught us the opposite lesson.
The difficult part is reporting exactly what was verified after a command leaves the conversation.
This article was checked on July 21, 2026 against configuration, prior execution records, and responses from running APIs.
Connection details, credentials, and personal information are intentionally omitted.
What Jarvis connects
Jarvis is not a particular smart-speaker product here.
It is the name of a home agent built around Hermes Agent that connects conversation, schedules, audio, and household devices.
Its job has four parts.
• Interpret whether a request means research, notification, or physical action.
• Retrieve Google Calendar events and turn them into a spoken briefing.
• Route actions to separate paths for speakers, television, air conditioning, and garden equipment.
• Report failure or missing verification instead of hiding it.
Photo: prototype control hardware. The display was masked because it contained a personal schedule. The image alone does not verify any connection target or physical device action.
We confirmed a scheduled job intended to retrieve the day's events at 8:00 every morning and prepare the spoken text.
We also confirmed configuration and prior records for registration and cleanup through the audio API.
During this review, we did not verify those operations succeeding or the physical speaker actually producing sound.
“A morning briefing is configured” and “the speaker played it this morning” are different facts.
That distinction is the foundation for every automation in this series.
Completion has four levels
Calling every device action a success hides important uncertainty.
Jarvis therefore treats completion as a ladder.
Level
What it establishes
What remains unknown
Request accepted
The conversation or button input was understood
Whether it reached the execution path
Command sent
The API or infrared sender responded
Whether the device received it
Known state changed
Software updated its stored state
Whether the room or appliance physically changed
Physical result observed
A sensor or independent observation detected the change
Whether it will remain reliable over time
This model makes an honest response possible: “The command was sent; the physical result is not yet verified.”
It is less polished than a confident completion message, but much safer in a real home.
Keep voice paths separate
One voice path combines speech recognition and text-to-speech.
Another handles a direct real-time speech-to-speech session.
Running both for the same conversation can produce duplicate replies or let the wrong audio engine interrupt.
The normal voice path and real-time path therefore remain separate.
An alarm or calendar briefing is also not complete merely because text reached Discord.
If the requirement is audible playback, speaker playback is part of the job.
What a false intercom alert taught us
There was one report of an intercom notification appearing even though the bell had not rung.
The cause remains unconfirmed.
The dangerous response would be to choose a plausible cause and implement a fix as though that cause were established fact.
Sensor bounce, network retries, and stale state are all possible, but the existing record does not select one.
The next occurrence needs a better trace: detection time, input value, decision reason, notification ID, and retry count joined by one correlation ID.
Until false positives are understood, the message should say “signal detected,” not “visitor confirmed.”
Build the stop path before expanding control
Jarvis can reach functions with very different risk profiles: volume changes, infrared air-conditioner commands, a watering pump, and a bath switch.
They should not share one permission level or one definition of success.
• Change volume in small steps.
• Send complete air-conditioner state signals and rate-limit them.
• Give a pump a maximum runtime and a manual stop.
• Do not retry a bath toggle while its state is unknown.
• When a sensor or execution path is uncertain, default to no action.
For home automation, stopping conditions matter more than the ability to start.
Automation that cannot be stopped is not convenience; it is a new failure mode.
An honest snapshot of the system
As of July 21, 2026, we confirmed configuration for an 8:00 JST Google Calendar briefing and audio-API registration and cleanup, a prior invocation of the bath-toggle route, reported garden API state, and the air conditioner's software-known state cool_24.
We did not verify the calendar or audio operations succeeding during this review, physical speaker playback, infrared reception and room-temperature change, bath-fill start or completion, actual watering, or long-term automatic watering.
Closing that gap is not mainly an AI-model problem.
Each device needs independent observation, commands and outcomes need a shared trace, and Jarvis must preserve the word “unverified” when that is the truth.
A fluent agent that says “done” is less trustworthy than one that can say, “The command was sent, but I cannot yet confirm the physical result.”
