Hermes plugin hackathon: bounded round-3 trial judgment
Trial places: 1 ratcatcher (27), 2 omlx (26), 3 pit-crew (26), 4 idm-radio (25), 5 stageglass (24). Runners-up: 1 brainvault (24), 2 clack (21), 3 token-dashboard (19), 4 pow-tracker (16), 5 apphitlist (16). These are this write-up’s ordering of the ten evaluated entrants, using the unchanged seat scores in [T1] and [T2]; they are not official Nous awards or a judgment of the entire field.
The pinned field separates done-bell and homeassistant into refs; neither is a competitive entrant. Its entrant list contains the ten trial contestants among many others. The judging plan, “Why it is built this way,” explains that references measure seat grading. They retain comparison scores below and receive no prize or runner-up slot. No results for the remaining field were supplied; full-field winners cannot be established here.
[T1] and [T2] denote supplied .imd/reads/artifacts/item-trial-1 and item-trial-2 JSON results. Each row identifies its seat and pinned source. Runtime/test claims are attributed seat observations, not rerun for this write-up.
The brief’s score order is Useful / Reliable / Polish / Creative / Beyond a prompt; all are integers 1–5. Total = U + R + P + C + 2B, maximum 30. No scores are averaged or recalibrated. Total ties use higher Reliable, then Useful, then Beyond, then Polish, then Creative. Residual ties use demonstrated functionality, then alphabetical name. Thus omlx beats pit-crew on Reliable; stageglass and brainvault remain equal on components and their distinct working paths do not support a strict functional preference, so alphabetical order puts stageglass fifth. Pow-tracker’s demonstrated billing/CSV API path beats apphitlist’s unavailable remote tools. These are editorial trial tie-breaks, not Nous rules.
Both seats report successful Hermes installation with uv sync --locked --python 3.14 and CLI install/validate/doctor availability. T1 used Hermes ddc5d27eb38ec60f098b2168e48e7356cb4735a0, Python 3.14.10; T2 used 8ec227311bbbe7863d3982ce8f35012d6a59e7d7, Python 3.14.4. Both used supplied runners, real AIAgent and a fake model server. These different core pins limit direct comparability. Empty initial discovery diffs and returned unknown-tool messages are not proof of functioning plugin tools. [T1: setup; T2: setup]
| Trial rank / plugin / evidence | Hermes installed on seat | Catalog check | Loaded inside Hermes; planned calls and supplemental probes | U/R/P/C/B → total |
|---|
Place 1: ratcatcher (PR 151395; T1: ratcatcher) | Yes, T1 | Pass; safe scan, matching declarations; in-process Telegram handler required. | Loaded: 4 tools, 1 hook. Five calls: scam scored 1.0/remove; benign warning 0.34/second look; empty report/disconnected; common-domain blocklisting refused; empty text error. No live moderation. | 4/4/5/4/5 → 27/30 |
Place 2: omlx (PR 150815; T2: omlx) | Yes, T2 | Pass; no warnings; live provider client requires in-process execution. | Provider loaded, no agent tools. CLI discovery is not inference. Real Hermes web API: status 200/unsupported; start 400/Apple Silicon requirement; invalid download 422. No native generation. | 4/4/5/3/5 → 26/30 |
Place 3: pit-crew (PR 151709; T2: pit-crew) | Yes, T2 | Pass; private core import warnings. | Loaded: 6 tools, 3 hooks. Status, history and readiness returned labelled simulation; lock returned dry_run; fly returned invalid/audit. Follow-up disable/list/unknown-rule calls worked. No car contacted. | 4/3/5/4/5 → 26/30 |
Place 4: idm-radio (PR 152819; T1: idm-radio) | Yes, T1 | Pass; no warnings. | Loaded: 1 tool, 2 hooks. Status worked; play/stop honestly returned desktop_received=false; dance rejected by schema. Demo WAV rendered; real Desktop playback/export untested. | 3/3/5/4/5 → 25/30 |
Place 5: stageglass (PR 153194; T1: stageglass) | Yes, T1 | Fail: capability probe produced no result despite embedded registration JSON. Enable/doctor pass; Reliable capped at 2. | Loaded: 7 hooks, no tools. README/missing-file calls completed via Hermes built-ins. Supplemental registered /onair on/hold/invalid calls and HTTP snapshot verified hook feed and clearing; OBS untested. | 4/2/4/4/5 → 24/30 |
Runner-up 1: brainvault (PR 152550; T2: brainvault) | Yes, T2 | Pass scanner; manual review finds catalog rule-11 violation in key storage. | Memory provider loaded; README-required provider setting exposed 11 tools. Status unpaired; remember/recall preserved text; invalid wallet rejected. No pairing/mint/restore. | 4/2/4/4/5 → 24/30 |
Runner-up 2: clack (PR 149183; T2: clack) | Yes, T2 | Pass scanner; manual review finds catalog rule-11 violation in file credentials. | Loaded 9 tools. whoami unenrolled and reconcile not pending worked; poll(limit=0) rejected by schema; send safely failed before enrollment. No live delivery/ack. | 3/2/4/4/4 → 21/30 |
Runner-up 3: token-dashboard (PR 149187; T2: token-dashboard) | Yes, T2 | Pass; JS-only capability probe skipped. CLI doctor fails: no init.py. | CLI plugin not loaded (no init.py); JS Desktop path unverified. Empty chat completed; discovery probes were not plugin calls. Stubbed Node VM confirmed pane/chip registration and helper logic. | 4/2/3/2/4 → 19/30 |
Runner-up 4: pow-tracker (PR 150003; T1: pow-tracker) | Yes, T1 | Pass; ignored top-level dashboard manifest field warning. | Loaded, intentionally 0 tools/hooks. Built-in file calls completed, not billing tests. Supplemental router: client/task/time/invoice/CSV succeeded, reversed span 422; client active field corrupted by folder string. | 3/2/3/2/3 → 16/30 |
Runner-up 5: apphitlist (PR 151933; T2: apphitlist) | Yes, T2 | Pass; portable MCP manifest/security checks. | Enabled; doctor 0 native tools/hooks. Portable endpoint/skill only. search_apps and empty get_app both unknown-tool errors: neither reached service. No usable research call. | 3/2/3/2/3 → 16/30 |
Reference only: homeassistant (PR reference; T1: homeassistant) | Yes, T1 | Pass; private gateway helper warning; trusted_inbound disclosed. | Loaded: 4 tools, gateway registration. Missing token hid tools: list/get unavailable, malformed third call undispatched. Direct traversal/shell_command refused; valid reads returned connection errors. No device control. | 4/4/4/2/5 → 24/30 |
Reference only: done-bell (PR reference; T1: done-bell) | Yes, T1 | Pass; no warnings. | Loaded: 1 tool, 4 hooks. Four status/mute calls worked, including coercing string false; persisted completion event checked. OS effects suppressed; Desktop untested. | 4/4/3/2/4 → 21/30 |
Evidence and judging rationale (each paragraph is sourced to the named plugin object in its trial result; source pins are in the table):
ratcatcher [T1]. Best: A real pre-agent Telegram bouncer combines deterministic scam rules, model-assisted review and human-approved administrative changes with reversible moderation and strong local tests. Worst: Default enforce mode guards every group and can ban on sight, while the benign warning probe still triggered second-look review, so mistakes and model cost need careful operator tuning. Checks: 226 pytest passed; live Telegram/model review unverified.
omlx [T2]. Best: Makes local Apple Silicon inference a managed Hermes provider and Desktop workflow, with profile-scoped storage, credential/ownership guards, cancellation and validated model selection. Worst: Actual runtime installation, generation and rendered Desktop operation remain unverified here because macOS 15+ Apple Silicon is required, and the provider cannot run under strict host-process isolation. Checks: 30 pytest passed plus real web API probes; native hardware path unavailable.
pit-crew [T2]. Best: A complete proactive hardware companion combines persistent alerts, trip readiness, freshness-aware multi-source state, a simulator and approval-aware commands with Desktop and dashboard pages. Worst: The large test suite still has eight failures in this Hermes environment and admission warns about private core imports; simulator success cannot establish safe behavior against a real car or every supported integration. Checks: 833 passed, 8 failed, 79 skipped; failures include rule readback, alert and threshold cases. Real-Hermes follow-up resolved rule-disable probe, not all suite failures.
idm-radio [T1]. Best: It makes the agent's actual work audible and visible through Desktop events, synthesis and saveable session tracks, with a complete visual identity and no remote audio service. Worst: The TypeScript check fails in promotional code and there is no assertion suite; successful CLI control responses cannot establish that Desktop playback and WAV export work. Checks: No assertion suite; typecheck failed with two TS2345 errors; demo build and non-silent WAV render passed.
stageglass [T1]. Best: An opt-in audience feed observes real agent and approval hooks while keeping raw session text out of the overlay through layered redaction and fail-closed holds. Worst: Catalog admission fails at the capability probe in this environment despite successful doctor and live overlay probes, and novel secret formats still depend on heuristic detection. Checks: 22 unittest passed after isolating shadowed test namespace; real-Hermes overlay probes passed.
brainvault [T2]. Best: Extends Hermes memory itself with working local capture/recall, provenance and secret screening, plus an implemented encrypted, fingerprint-checked path toward owner-held portable memory. Worst: _core.py directly reads decrypt-only secrets from agent.key or OS keyring instead of Hermes secret scope, violating catalog rule 11; its distinctive pairing/mint/restore flow still requires an external wallet and was not demonstrated here. Checks: 32 passed, 3 live tests skipped.
clack [T2]. Best: Adds encrypted agent coordination with durable outbox reconciliation, explicit acknowledgements and independent relay/inviter verification rather than merely messaging instructions. Worst: client.py directly reads identity.key and decrypts a service token from identity.json instead of Hermes secret storage, violating catalog rule 11 despite careful file permissions; unconfigured send also hides the actionable cause behind a generic failure. Checks: 38 unittest passed with documented runner; initial pytest harness errors resolved.
token-dashboard [T2]. Best: Surfaces focused-session usage and cost in Hermes Desktop without sending data elsewhere, and honestly avoids inventing historical session token totals. Worst: No shipped tests or demo prove the Desktop path, CLI Doctor cannot load this JS-only plugin (conservative Reliable cap 2), and unknown models use a generic estimated rate while the visible status chip shows tokens rather than the README-promised always-visible cost. Checks: No shipped tests; syntax check and 6 stubbed VM assertions passed.
pow-tracker [T1]. Best: Local per-client SQLite records, opt-in file-activity capture and authenticated invoice exports make a useful dashboard for freelancers. Worst: Creating a client stores its folder path in active, making an inactive client truthy in the UI, and the passing tests miss this basic multi-client workflow bug. Checks: 30 unittest passed; supplemental billing probes exposed client active-field defect.
apphitlist [T2]. Best: Gives indie app developers a clearly documented remote route to app-market, advertising and complaint research that a prompt alone cannot retrieve. Worst: The pinned package is mostly setup documentation and a skill around a closed remote service, and no research tool was usable in the credential-free Hermes run; Reliable is capped at 2 for that observed runtime gap. Checks: No tests or local server source; unknown-tool responses do not demonstrate service reliability.
homeassistant [T1]. Best: A full gateway platform turns the user's own home events into agent input and supports constrained device tools and cron notifications through Hermes secret handling. Worst: Real device and reconnect behavior remains unverified here, and the catalog check flags reliance on a private gateway helper despite otherwise extensive mocked coverage. Checks: 212 passed, 4 skipped; mocked coverage, no live HA.
done-bell [T1]. Best: A small, credential-free extension that makes turn completion and approval waits visible through real lifecycle hooks, notifications and a Desktop status chip. Worst: The README's installation path is personal and Windows-specific, with a manual Desktop copy step and no demonstrated cross-platform notification delivery. Checks: 9 pytest passed; no OS/Desktop end-to-end test.
Ratcatcher terminology check: the planned refusal was common-domain blocklisting, not allowlisting. At the pin, tools.py routes blocklist additions through clean_entry; store.py protects github.com from blocklisting to avoid removing ordinary links. This corroborates T1’s refusal observation.
Seat blockers and limits, one line per supplied seat:
- T1: No seat-wide blocker (
blocked=null for every entry); no live Home Assistant, Telegram or Hermes Desktop, so device control, actual moderation, OS notifications and real Desktop output remain unverified. Stageglass admission failed; idm-radio typecheck failed; pow-tracker API probes found a functional defect. [T1: setup and plugin objects]
- T2: No seat-wide blocker (
blocked=null throughout); no Apple Silicon/macOS inference hardware, real car, rendered Desktop or authenticated remote-service/pairing setup. AppHitlist tools were unavailable; clack delivery and BrainVault wallet restore were not exercised. Test harness/dependency issues were resolved as reported; pit-crew’s eight suite failures remain. [T2: setup and plugin objects]
Passing admission does not certify policy compliance: T2’s secret-storage findings remain adverse evidence. Neither supplied seat reported judge steering; no independent exhaustive audit is claimed. Full-round aggregation and official winners were not supplied.