anuna-research/hark
1
0
Fork 0
Code Issues 3 Pull requests 1 Projects Packages Wiki Activity Actions

feat(chat): auto-reconnect the daemon on hub drop, without losing handle state #18

Open
hugooconnor wants to merge 12 commits from feat/auto-reconnect into main
pull from: feat/auto-reconnect
merge into: anuna-research:main
Conversation 0 Commits 12 Files changed 13 +2784 -273
hugooconnor commented 2026-07-17 04:41:59 +00:00 (Migrated from codeberg.org)
Copy link

Why

The hub (cbcl-bus) is a single instance with an immediate deploy strategy — it must be, because its persistence is single-writer (Litestream → one bucket + Mnesia → one node). So every redeploy replaces the machine and drops all sockets at once (hub_closed / "peer closed without TLS close_notify" across every handle simultaneously).

Today the chat receive loop just marks the handle unhealthy and ends. The operator then bounces the daemon — which wipes handle state (agents: 0) — and since pairing codes are one-time, the codes on hand are now spent (UNKNOWN_PAIRING), forcing a full re-pair of every agent. A routine hub restart shouldn't cost the whole agent fleet.

What

Ride through hub restarts automatically, preserving handle + MLS state.

  • Reusable connect/join. Factor the connect → bootstrap → signed hello → join-ack → MLS on_roomcfg/(create)/join_frames → announce sequence into connect_and_join(), preserving the exact frame order and every validation (downgrade refusal, join-ack gating, UndeclaredDialect, CBCL announce self-check). Called for the first connect and re-run verbatim on each reconnect.
  • Typed exit reasons. run_receive_loop returns LoopExit { Stopped, LocalClosed, HubClosed }. spawn_receive_loop wraps it in an outer loop that reconnects only for HubClosed, reusing the same identity, MlsSession, responder, store handle, and both control channels.
  • Backoff 1s → 30s, reset on success, retry indefinitely; an explicit daemon stop during backoff aborts cleanly.
  • MLS on reconnect. Same MlsSession reused; mls_create is false on every reconnect so the group is created at most once — re-creating would mint a new group_id and fork every peer (WrongGroupId). join_frames re-publishes KeyPackages + idkey to the restarted hub, and the owner re-adds present members on presence.
  • Store health without a false stop. New AgentStore::mark_reconnecting reflects the outage without firing the connection's close signal. This is load-bearing: mark_unhealthy consumes/fires close_tx, which would resolve the loop's close_rx and be indistinguishable from an explicit stop — aborting the very reconnect we want. The HubClosed arms therefore no longer call mark_unhealthy; mark_connected restores health in place on rejoin. A unit test pins the invariant.

Testing

  • cargo build — clean
  • cargo clippy --all-targets -- -D warnings — clean
  • cargo test — 310 passed, 3 ignored (+1 new test: reconnecting_preserves_close_signal_then_connected_restores)

Companion change

The cbcl-bus web client gained matching auto-reconnect (exponential backoff, resume channels + MLS from persisted state, immediate retry on online/tab-focus), so a redeploy is a brief blip rather than a dead tab. This PR gives the daemon the equivalent.

🤖 Generated with Claude Code

## Why The hub (cbcl-bus) is a **single instance** with an *immediate* deploy strategy — it must be, because its persistence is single-writer (Litestream → one bucket + Mnesia → one node). So **every redeploy replaces the machine and drops all sockets at once** (`hub_closed` / "peer closed without TLS close_notify" across every handle simultaneously). Today the chat receive loop just marks the handle unhealthy and ends. The operator then bounces the daemon — which **wipes handle state** (`agents: 0`) — and since pairing codes are **one-time**, the codes on hand are now spent (`UNKNOWN_PAIRING`), forcing a full re-pair of every agent. A routine hub restart shouldn't cost the whole agent fleet. ## What Ride through hub restarts automatically, preserving handle + MLS state. - **Reusable connect/join.** Factor the `connect → bootstrap → signed hello → join-ack → MLS on_roomcfg/(create)/join_frames → announce` sequence into `connect_and_join()`, preserving the exact frame order and every validation (downgrade refusal, join-ack gating, `UndeclaredDialect`, CBCL announce self-check). Called for the first connect and re-run verbatim on each reconnect. - **Typed exit reasons.** `run_receive_loop` returns `LoopExit { Stopped, LocalClosed, HubClosed }`. `spawn_receive_loop` wraps it in an outer loop that reconnects **only** for `HubClosed`, reusing the same identity, `MlsSession`, responder, store handle, and both control channels. - **Backoff** 1s → 30s, reset on success, retry indefinitely; an explicit `daemon stop` during backoff aborts cleanly. - **MLS on reconnect.** Same `MlsSession` reused; `mls_create` is **false** on every reconnect so the group is created at most once — re-creating would mint a new `group_id` and fork every peer (`WrongGroupId`). `join_frames` re-publishes KeyPackages + idkey to the restarted hub, and the owner re-adds present members on presence. - **Store health without a false stop.** New `AgentStore::mark_reconnecting` reflects the outage **without** firing the connection's close signal. This is load-bearing: `mark_unhealthy` consumes/fires `close_tx`, which would resolve the loop's `close_rx` and be indistinguishable from an explicit stop — aborting the very reconnect we want. The `HubClosed` arms therefore no longer call `mark_unhealthy`; `mark_connected` restores health in place on rejoin. A unit test pins the invariant. ## Testing - `cargo build` — clean - `cargo clippy --all-targets -- -D warnings` — clean - `cargo test` — **310 passed, 3 ignored** (+1 new test: `reconnecting_preserves_close_signal_then_connected_restores`) ## Companion change The cbcl-bus **web client** gained matching auto-reconnect (exponential backoff, resume channels + MLS from persisted state, immediate retry on `online`/tab-focus), so a redeploy is a brief blip rather than a dead tab. This PR gives the daemon the equivalent. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/auto-reconnect:feat/auto-reconnect
git switch feat/auto-reconnect

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff feat/auto-reconnect
git switch feat/auto-reconnect
git rebase main
git switch main
git merge --ff-only feat/auto-reconnect
git switch feat/auto-reconnect
git rebase main
git switch main
git merge --no-ff feat/auto-reconnect
git switch main
git merge --squash feat/auto-reconnect
git switch main
git merge --ff-only feat/auto-reconnect
git switch main
git merge feat/auto-reconnect
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
anuna-research/hark!18
No description provided.