daemon status exposes no per-peer sync health (last-sync/staleness) #15

New issue

Closed

opened 2026-07-16 04:09:49 +00:00 by hugooconnor · 1 comment

hugooconnor commented

2026-07-16 04:09:49 +00:00

(Migrated from codeberg.org)

Summary

daemon status [--json] exposes only local, session-scoped counters — there is no per-peer/per-member sync health (last-successful-sync time, last error, staleness). A node cannot locally tell that it has diverged from a peer.

Repro

On machine A, daemon status --json reported healthy counters (entries_merged: 35, closures_run: 154, uptime_s: 9518) and 67 entries — with no indication of peer state. Concurrently machine B reported its daemon Stale with sync session failed: transport: connection lost. From A's status alone, the degraded link and possible A→B divergence were invisible.

Expected

daemon status should include, per theory and per member:

last_sync_ok timestamp (and last_sync_error + message),
a stale / healthy flag derived from the sync interval,
so either side can detect "I have not successfully synced with peer X since T".
Note entries_merged resets per daemon session, so it is not a convergence signal; a per-peer high-water mark would be.

Impact

When the QUIC/relay path drops, both agents believe they are fine while silently diverging; the only current recourse is comparing fingerprints out-of-band.

Env

elephant v0.1.2 (HEAD 4d58f06), macOS. FLF two-machine (Arm 2) Rootclaim run.

### Summary `daemon status [--json]` exposes only local, session-scoped counters — there is no per-peer/per-member sync health (last-successful-sync time, last error, staleness). A node cannot locally tell that it has diverged from a peer. ### Repro On machine A, `daemon status --json` reported healthy counters (`entries_merged: 35`, `closures_run: 154`, `uptime_s: 9518`) and 67 entries — with **no** indication of peer state. Concurrently machine B reported its daemon `Stale` with `sync session failed: transport: connection lost`. From A's status alone, the degraded link and possible A→B divergence were invisible. ### Expected `daemon status` should include, per theory and per member: - `last_sync_ok` timestamp (and `last_sync_error` + message), - a `stale` / `healthy` flag derived from the sync interval, so either side can detect "I have not successfully synced with peer X since T". Note `entries_merged` resets per daemon session, so it is not a convergence signal; a per-peer high-water mark would be. ### Impact When the QUIC/relay path drops, both agents believe they are fine while silently diverging; the only current recourse is comparing fingerprints out-of-band. ### Env elephant v0.1.2 (HEAD 4d58f06), macOS. FLF two-machine (Arm 2) Rootclaim run.

anuna-02 commented

2026-07-16 04:28:24 +00:00

(Migrated from codeberg.org)

The skeptic-side run provides two additional implementation-level observations for this issue:

rc-live outbound sessions repeatedly failed with transport: connection lost, but the cadence was also obscured by unrelated peer timeouts: with ELEPHANT_SYNC_INTERVAL=30, attempts for this theory were about 94-96 seconds apart. The scheduler/head-of-line aspect is tracked separately in #18.
A remote merge did occur during the run (the local journal advanced from 26 to 48 entries), but status still had no record of the successful peer/session. The current refresh path cannot account for it: note_synced sends only the theory id over RefreshTx, dropping applied, while entries_merged is incremented only by the HTTP merge_and_close path. Therefore neither successful remote imports nor their entry count can appear in status today.

Suggested status fields per (theory, peer):

last_attempt_at, last_success_at, and stage-specific last_error
local and last-seen peer version-vector fingerprint/high-water mark
entries imported, entries exported, and quarantined entries
current retry/backoff and derived health/staleness

Successes should also produce a structured session log; currently only failures are visible, so the inbound convergence event cannot be reconstructed from daemon logs alone.

The skeptic-side run provides two additional implementation-level observations for this issue: 1. `rc-live` outbound sessions repeatedly failed with `transport: connection lost`, but the cadence was also obscured by unrelated peer timeouts: with `ELEPHANT_SYNC_INTERVAL=30`, attempts for this theory were about 94-96 seconds apart. The scheduler/head-of-line aspect is tracked separately in #18. 2. A remote merge did occur during the run (the local journal advanced from 26 to 48 entries), but status still had no record of the successful peer/session. The current refresh path cannot account for it: `note_synced` sends only the theory id over `RefreshTx`, dropping `applied`, while `entries_merged` is incremented only by the HTTP `merge_and_close` path. Therefore neither successful remote imports nor their entry count can appear in status today. Suggested status fields per `(theory, peer)`: - `last_attempt_at`, `last_success_at`, and stage-specific `last_error` - local and last-seen peer version-vector fingerprint/high-water mark - entries imported, entries exported, and quarantined entries - current retry/backoff and derived health/staleness Successes should also produce a structured session log; currently only failures are visible, so the inbound convergence event cannot be reconstructed from daemon logs alone.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

anuna-research/elephant#15

No description provided.

Rows
Columns