System health monitoring combines a process-owned backend host sampler, an on-read storage capacity coordinator, a Companion local-drive capacity cache, and a frontend state hook that merges visible sources in RAM.
Keep /api/health in backend/app/main.py unchanged as the lightweight Docker container liveness check. Resource health lives under /api/health-monitoring/* and /api/admin/settings/health-monitoring.
Sampling and Storage Architecture
- Host sampler (
5sinterval,15sstale threshold):HealthMonitoringCoordinatorstarts and stops in FastAPIlifespan. Every 5 seconds on a worker thread, it samples/host/proc/meminfo(MemAvailable),/host/proc/loadavgnormalized by logical CPU count from/host/proc/cpuinfo(exposing normalized 1-, 5-, and 15-minute values only and evaluating the 5-minute value),/host/sysCPU thermal candidates (coretemp,k10temp, and verified Raspberry Picpu-thermal), and container memory from/sys/fs/cgroup. It performs zero SQLite queries and zero storage capacity probes inside the 5-second loop. - Container memory accounting:
used_bytesand percentage alerts exclude file cache:memory.current - (file - shmem)on cgroup v2.cache_bytesis the excludedfile - shmem. Shared memory, tmpfs, and kernel allocations remain included. Cgroup v1 usesmemory.usage_in_bytes,total_cache, andtotal_shmem; hierarchical counters avoid mixing parent and child scopes. Missing, malformed, or inconsistent counters produce Unknown rather than a guessed value. The API retainsworking_set_bytes = max(0, current - inactive_file)for compatibility, but the overview and thresholds useused_bytes. All readings cover the container, not only Sambee. - On-read storage coordinator (
60sinterval,180sstale threshold): Snapshot requests (GET /api/health-monitoring/snapshot) accept at most twoconnection_idquery parameters (422when more than two are supplied), filter connections withcan_view_connectionbefore scheduling probes or computing summaries, return current RAM state immediately, and schedule non-blocking background probes only when an entry is missing or at least 60 seconds old and not in flight or retry backoff. - Concurrency and rate bounds: Backend storage probes are staggered (
150 ms), capped at two concurrent worker slots per process, rate-limited per user for new SMB triggers (4per60seconds), and stored in a32-entry bounded SMB LRU cache invalidated on connection edits, deletions, or SMB policy changes. Admission evicts only entries without a check in flight. If every entry is busy, the response reports Unknown with a capacity-wait message, without scheduling a task or consuming a trigger token. - Probe admission: An event-loop-owned
asyncio.BoundedSemaphorecontrols backend worker slots. A cancelled probe that has not started a worker clears its in-flight flag and releases any acquired slot. Once a worker starts, its completion callback owns slot release, including after a timeout or cancellation. - CPU summary selection: All candidate paths use the same severity ordering: Error, Warning, Unknown, OK, then Informational. A failed package cannot be hidden by a healthy package; missing optional limits do not by themselves make a successful reading Unknown.
- Optional readings: Unconfigured host mounts and CPU temperature without a supported sensor return Informational and are excluded from required checks. Read errors, lost mounts, and stale readings remain Unknown. The frontend shows an informational Local drives placeholder when Companion is not connected or paired; pairing errors and failed capacity requests remain required Unknown checks.
Non-Aborting SMB and Companion Capacity Probes
- Backend SMB (
SMBBackend.get_volume_capacity): Queriessmbclient.stat_volume()overSMBConnectionPool.get_connection(). On timeout or request cancellation, it callsretain_connection_until_future_complete()and keeps both its 2-probe slot and per-connectionin_flightflag owned until the executor thread exits (add_done_callback). Unlike mutating file operations, it never callsinvalidate_connection_cache(..., abort=True)and never disconnects active file reads, writes, moves, or archive streams sharing the pooled connection. - Connection changes: A mismatched credential/path fingerprint hides the old reading. The coordinator retains the old entry while its worker is in flight and returns an uncached Unknown result for the current context. Once the worker exits, a later snapshot can replace the entry and schedule the new context.
- Companion local drives (
GET /api/drives/capacity):drives.rspopulates an in-memoryDRIVE_ROOT_MAPduringenumerate_drives()(re-enumerating at most once on an unknown drive ID) and shares a#[cfg(unix)]libc::statvfshelper across Linux and macOS alongside WindowsGetDiskFreeSpaceExW.DriveCapacityCoordinatormoves both the 2-probeOwnedSemaphorePermitand theDriveInFlightGuardintotokio::task::spawn_blockingso a wedged network or removable mount retains its slot until the OS thread exits without spawning duplicate threads. Virtual drives reporting zero total blocks map tounsupportedrather than zero free space.
Settings, Validation, and Support Report Redaction
- Numeric thresholds are persisted and transmitted as exact scaled integers matching
IntegerSystemSettingDefinition: tenths of a percent (1..1000for memory,0..999for available space), hundredths (>= 1up toMAX_WIRE_INTEGER) for normalized 5-minute load, and integer millidegrees in100-millidegree (0.1 °C) steps for CPU overrides. Inputs with fractional excess precision or wrong types are rejected without rounding. - Per-source CPU overrides use discriminated
StrictSettingUpdatevariants (cpu_temp_warning_overrideandcpu_temp_error_override) carrying{ source_id, value_millidegrees }. Only currently discovered CPU sources or existing persisted override rows (health_monitoring.cpu_temp_override.<sha256>) are accepted so arbitrary keys cannot create unbounded database rows. - The settings page uses
useSystemSettingPersistencefor per-field save state, deadlines, cancellation, and response matching. A small adapter gives each CPU source and threshold a separate field key. Ordinary save responses update only the confirmed field. CPU saves trigger a separate, cancellable metadata read that updates only CPU details and preserves other drafts and confirmed thresholds. A failed metadata read reports that the save succeeded and offers Retry without repeating the write. build_public_support_report_read()exports[ui_configuration.health_monitoring]and aliases every persisted CPUsource_idviaPublicSupportReportAliases(cpu-thermal-source-N) without including raw sysfs paths or live metric readings.
RAM-Only and Browser Lifecycle Rules
- All
/api/health-monitoring/*and Companion/api/drives/capacityresponses sendCache-Control: no-store. - Neither the backend nor the frontend writes metric samples to SQLite,
localStorage,sessionStorage, or log payloads; backend logs record only high-level severity transitions. useSystemHealthMonitoringpolls the server snapshot and Companion capacity independently while the tab is visible. The normal intervals are 15 seconds for the indicator and 5 seconds for the dialog. Each request has an 8-second deadline. Retries start at 2 seconds and back off to 30 seconds; a check in progress retries no later than its normal interval.- The browser keeps valid readings in RAM until their freshness limits, using the age reported by each source and elapsed time since receipt. After a sleep or background gap, it refreshes and shows the previous status for up to 20 seconds without presenting old readings as current. Wall time helps detect a gap when the elapsed-time clock pauses; it does not set freshness.
- Display policy is refreshed every 60 seconds, on opening the dialog, on return to a visible tab, and after a health settings change. A temporary policy failure can use the last valid policy for up to 120 seconds from its receipt. Sign-out, identity changes, and confirmed denial discard health data immediately. Disabled and unauthorized users make policy requests only.
- After monitoring is re-enabled, the paired drive check stays unknown until Companion returns a capacity response. A successful capacity response also clears a previous update prompt.
- Hidden tabs and signed-out browsers start no monitoring requests. Browser metric state is cleared on
authSessionsign-out or identity change.