Live Logs — Observatory
levels:
| Time | Level | Message |
|---|---|---|
| 2026-03-11 09:07:34.413 | STATUS | BOR received — run #41. Created /data/runs/run41.h5 (SWMR). Analyst can read live. |
| 2026-03-11 09:07:34.433 | STATUS | BOR received — run #41. Snapshot: /data/runs/run41_bor.msgpack. |
| 2026-03-11 09:07:34.483 | STATUS | BOR received — run #41. TX at 800 Hz. |
| 2026-03-11 09:07:34.533 | STATUS | BOR received — run #41. Config snapshot written. |
| 2026-03-11 09:07:34.533 | STATUS | BOR received — run #41. Environment logging started. |
| 2026-03-11 09:07:34.633 | STATUS | ← start received. Recording. |
| 2026-03-11 09:07:34.733 | STATUS | ← start received. Recording active. |
| 2026-03-11 09:07:34.833 | STATUS | ← start received. Entering RUN. |
| 2026-03-11 09:07:34.833 | STATUS | ← start received. Event loop active. |
| 2026-03-11 09:07:34.933 | STATUS | ← start received. |
| 2026-03-11 09:07:38.333 | INFO | STAT/CPU_LOAD = 31.2 STAT/CURRENT_UA = 487.6 |
| 2026-03-11 09:07:40.333 | INFO | STAT/CPU_LOAD = 39.5 STAT/WRITE_RATE_MB = 7.9 |
| 2026-03-11 09:07:42.333 | INFO | STAT/WRITE_RATE_MB = 11.4 STAT/FILE_SIZE_MB = 42.8 |
| 2026-03-11 09:08:04.333 | WARNING | Temperature spike: 41.3°C exceeded threshold 40°C. Alert on CMDP UI/ topic. |
| 2026-03-11 09:08:09.333 | CRITICAL | Thermal alert unresolved. Entering SAFE state. HighVoltage.hv1 will autonomously ramp down via CHP observation. |
| 2026-03-11 09:14:13.333 | INFO | STAT/CPU_LOAD = 33.8 STAT/CURRENT_UA = 484.1 |
| 2026-03-11 09:14:13.833 | WARNING | Run condition degraded — TempSensor.ts1 reported failure mid-run. |
| 2026-03-11 09:14:14.333 | STATUS | EOR received — run #41 complete. 12,841 events. Condition: DEGRADED. |
| 2026-03-11 09:14:14.333 | STATUS | EOR received — run #41. 12,841 events. File finalised: run41.h5 (238 MB). |
| 2026-03-11 09:14:14.333 | STATUS | EOR received — run #41. 12,841 events. Condition: DEGRADED. Duration: 400s. |
| 2026-03-11 09:14:14.333 | STATUS | EOR received — run #41. Duration: 400s. TX avg: 798.2 Hz. |
| 2026-03-11 09:14:14.333 | STATUS | EOR received — run #41. Condition: DEGRADED (thermal fault). Duration: 400s. |
| 2026-03-11 09:14:14.533 | INFO | ⚡ Extrasystole emitted — RUN → ORBIT after EOR |
| 2026-03-11 09:14:14.533 | INFO | ⚡ Extrasystole emitted — RUN → ORBIT after EOR |
| 2026-03-11 09:14:14.633 | INFO | ⚡ Extrasystole emitted — RUN → ORBIT after EOR |
| 2026-03-11 09:14:14.833 | INFO | ⚡ Extrasystole emitted — SAFE → ORBIT (recovered) |
| 2026-03-11 10:05:32.333 | INFO | CHIRP offer beacon sent — TempSensor.ts1 (Python/Mariner) on 192.168.1.20:5558 |
| 2026-03-11 10:05:32.833 | INFO | CHIRP offer beacon sent — DevNullReceiver.dnr1 (C++/Sputnik) on 192.168.1.11:5559 |
| 2026-03-11 10:05:33.333 | INFO | CHIRP offer beacon sent — RandomTransmitter.tx1 (C++/Sputnik) on 192.168.1.11:5557 |
| 2026-03-11 10:05:33.533 | STATUS | ← initialize received. Lakeshore 218, 8 channels, RS-232 9600 baud. Reply: SUCCESS |
| 2026-03-11 10:05:33.833 | INFO | CHIRP offer beacon sent — FlightRecorder.fr1 (C++/Sputnik) on 192.168.1.10:5556 |
| 2026-03-11 10:05:33.933 | INFO | CHIRP offer beacon sent — EudaqNativeWriter.eudw1 (C++/Sputnik) on 192.168.1.10:5562 |
| 2026-03-11 10:05:34.133 | INFO | CHIRP offer beacon sent — H5DataWriter.h5w1 (C++/Sputnik) on 192.168.1.10:5561 |
| 2026-03-11 10:05:34.133 | STATUS | ← initialize received. Buffer: 64MB. Role: NONE. Reply: SUCCESS |
| 2026-03-11 10:05:34.333 | INFO | CHIRP offer beacon sent — HighVoltage.hv1 (C++/Sputnik) on 192.168.1.10:5555 |
| 2026-03-11 10:05:34.433 | INFO | CHIRP offer — Observatory subscribing to all CMDP topics (source-side filter: DEBUG+). |
| 2026-03-11 10:05:34.483 | INFO | CHIRP offer — InfluxForwarder subscribing to STAT topics → InfluxDB (source-side filter: INFO+). |
| 2026-03-11 10:05:34.533 | INFO | CHP heartbeat publisher started — interval: 2000ms (RPi reduced tick), maxLives: 3 |
| 2026-03-11 10:05:34.533 | INFO | Broadcast "initialize" → 7 satellites. All replied SUCCESS. |
| 2026-03-11 10:05:34.833 | STATUS | ← initialize received. event_rate_hz=1000, payload=256B. Reply: SUCCESS |
| 2026-03-11 10:05:35.033 | STATUS | ← initialize received. EUDAQ2 native. write_as_blocks=true, newtic_event_tag=ThorlabDataEvent. Reply: SUCCESS |
| 2026-03-11 10:05:35.033 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-03-11 10:05:35.133 | STATUS | ← initialize received. output_dir=/data/runs, max_file_size_mb=1024. Reply: SUCCESS |
| 2026-03-11 10:05:35.233 | STATUS | ← initialize received. HDF5 v1.14, SWMR enabled, POSIX file locks active. Reply: SUCCESS |
| 2026-03-11 10:05:35.333 | STATUS | ← initialize received. Reply: SUCCESS. Entering INIT. |
| 2026-03-11 10:05:35.533 | STATUS | ← launch received. Sensor sampling active on 8 channels. |
| 2026-03-11 10:05:35.833 | INFO | Keithley SMU config: bias_voltage=100V, max_current=500μA, ramp_rate=10V/s |
| 2026-03-11 10:05:35.933 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-03-11 10:05:36.033 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-03-11 10:05:36.033 | STATUS | ← launch received. Pausing — require_after: RandomTransmitter.tx1 must reach ORBIT. |
| 2026-03-11 10:05:36.133 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-03-11 10:05:36.233 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-03-11 10:05:36.333 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-03-11 10:05:37.033 | STATUS | ← launch received. Ready to write EUDAQ blocks. |
| 2026-03-11 10:05:37.033 | INFO | All ACK. Broadcasting "launch"… |
| 2026-03-11 10:05:37.133 | STATUS | ← launch received. Output: /data/runs/. Chunk: 1024. Compression: gzip. |
| 2026-03-11 10:05:37.133 | STATUS | ← launch received. PRNG seeded. Zero-copy allocator engaged. |
| 2026-03-11 10:05:37.233 | STATUS | ← launch received. File handle opened. Ready to record. |
| 2026-03-11 10:05:37.333 | STATUS | ← launch received. Ramping HV to 100V… |
| 2026-03-11 10:05:37.383 | INFO | DevNullReceiver.dnr1 pausing — require_after: RandomTransmitter.tx1 must reach ORBIT. |
| 2026-03-11 10:05:37.433 | INFO | Dependency resolved: RandomTransmitter.tx1 reached ORBIT. Resuming launch. |
| 2026-03-11 10:05:37.533 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-03-11 10:05:37.833 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-03-11 10:05:38.333 | INFO | HV ramp: 0V → 50V → 100V. Stabilised at 100.02V. |
| 2026-03-11 10:05:38.533 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-03-11 10:05:38.633 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-03-11 10:05:38.633 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-03-11 10:05:38.633 | INFO | Data grouping active — payloads <1KB auto-buffered. Saturating 10G link with minimal CPU overhead. |
| 2026-03-11 10:05:38.733 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-03-11 10:05:38.833 | INFO | ⚡ Extrasystole emitted — state change INIT → ORBIT detected |
| 2026-03-11 10:05:39.133 | STATUS | All 7 satellites ORBIT. Dependency graph resolved. Ready for run. |
| 2026-03-11 10:06:49.233 | STATUS | BOR broadcast — run #42. User tags: beam=120 GeV/c π⁺, detector=FEI4 quad module. |
| 2026-03-11 10:06:49.333 | STATUS | BOR received — run #42. User tags: beam=120 GeV/c π⁺, detector=FEI4. |
| 2026-03-11 10:06:49.363 | STATUS | BOR received — run #42. Created /data/runs/run42.h5 (SWMR). Live read enabled. |
| 2026-03-11 10:06:49.383 | STATUS | BOR received — run #42. Snapshot: /data/runs/run42_bor.msgpack. |
| 2026-03-11 10:06:49.383 | INFO | Broadcast "start" → 6 active + 2 data receivers. |
| 2026-03-11 10:06:49.393 | STATUS | BOR received — run #42. eDAQ tags: ThorlabDataEvent, blocks=true. Output: run42.raw. |
| 2026-03-11 10:06:49.433 | STATUS | BOR received — run #42. TX at 1000 Hz. |
| 2026-03-11 10:06:49.483 | STATUS | BOR received — run #42. Logging environment data. |
| 2026-03-11 10:06:49.533 | STATUS | ← start received. Entering RUN. |
| 2026-03-11 10:06:49.583 | STATUS | ← start received. Recording. |
| 2026-03-11 10:06:49.613 | STATUS | ← start received. Recording EUDAQ blocks. |
| 2026-03-11 10:06:49.633 | STATUS | ← start received. Recording active. |
| 2026-03-11 10:06:49.733 | STATUS | ← start received. Event loop active. |
| 2026-03-11 10:06:49.833 | STATUS | ← start received. |
| 2026-03-11 10:06:50.833 | INFO | STAT/CPU_LOAD = 34.7 STAT/CURRENT_UA = 482.3 |
| 2026-03-11 10:06:54.333 | INFO | STAT/CPU_LOAD = 42.1 STAT/WRITE_RATE_MB = 8.4 |
| 2026-03-11 10:06:54.333 | INFO | get_drop_count → 0 frames dropped. Reply: SUCCESS |
| 2026-03-11 10:07:24.333 | INFO | STAT/TEMPERATURE = 23.4°C STAT/HUMIDITY = 41% |
| 2026-03-11 10:07:24.333 | WARNING | Lives: 2/3 — 1 missed heartbeat |
| 2026-03-11 10:07:26.333 | WARNING | Missed heartbeat — lives: 2/3. Last HB 1847ms ago (threshold: 1500ms). |
| 2026-03-11 10:07:28.333 | INFO | STAT/WRITE_RATE_MB = 8.2 STAT/BLOCK_COUNT = 14089 |
| 2026-03-11 10:07:29.133 | DEBUG | Heartbeat OK — state: ORBIT, lives: 2/3 |
| 2026-03-11 10:07:29.333 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3, interval: 1000ms |
| 2026-03-11 10:07:29.333 | DEBUG | Forwarded 847 STAT/ datapoints to InfluxDB in last 60s. |
| 2026-03-11 10:07:30.333 | INFO | STAT/WRITE_RATE_MB = 12.5 STAT/FILE_SIZE_MB = 87.3 |
| 2026-03-11 10:07:30.333 | WARNING | Temperature rising: 38.2°C (threshold: 40°C). Monitoring. |
| 2026-03-11 10:07:31.333 | INFO | 14,392 events buffered. Write queue: 0. File size: 3.7 MB. |
| 2026-03-11 10:07:32.333 | INFO | STAT/CPU_LOAD = 58.2 STAT/TX_RATE_HZ = 998.4 |
| 2026-03-11 10:07:33.133 | TRACE | Heartbeat: ORBIT, lives: 2/3, interval: 1000ms |
| 2026-03-11 10:07:33.333 | INFO | STAT/CPU_LOAD = 36.1 STAT/CURRENT_UA = 481.8 |
| 2026-03-11 10:07:33.433 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3 |
| 2026-03-11 10:07:33.533 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3 |
| 2026-03-11 10:07:33.633 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3, interval: 2000ms |
| 2026-03-11 10:07:33.733 | DEBUG | Heartbeat recovered — state: RUN, lives: 2/3 |
| 2026-03-11 10:07:33.833 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3 |
Subscription: INFO+ 106 / 106 messages