Live Logs — Observatory
levels:
| Time | Level | Message |
|---|---|---|
| 2026-09-09 16:48:34.102 | STATUS | BOR received — run #41. Created /data/runs/run41.h5 (SWMR). Analyst can read live. |
| 2026-09-09 16:48:34.122 | STATUS | BOR received — run #41. Snapshot: /data/runs/run41_bor.msgpack. |
| 2026-09-09 16:48:34.172 | STATUS | BOR received — run #41. TX at 800 Hz. |
| 2026-09-09 16:48:34.222 | STATUS | BOR received — run #41. Config snapshot written. |
| 2026-09-09 16:48:34.222 | STATUS | BOR received — run #41. Environment logging started. |
| 2026-09-09 16:48:34.322 | STATUS | ← start received. Recording. |
| 2026-09-09 16:48:34.422 | STATUS | ← start received. Recording active. |
| 2026-09-09 16:48:34.522 | STATUS | ← start received. Entering RUN. |
| 2026-09-09 16:48:34.522 | STATUS | ← start received. Event loop active. |
| 2026-09-09 16:48:34.622 | STATUS | ← start received. |
| 2026-09-09 16:48:38.022 | INFO | STAT/CPU_LOAD = 31.2 STAT/CURRENT_UA = 487.6 |
| 2026-09-09 16:48:40.022 | INFO | STAT/CPU_LOAD = 39.5 STAT/WRITE_RATE_MB = 7.9 |
| 2026-09-09 16:48:42.022 | INFO | STAT/WRITE_RATE_MB = 11.4 STAT/FILE_SIZE_MB = 42.8 |
| 2026-09-09 16:49:04.022 | WARNING | Temperature spike: 41.3°C exceeded threshold 40°C. Alert on CMDP UI/ topic. |
| 2026-09-09 16:49:09.022 | CRITICAL | Thermal alert unresolved. Entering SAFE state. HighVoltage.hv1 will autonomously ramp down via CHP observation. |
| 2026-09-09 16:55:13.022 | INFO | STAT/CPU_LOAD = 33.8 STAT/CURRENT_UA = 484.1 |
| 2026-09-09 16:55:13.522 | WARNING | Run condition degraded — TempSensor.ts1 reported failure mid-run. |
| 2026-09-09 16:55:14.022 | STATUS | EOR received — run #41 complete. 12,841 events. Condition: DEGRADED. |
| 2026-09-09 16:55:14.022 | STATUS | EOR received — run #41. 12,841 events. File finalised: run41.h5 (238 MB). |
| 2026-09-09 16:55:14.022 | STATUS | EOR received — run #41. 12,841 events. Condition: DEGRADED. Duration: 400s. |
| 2026-09-09 16:55:14.022 | STATUS | EOR received — run #41. Duration: 400s. TX avg: 798.2 Hz. |
| 2026-09-09 16:55:14.022 | STATUS | EOR received — run #41. Condition: DEGRADED (thermal fault). Duration: 400s. |
| 2026-09-09 16:55:14.222 | INFO | ⚡ Extrasystole emitted — RUN → ORBIT after EOR |
| 2026-09-09 16:55:14.222 | INFO | ⚡ Extrasystole emitted — RUN → ORBIT after EOR |
| 2026-09-09 16:55:14.322 | INFO | ⚡ Extrasystole emitted — RUN → ORBIT after EOR |
| 2026-09-09 16:55:14.522 | INFO | ⚡ Extrasystole emitted — SAFE → ORBIT (recovered) |
| 2026-09-09 17:46:32.022 | INFO | CHIRP offer beacon sent — TempSensor.ts1 (Python/Mariner) on 192.168.1.20:5558 |
| 2026-09-09 17:46:32.522 | INFO | CHIRP offer beacon sent — DevNullReceiver.dnr1 (C++/Sputnik) on 192.168.1.11:5559 |
| 2026-09-09 17:46:33.022 | INFO | CHIRP offer beacon sent — RandomTransmitter.tx1 (C++/Sputnik) on 192.168.1.11:5557 |
| 2026-09-09 17:46:33.222 | STATUS | ← initialize received. Lakeshore 218, 8 channels, RS-232 9600 baud. Reply: SUCCESS |
| 2026-09-09 17:46:33.522 | INFO | CHIRP offer beacon sent — FlightRecorder.fr1 (C++/Sputnik) on 192.168.1.10:5556 |
| 2026-09-09 17:46:33.622 | INFO | CHIRP offer beacon sent — EudaqNativeWriter.eudw1 (C++/Sputnik) on 192.168.1.10:5562 |
| 2026-09-09 17:46:33.822 | INFO | CHIRP offer beacon sent — H5DataWriter.h5w1 (C++/Sputnik) on 192.168.1.10:5561 |
| 2026-09-09 17:46:33.822 | STATUS | ← initialize received. Buffer: 64MB. Role: NONE. Reply: SUCCESS |
| 2026-09-09 17:46:34.022 | INFO | CHIRP offer beacon sent — HighVoltage.hv1 (C++/Sputnik) on 192.168.1.10:5555 |
| 2026-09-09 17:46:34.122 | INFO | CHIRP offer — Observatory subscribing to all CMDP topics (source-side filter: DEBUG+). |
| 2026-09-09 17:46:34.172 | INFO | CHIRP offer — InfluxForwarder subscribing to STAT topics → InfluxDB (source-side filter: INFO+). |
| 2026-09-09 17:46:34.222 | INFO | CHP heartbeat publisher started — interval: 2000ms (RPi reduced tick), maxLives: 3 |
| 2026-09-09 17:46:34.222 | INFO | Broadcast "initialize" → 7 satellites. All replied SUCCESS. |
| 2026-09-09 17:46:34.522 | STATUS | ← initialize received. event_rate_hz=1000, payload=256B. Reply: SUCCESS |
| 2026-09-09 17:46:34.722 | STATUS | ← initialize received. EUDAQ2 native. write_as_blocks=true, newtic_event_tag=ThorlabDataEvent. Reply: SUCCESS |
| 2026-09-09 17:46:34.722 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-09-09 17:46:34.822 | STATUS | ← initialize received. output_dir=/data/runs, max_file_size_mb=1024. Reply: SUCCESS |
| 2026-09-09 17:46:34.922 | STATUS | ← initialize received. HDF5 v1.14, SWMR enabled, POSIX file locks active. Reply: SUCCESS |
| 2026-09-09 17:46:35.022 | STATUS | ← initialize received. Reply: SUCCESS. Entering INIT. |
| 2026-09-09 17:46:35.222 | STATUS | ← launch received. Sensor sampling active on 8 channels. |
| 2026-09-09 17:46:35.522 | INFO | Keithley SMU config: bias_voltage=100V, max_current=500μA, ramp_rate=10V/s |
| 2026-09-09 17:46:35.622 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-09-09 17:46:35.722 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-09-09 17:46:35.722 | STATUS | ← launch received. Pausing — require_after: RandomTransmitter.tx1 must reach ORBIT. |
| 2026-09-09 17:46:35.822 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-09-09 17:46:35.922 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-09-09 17:46:36.022 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-09-09 17:46:36.722 | STATUS | ← launch received. Ready to write EUDAQ blocks. |
| 2026-09-09 17:46:36.722 | INFO | All ACK. Broadcasting "launch"… |
| 2026-09-09 17:46:36.822 | STATUS | ← launch received. Output: /data/runs/. Chunk: 1024. Compression: gzip. |
| 2026-09-09 17:46:36.822 | STATUS | ← launch received. PRNG seeded. Zero-copy allocator engaged. |
| 2026-09-09 17:46:36.922 | STATUS | ← launch received. File handle opened. Ready to record. |
| 2026-09-09 17:46:37.022 | STATUS | ← launch received. Ramping HV to 100V… |
| 2026-09-09 17:46:37.072 | INFO | DevNullReceiver.dnr1 pausing — require_after: RandomTransmitter.tx1 must reach ORBIT. |
| 2026-09-09 17:46:37.122 | INFO | Dependency resolved: RandomTransmitter.tx1 reached ORBIT. Resuming launch. |
| 2026-09-09 17:46:37.222 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-09-09 17:46:37.522 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-09-09 17:46:38.022 | INFO | HV ramp: 0V → 50V → 100V. Stabilised at 100.02V. |
| 2026-09-09 17:46:38.222 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-09-09 17:46:38.322 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-09-09 17:46:38.322 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-09-09 17:46:38.322 | INFO | Data grouping active — payloads <1KB auto-buffered. Saturating 10G link with minimal CPU overhead. |
| 2026-09-09 17:46:38.422 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-09-09 17:46:38.522 | INFO | ⚡ Extrasystole emitted — state change INIT → ORBIT detected |
| 2026-09-09 17:46:38.822 | STATUS | All 7 satellites ORBIT. Dependency graph resolved. Ready for run. |
| 2026-09-09 17:47:48.922 | STATUS | BOR broadcast — run #42. User tags: beam=120 GeV/c π⁺, detector=FEI4 quad module. |
| 2026-09-09 17:47:49.022 | STATUS | BOR received — run #42. User tags: beam=120 GeV/c π⁺, detector=FEI4. |
| 2026-09-09 17:47:49.052 | STATUS | BOR received — run #42. Created /data/runs/run42.h5 (SWMR). Live read enabled. |
| 2026-09-09 17:47:49.072 | STATUS | BOR received — run #42. Snapshot: /data/runs/run42_bor.msgpack. |
| 2026-09-09 17:47:49.072 | INFO | Broadcast "start" → 6 active + 2 data receivers. |
| 2026-09-09 17:47:49.082 | STATUS | BOR received — run #42. eDAQ tags: ThorlabDataEvent, blocks=true. Output: run42.raw. |
| 2026-09-09 17:47:49.122 | STATUS | BOR received — run #42. TX at 1000 Hz. |
| 2026-09-09 17:47:49.172 | STATUS | BOR received — run #42. Logging environment data. |
| 2026-09-09 17:47:49.222 | STATUS | ← start received. Entering RUN. |
| 2026-09-09 17:47:49.272 | STATUS | ← start received. Recording. |
| 2026-09-09 17:47:49.302 | STATUS | ← start received. Recording EUDAQ blocks. |
| 2026-09-09 17:47:49.322 | STATUS | ← start received. Recording active. |
| 2026-09-09 17:47:49.422 | STATUS | ← start received. Event loop active. |
| 2026-09-09 17:47:49.522 | STATUS | ← start received. |
| 2026-09-09 17:47:50.522 | INFO | STAT/CPU_LOAD = 34.7 STAT/CURRENT_UA = 482.3 |
| 2026-09-09 17:47:54.022 | INFO | STAT/CPU_LOAD = 42.1 STAT/WRITE_RATE_MB = 8.4 |
| 2026-09-09 17:47:54.022 | INFO | get_drop_count → 0 frames dropped. Reply: SUCCESS |
| 2026-09-09 17:48:24.022 | INFO | STAT/TEMPERATURE = 23.4°C STAT/HUMIDITY = 41% |
| 2026-09-09 17:48:24.022 | WARNING | Lives: 2/3 — 1 missed heartbeat |
| 2026-09-09 17:48:26.022 | WARNING | Missed heartbeat — lives: 2/3. Last HB 1847ms ago (threshold: 1500ms). |
| 2026-09-09 17:48:28.022 | INFO | STAT/WRITE_RATE_MB = 8.2 STAT/BLOCK_COUNT = 14089 |
| 2026-09-09 17:48:28.822 | DEBUG | Heartbeat OK — state: ORBIT, lives: 2/3 |
| 2026-09-09 17:48:29.022 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3, interval: 1000ms |
| 2026-09-09 17:48:29.022 | DEBUG | Forwarded 847 STAT/ datapoints to InfluxDB in last 60s. |
| 2026-09-09 17:48:30.022 | INFO | STAT/WRITE_RATE_MB = 12.5 STAT/FILE_SIZE_MB = 87.3 |
| 2026-09-09 17:48:30.022 | WARNING | Temperature rising: 38.2°C (threshold: 40°C). Monitoring. |
| 2026-09-09 17:48:31.022 | INFO | 14,392 events buffered. Write queue: 0. File size: 3.7 MB. |
| 2026-09-09 17:48:32.022 | INFO | STAT/CPU_LOAD = 58.2 STAT/TX_RATE_HZ = 998.4 |
| 2026-09-09 17:48:32.822 | TRACE | Heartbeat: ORBIT, lives: 2/3, interval: 1000ms |
| 2026-09-09 17:48:33.022 | INFO | STAT/CPU_LOAD = 36.1 STAT/CURRENT_UA = 481.8 |
| 2026-09-09 17:48:33.122 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3 |
| 2026-09-09 17:48:33.222 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3 |
| 2026-09-09 17:48:33.322 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3, interval: 2000ms |
| 2026-09-09 17:48:33.422 | DEBUG | Heartbeat recovered — state: RUN, lives: 2/3 |
| 2026-09-09 17:48:33.522 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3 |
Subscription: INFO+ 106 / 106 messages