Live Logs — Observatory
levels:
| Time | Level | Message |
|---|---|---|
| 2026-07-07 06:23:04.685 | STATUS | BOR received — run #41. Created /data/runs/run41.h5 (SWMR). Analyst can read live. |
| 2026-07-07 06:23:04.705 | STATUS | BOR received — run #41. Snapshot: /data/runs/run41_bor.msgpack. |
| 2026-07-07 06:23:04.755 | STATUS | BOR received — run #41. TX at 800 Hz. |
| 2026-07-07 06:23:04.805 | STATUS | BOR received — run #41. Config snapshot written. |
| 2026-07-07 06:23:04.805 | STATUS | BOR received — run #41. Environment logging started. |
| 2026-07-07 06:23:04.905 | STATUS | ← start received. Recording. |
| 2026-07-07 06:23:05.005 | STATUS | ← start received. Recording active. |
| 2026-07-07 06:23:05.105 | STATUS | ← start received. Entering RUN. |
| 2026-07-07 06:23:05.105 | STATUS | ← start received. Event loop active. |
| 2026-07-07 06:23:05.205 | STATUS | ← start received. |
| 2026-07-07 06:23:08.605 | INFO | STAT/CPU_LOAD = 31.2 STAT/CURRENT_UA = 487.6 |
| 2026-07-07 06:23:10.605 | INFO | STAT/CPU_LOAD = 39.5 STAT/WRITE_RATE_MB = 7.9 |
| 2026-07-07 06:23:12.605 | INFO | STAT/WRITE_RATE_MB = 11.4 STAT/FILE_SIZE_MB = 42.8 |
| 2026-07-07 06:23:34.605 | WARNING | Temperature spike: 41.3°C exceeded threshold 40°C. Alert on CMDP UI/ topic. |
| 2026-07-07 06:23:39.605 | CRITICAL | Thermal alert unresolved. Entering SAFE state. HighVoltage.hv1 will autonomously ramp down via CHP observation. |
| 2026-07-07 06:29:43.605 | INFO | STAT/CPU_LOAD = 33.8 STAT/CURRENT_UA = 484.1 |
| 2026-07-07 06:29:44.105 | WARNING | Run condition degraded — TempSensor.ts1 reported failure mid-run. |
| 2026-07-07 06:29:44.605 | STATUS | EOR received — run #41 complete. 12,841 events. Condition: DEGRADED. |
| 2026-07-07 06:29:44.605 | STATUS | EOR received — run #41. 12,841 events. File finalised: run41.h5 (238 MB). |
| 2026-07-07 06:29:44.605 | STATUS | EOR received — run #41. 12,841 events. Condition: DEGRADED. Duration: 400s. |
| 2026-07-07 06:29:44.605 | STATUS | EOR received — run #41. Duration: 400s. TX avg: 798.2 Hz. |
| 2026-07-07 06:29:44.605 | STATUS | EOR received — run #41. Condition: DEGRADED (thermal fault). Duration: 400s. |
| 2026-07-07 06:29:44.805 | INFO | ⚡ Extrasystole emitted — RUN → ORBIT after EOR |
| 2026-07-07 06:29:44.805 | INFO | ⚡ Extrasystole emitted — RUN → ORBIT after EOR |
| 2026-07-07 06:29:44.905 | INFO | ⚡ Extrasystole emitted — RUN → ORBIT after EOR |
| 2026-07-07 06:29:45.105 | INFO | ⚡ Extrasystole emitted — SAFE → ORBIT (recovered) |
| 2026-07-07 07:21:02.605 | INFO | CHIRP offer beacon sent — TempSensor.ts1 (Python/Mariner) on 192.168.1.20:5558 |
| 2026-07-07 07:21:03.105 | INFO | CHIRP offer beacon sent — DevNullReceiver.dnr1 (C++/Sputnik) on 192.168.1.11:5559 |
| 2026-07-07 07:21:03.605 | INFO | CHIRP offer beacon sent — RandomTransmitter.tx1 (C++/Sputnik) on 192.168.1.11:5557 |
| 2026-07-07 07:21:03.805 | STATUS | ← initialize received. Lakeshore 218, 8 channels, RS-232 9600 baud. Reply: SUCCESS |
| 2026-07-07 07:21:04.105 | INFO | CHIRP offer beacon sent — FlightRecorder.fr1 (C++/Sputnik) on 192.168.1.10:5556 |
| 2026-07-07 07:21:04.205 | INFO | CHIRP offer beacon sent — EudaqNativeWriter.eudw1 (C++/Sputnik) on 192.168.1.10:5562 |
| 2026-07-07 07:21:04.405 | INFO | CHIRP offer beacon sent — H5DataWriter.h5w1 (C++/Sputnik) on 192.168.1.10:5561 |
| 2026-07-07 07:21:04.405 | STATUS | ← initialize received. Buffer: 64MB. Role: NONE. Reply: SUCCESS |
| 2026-07-07 07:21:04.605 | INFO | CHIRP offer beacon sent — HighVoltage.hv1 (C++/Sputnik) on 192.168.1.10:5555 |
| 2026-07-07 07:21:04.705 | INFO | CHIRP offer — Observatory subscribing to all CMDP topics (source-side filter: DEBUG+). |
| 2026-07-07 07:21:04.755 | INFO | CHIRP offer — InfluxForwarder subscribing to STAT topics → InfluxDB (source-side filter: INFO+). |
| 2026-07-07 07:21:04.805 | INFO | CHP heartbeat publisher started — interval: 2000ms (RPi reduced tick), maxLives: 3 |
| 2026-07-07 07:21:04.805 | INFO | Broadcast "initialize" → 7 satellites. All replied SUCCESS. |
| 2026-07-07 07:21:05.105 | STATUS | ← initialize received. event_rate_hz=1000, payload=256B. Reply: SUCCESS |
| 2026-07-07 07:21:05.305 | STATUS | ← initialize received. EUDAQ2 native. write_as_blocks=true, newtic_event_tag=ThorlabDataEvent. Reply: SUCCESS |
| 2026-07-07 07:21:05.305 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-07-07 07:21:05.405 | STATUS | ← initialize received. output_dir=/data/runs, max_file_size_mb=1024. Reply: SUCCESS |
| 2026-07-07 07:21:05.505 | STATUS | ← initialize received. HDF5 v1.14, SWMR enabled, POSIX file locks active. Reply: SUCCESS |
| 2026-07-07 07:21:05.605 | STATUS | ← initialize received. Reply: SUCCESS. Entering INIT. |
| 2026-07-07 07:21:05.805 | STATUS | ← launch received. Sensor sampling active on 8 channels. |
| 2026-07-07 07:21:06.105 | INFO | Keithley SMU config: bias_voltage=100V, max_current=500μA, ramp_rate=10V/s |
| 2026-07-07 07:21:06.205 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-07-07 07:21:06.305 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-07-07 07:21:06.305 | STATUS | ← launch received. Pausing — require_after: RandomTransmitter.tx1 must reach ORBIT. |
| 2026-07-07 07:21:06.405 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-07-07 07:21:06.505 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-07-07 07:21:06.605 | INFO | CHP heartbeat publisher started — interval: 1000ms, maxLives: 3 |
| 2026-07-07 07:21:07.305 | STATUS | ← launch received. Ready to write EUDAQ blocks. |
| 2026-07-07 07:21:07.305 | INFO | All ACK. Broadcasting "launch"… |
| 2026-07-07 07:21:07.405 | STATUS | ← launch received. Output: /data/runs/. Chunk: 1024. Compression: gzip. |
| 2026-07-07 07:21:07.405 | STATUS | ← launch received. PRNG seeded. Zero-copy allocator engaged. |
| 2026-07-07 07:21:07.505 | STATUS | ← launch received. File handle opened. Ready to record. |
| 2026-07-07 07:21:07.605 | STATUS | ← launch received. Ramping HV to 100V… |
| 2026-07-07 07:21:07.655 | INFO | DevNullReceiver.dnr1 pausing — require_after: RandomTransmitter.tx1 must reach ORBIT. |
| 2026-07-07 07:21:07.705 | INFO | Dependency resolved: RandomTransmitter.tx1 reached ORBIT. Resuming launch. |
| 2026-07-07 07:21:07.805 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-07-07 07:21:08.105 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-07-07 07:21:08.605 | INFO | HV ramp: 0V → 50V → 100V. Stabilised at 100.02V. |
| 2026-07-07 07:21:08.805 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-07-07 07:21:08.905 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-07-07 07:21:08.905 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-07-07 07:21:08.905 | INFO | Data grouping active — payloads <1KB auto-buffered. Saturating 10G link with minimal CPU overhead. |
| 2026-07-07 07:21:09.005 | INFO | ⚡ Extrasystole emitted — INIT → ORBIT |
| 2026-07-07 07:21:09.105 | INFO | ⚡ Extrasystole emitted — state change INIT → ORBIT detected |
| 2026-07-07 07:21:09.405 | STATUS | All 7 satellites ORBIT. Dependency graph resolved. Ready for run. |
| 2026-07-07 07:22:19.505 | STATUS | BOR broadcast — run #42. User tags: beam=120 GeV/c π⁺, detector=FEI4 quad module. |
| 2026-07-07 07:22:19.605 | STATUS | BOR received — run #42. User tags: beam=120 GeV/c π⁺, detector=FEI4. |
| 2026-07-07 07:22:19.635 | STATUS | BOR received — run #42. Created /data/runs/run42.h5 (SWMR). Live read enabled. |
| 2026-07-07 07:22:19.655 | STATUS | BOR received — run #42. Snapshot: /data/runs/run42_bor.msgpack. |
| 2026-07-07 07:22:19.655 | INFO | Broadcast "start" → 6 active + 2 data receivers. |
| 2026-07-07 07:22:19.665 | STATUS | BOR received — run #42. eDAQ tags: ThorlabDataEvent, blocks=true. Output: run42.raw. |
| 2026-07-07 07:22:19.705 | STATUS | BOR received — run #42. TX at 1000 Hz. |
| 2026-07-07 07:22:19.755 | STATUS | BOR received — run #42. Logging environment data. |
| 2026-07-07 07:22:19.805 | STATUS | ← start received. Entering RUN. |
| 2026-07-07 07:22:19.855 | STATUS | ← start received. Recording. |
| 2026-07-07 07:22:19.885 | STATUS | ← start received. Recording EUDAQ blocks. |
| 2026-07-07 07:22:19.905 | STATUS | ← start received. Recording active. |
| 2026-07-07 07:22:20.005 | STATUS | ← start received. Event loop active. |
| 2026-07-07 07:22:20.105 | STATUS | ← start received. |
| 2026-07-07 07:22:21.105 | INFO | STAT/CPU_LOAD = 34.7 STAT/CURRENT_UA = 482.3 |
| 2026-07-07 07:22:24.605 | INFO | STAT/CPU_LOAD = 42.1 STAT/WRITE_RATE_MB = 8.4 |
| 2026-07-07 07:22:24.605 | INFO | get_drop_count → 0 frames dropped. Reply: SUCCESS |
| 2026-07-07 07:22:54.605 | INFO | STAT/TEMPERATURE = 23.4°C STAT/HUMIDITY = 41% |
| 2026-07-07 07:22:54.605 | WARNING | Lives: 2/3 — 1 missed heartbeat |
| 2026-07-07 07:22:56.605 | WARNING | Missed heartbeat — lives: 2/3. Last HB 1847ms ago (threshold: 1500ms). |
| 2026-07-07 07:22:58.605 | INFO | STAT/WRITE_RATE_MB = 8.2 STAT/BLOCK_COUNT = 14089 |
| 2026-07-07 07:22:59.405 | DEBUG | Heartbeat OK — state: ORBIT, lives: 2/3 |
| 2026-07-07 07:22:59.605 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3, interval: 1000ms |
| 2026-07-07 07:22:59.605 | DEBUG | Forwarded 847 STAT/ datapoints to InfluxDB in last 60s. |
| 2026-07-07 07:23:00.605 | INFO | STAT/WRITE_RATE_MB = 12.5 STAT/FILE_SIZE_MB = 87.3 |
| 2026-07-07 07:23:00.605 | WARNING | Temperature rising: 38.2°C (threshold: 40°C). Monitoring. |
| 2026-07-07 07:23:01.605 | INFO | 14,392 events buffered. Write queue: 0. File size: 3.7 MB. |
| 2026-07-07 07:23:02.605 | INFO | STAT/CPU_LOAD = 58.2 STAT/TX_RATE_HZ = 998.4 |
| 2026-07-07 07:23:03.405 | TRACE | Heartbeat: ORBIT, lives: 2/3, interval: 1000ms |
| 2026-07-07 07:23:03.605 | INFO | STAT/CPU_LOAD = 36.1 STAT/CURRENT_UA = 481.8 |
| 2026-07-07 07:23:03.705 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3 |
| 2026-07-07 07:23:03.805 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3 |
| 2026-07-07 07:23:03.905 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3, interval: 2000ms |
| 2026-07-07 07:23:04.005 | DEBUG | Heartbeat recovered — state: RUN, lives: 2/3 |
| 2026-07-07 07:23:04.105 | DEBUG | Heartbeat OK — state: RUN, lives: 3/3 |
Subscription: INFO+ 106 / 106 messages