feat: add health monitor provider for tracking provider status

[0xbyt4]

Summary

Add centralized health monitoring system with automatic recovery for tracking and healing provider status across the OM1 system.

Features

Health Monitoring

• Thread-safe singleton for monitoring provider health
• Heartbeat tracking with configurable timeout (default: 30s)
• Error reporting with threshold-based degradation (default: 5 errors)
• Background monitoring thread with smart logging (no spam when healthy)

Auto-Recovery

• Automatic recovery attempts when providers become unhealthy
• Configurable max attempts (default: 3) and cooldown (default: 60s)
• Recovery callbacks registered per-provider
• RECOVERING status during recovery attempts
• Recovery state resets on successful heartbeat

Integrated Providers (16 total)

Core Providers

Provider	Type	Recovery
ASRProvider	Speech	stop() + start()
GpsProvider	Sensor	stop() + start()
ElevenLabsTTSProvider	TTS	stop() + start()
VLMOpenAIProvider	Vision	stop() + start()
VLMGeminiProvider	Vision	stop() + start()
VLMVilaProvider	Vision	stop() + start()
RivaTTSProvider	TTS	stop() + start()
UbTtsProvider	TTS	health check ping

Alternative/Extended Providers

Provider	Type	Recovery
ASRRTSPProvider	Speech (RTSP)	stop() + start()
UbtechASRProvider	Speech (Ubtech)	stop() + start()
UbtechVLMProvider	Vision (Ubtech)	stop() + start()
VLMOpenAIRTSPProvider	Vision (RTSP)	stop() + start()
VLMVilaRTSPProvider	Vision (RTSP)	stop() + start()
VLMVilaZenohProvider	Vision (Zenoh)	stop() + start()
RtkProvider	RTK GPS	stop() + start()
D435Provider	RealSense D435	stop() + start()

Runtime Integration

Runtime	Integration
CortexRuntime	start_monitoring on run, stop on cleanup
ModeCortexRuntime	start_monitoring on run, stop on cleanup
InputOrchestrator	Auto-registers all input sensors

How It Works

Provider stops sending heartbeats
    └── Background thread detects timeout (10s check interval)
            └── Logs ERROR: "Unhealthy providers: GpsProvider (no heartbeat for 35.0s)"
                    └── If recovery_callback registered:
                            └── Attempt 1/3: Call _recover()
                                    ├── Success: Reset attempts, provider healthy
                                    └── Failure: Wait cooldown (60s), try again

Example Output

# Detection
ERROR: Unhealthy providers: GpsProvider (no heartbeat for 35.0s)

# Recovery attempt
INFO: Attempting recovery for 'GpsProvider' (attempt 1/3)
INFO: GpsProvider: Attempting recovery...
INFO: GpsProvider: Recovery successful
INFO: Recovery successful for 'GpsProvider'

# If recovery keeps failing
WARNING: Provider 'GpsProvider' exceeded max recovery attempts (3), giving up

Test Coverage

Test Category	Tests
HealthMonitorProvider unit tests	32
Integration tests (runtime simulation)	27
Core provider tests	45
Extended provider tests	65
Total	169+

Configuration

HealthMonitorProvider(
    heartbeat_timeout=30.0,    # Seconds before unhealthy
    error_threshold=5,          # Errors before degraded
    check_interval=10.0,        # Background check interval
    max_recovery_attempts=3,    # Max recovery tries
    recovery_cooldown=60.0,     # Seconds between attempts
    auto_recovery=True,         # Enable/disable recovery
)

Test plan

• All 800+ tests pass locally
• 16 providers integrated with health monitoring
• Auto-recovery tested with success/failure scenarios
• Max attempts and cooldown verified
• Manual verification with simulated providers
• pre-commit hooks pass

[openminddev]

Hi @0xbyt4 the idea is good, but why don't we develop with prometheus?

[0xbyt4]

Hi @openminddev, I will definitely give it a try. Additionally, independent of the main topic, I would like you to review the technical analysis I conducted through reverse engineering on a mobile app grid and telemetry system. More specifically, I would appreciate your feedback. Thank you. I know you are very busy. ( x article https://x.com/eyeofquantum/status/2009412169289384366?s=20 )