🔍 Lightweight Observability Platform for Modern Applications
Collect, store, and explore your application's telemetry data with ease
WatchData is a high-performance, open-source observability platform designed for developers who need powerful monitoring without the complexity. Built on OpenTelemetry standards and powered by ClickHouse, it provides real-time insights into your applications' logs, metrics, and traces.
- Solo Developers building side projects
- Small Teams needing cost-effective monitoring
- Learning observability concepts hands-on
- Prototyping before scaling to enterprise solutions
|
|
|
|
graph TB
A[Applications] -->|OTLP gRPC| B[OpenTelemetry Collector]
B -->|Custom Exporter| C[ClickHouse Database]
C -->|Query API| D[API Server]
D -->|REST/WebSocket| E[Frontend Dashboard]
D -->|Real-time| F[WebSocket Clients]
style A fill:#e1f5fe
style B fill:#f3e5f5
style C fill:#fff3e0
style D fill:#e8f5e8
style E fill:#fce4ec
Learn more: 📖 Architecture Documentation
- Docker & Docker Compose (recommended)
- Go 1.24+ (for development)
- Node.js 18+ (for frontend development)
# Clone the repository
git clone https://github.com/Ricky004/watchdata.git
cd watchdata
# Start all services
make up
# Check logs
make logs
# Your services are now running:
# - ClickHouse: http://localhost:8123
# - Collector: grpc://localhost:4317
# - API Server: http://localhost:8080
# - Frontend: http://localhost:3000# Setup development environment
make dev-setup
# Build all components
make build
# Run server (in one terminal)
make run-server
# Send test data (in another terminal)
make run-client🟢 Node.js Example
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-grpc');
const sdk = new NodeSDK({
logRecordProcessor: new BatchLogRecordProcessor(
new OTLPLogExporter({
url: 'http://localhost:4317',
})
),
});
sdk.start();
// Your application logs will now be sent to WatchData
console.log('Hello from my application!');🔵 Python Example
from opentelemetry import logs
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.sdk.logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk.logs.export import BatchLogRecordProcessor
# Configure OpenTelemetry
logs.set_logger_provider(LoggerProvider())
exporter = OTLPLogExporter(endpoint="http://localhost:4317", insecure=True)
logs.get_logger_provider().add_log_record_processor(
BatchLogRecordProcessor(exporter)
)
# Your application logs will now be sent to WatchData
import logging
logging.info("Hello from my Python application!")🟡 Go Example
package main
import (
"context"
"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
"go.opentelemetry.io/otel/log/global"
"go.opentelemetry.io/otel/sdk/log"
)
func main() {
exporter, _ := otlploggrpc.New(context.Background(),
otlploggrpc.WithEndpoint("http://localhost:4317"),
otlploggrpc.WithInsecure(),
)
processor := log.NewBatchProcessor(exporter)
provider := log.NewLoggerProvider(log.WithProcessor(processor))
global.SetLoggerProvider(provider)
// Your application logs will now be sent to WatchData
logger := global.GetLoggerProvider().Logger("my-app")
logger.Emit(context.Background(), log.Record{
Body: log.StringValue("Hello from my Go application!"),
})
}# Send test logs
make run-client
# Or run directly
go run cmd/client/main.go| Method | Endpoint | Description |
|---|---|---|
GET |
/v1/logs |
Retrieve recent logs |
GET |
/v1/logs/since?timestamp=<unix> |
Get logs since timestamp |
GET |
/v1/logs/timerange?start=<unix>&end=<unix> |
Query logs in time range |
// Real-time log streaming
const ws = new WebSocket('ws://localhost:8080/ws');
ws.onmessage = (event) => {
const logData = JSON.parse(event.data);
console.log('New log:', logData);
};# ClickHouse Configuration
CLICKHOUSE_HOST=localhost
CLICKHOUSE_PORT=9000
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=pass
# Server Configuration
SERVER_PORT=8080
LOG_LEVEL=infoEdit configs/otel-collector-config.yaml:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
exporters:
watchdataexporter:
dsn: "tcp://clickhouse:9000/default?username=default&password=pass"
service:
pipelines:
logs:
receivers: [otlp]
exporters: [watchdataexporter]WatchData is built for performance:
- 🚀 10,000+ logs/second ingestion rate
- ⚡ Sub-second query response times
- 💾 90% compression ratio with ClickHouse
- 🔄 Real-time WebSocket updates
- 📈 Horizontal scaling support
- OpenTelemetry log ingestion
- ClickHouse storage backend
- REST API with WebSocket support
- Docker Compose deployment
- Frontend dashboard (Next.js)
- Authentication & authorization
- Metrics support (Prometheus-compatible)
- Distributed tracing visualization
- Alerting system with webhooks
- Multi-tenancy support
- Kubernetes operator
- Advanced analytics and ML insights
We love contributions! Here's how you can help:
Open an issue with details
Start a discussion to share your thoughts
- Fork the repository
- Create a feature branch:
git checkout -b feature/amazing-feature - Commit your changes:
git commit -m 'Add amazing feature' - Push to the branch:
git push origin feature/amazing-feature - Open a Pull Request
Read our Contributing Guide for detailed instructions.
- 🏗️ Architecture Overview
- ⚙️ Configuration Guide
- 🚀 Deployment Guide
- 🔌 API Documentation
- 🧪 Testing Guide
- 💬 Questions? Use GitHub Discussions
- 🐛 Bug Reports: GitHub Issues
- 📧 Email: tridipdam001@gmail.com
This project is licensed under the MIT License - see the LICENSE file for details.
⭐ Star this repo if you find it useful!
Made with ❤️ by Tridip Dam and contributors