go-log-inspector

A powerful log analysis engine written in Go that ingests large log files in various formats and provides comprehensive analysis capabilities.

Features

• Multiple Log Format Support: Plain text, JSON, Apache/Nginx access logs
• Streaming Parser: Efficient buffered IO for handling large files
• Filtering: Filter logs by level, time range, pattern, and source
• Grouping: Group logs by level, source, or time buckets
• Anomaly Detection: Automatically detect error spikes and unusual patterns
• Timeline Analysis: Generate timeline summaries and statistics
• Multiple Output Formats:
  • Colored terminal tables
  • JSON output
  • HTML reports

Installation

From Source

git clone https://github.com/BaseMax/go-log-inspector.git
cd go-log-inspector
go build -o log-inspector .

Using Go Install

go install github.com/BaseMax/go-log-inspector@latest

Usage

Analyze Command

Analyze log files for patterns, anomalies, and statistics:

# Basic analysis
log-inspector analyze /path/to/logfile.log

# With specific output format
log-inspector analyze /path/to/logfile.log --format html --output report.html
log-inspector analyze /path/to/logfile.log --format json --output report.json

# With filters
log-inspector analyze /path/to/logfile.log --level ERROR
log-inspector analyze /path/to/logfile.log --pattern "database"
log-inspector analyze /path/to/logfile.log --source "api-server"

Filter Command

Filter log entries based on specific criteria:

# Filter by log level
log-inspector filter /path/to/logfile.log --level ERROR

# Filter by pattern
log-inspector filter /path/to/logfile.log --pattern "timeout"

# Filter by time range
log-inspector filter /path/to/logfile.log --start 2024-01-15T10:00:00Z --end 2024-01-15T11:00:00Z

# Combine filters and limit results
log-inspector filter /path/to/logfile.log --level WARN --pattern "memory" --limit 10

# Output as JSON
log-inspector filter /path/to/logfile.log --level ERROR --format json

Stats Command

Get a quick statistical summary:

log-inspector stats /path/to/logfile.log

Supported Log Formats

Plain Text Logs

2024-01-15 10:00:00 INFO Application started
2024-01-15 10:00:05 ERROR Connection failed

JSON Logs

{"timestamp":"2024-01-15T10:00:00Z","level":"info","message":"Application started"}
{"timestamp":"2024-01-15T10:00:05Z","level":"error","message":"Connection failed"}

Apache/Nginx Access Logs

192.168.1.10 - - [15/Jan/2024:10:00:00 -0700] "GET /index.html HTTP/1.1" 200 1234

Examples

Example log files are provided in the examples/ directory:

• sample-plaintext.log - Plain text format logs
• sample-json.log - JSON structured logs
• sample-apache.log - Apache/Nginx access logs

Try them out:

./log-inspector analyze examples/sample-plaintext.log
./log-inspector analyze examples/sample-json.log --format html --output report.html
./log-inspector filter examples/sample-apache.log --level ERROR

Anomaly Detection

The tool automatically detects:

• Error Spikes: Sudden increases in error rates using statistical analysis
• High Error Rates: When error percentage exceeds threshold (10%)
• Consecutive Errors: Multiple consecutive error entries

Command Options

Analyze Command Options

  -f, --format string    Output format: table, json, html (default "table")
      --no-color         Disable colored output
  -l, --level string     Filter by log level (ERROR, WARN, INFO, DEBUG)
  -p, --pattern string   Filter by regex pattern
  -s, --source string    Filter by source
      --start string     Start time (RFC3339 format)
      --end string       End time (RFC3339 format)
  -o, --output string    Output file (default: stdout)

Filter Command Options

  -f, --format string    Output format: table, json (default "table")
      --no-color         Disable colored output
  -l, --level string     Filter by log level
  -p, --pattern string   Filter by regex pattern
  -s, --source string    Filter by source
      --start string     Start time (RFC3339 format)
      --end string       End time (RFC3339 format)
      --limit int        Limit number of results (0 = no limit)
  -o, --output string    Output file (default: stdout)

Dependencies

• spf13/cobra - CLI framework
• gonum - Statistical calculations

License

MIT License - see LICENSE file for details

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.