Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 43 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Data Availability metrics and monitoring tool for evstack using Celestia DA. Thi
- Retry logic with exponential backoff for pending DA submissions
- Prometheus metrics for tracking unverified block ranges
- Support for both streaming and one-shot block verification modes
- Account balance monitoring via Celestia consensus RPC with automatic failover

## Quick Start

Expand All @@ -32,8 +33,8 @@ The monitor command streams EVM block headers and verifies DA submission on Cele

```bash
./ev-metrics monitor \
--header-namespace collect_testnet_header \
--data-namespace collect_testnet_data
--header-namespace testnet_header \
--data-namespace testnet_data
```


Expand All @@ -54,8 +55,8 @@ Metrics will be available at `http://localhost:2112/metrics`
### Command-Line Flags

**Required:**
- `--header-namespace`: Header namespace (e.g. collect_testnet_header )
- `--data-namespace`: Data namespace (e.g. collect_testnet_data )
- `--header-namespace`: Header namespace (e.g. testnet_header )
- `--data-namespace`: Data namespace (e.g. testnet_data )

**Optional:**
- `--evnode-addr`: ev-node Connect RPC address (default: `http://localhost:7331`)
Expand All @@ -71,6 +72,9 @@ Metrics will be available at `http://localhost:2112/metrics`
- `--reference-node`: Reference node RPC endpoint URL (sequencer) for drift monitoring
- `--full-nodes`: Comma-separated list of full node RPC endpoint URLs for drift monitoring
- `--polling-interval`: Polling interval in seconds for checking node block heights (default: 10)
- `--balance.addresses`: Comma-separated Celestia addresses to monitor (enables balance checking)
- `--balance.consensus-rpc-urls`: Comma-separated consensus RPC URLs for balance queries (required if balance.addresses is set)
- `--balance.scrape-interval`: Balance check scrape interval in seconds (default: 30)
- `--verbose`: Enable verbose logging (default: false)

### Example with Custom Endpoints
Expand Down Expand Up @@ -100,6 +104,22 @@ Enable JSON-RPC request duration monitoring by providing the `--evm-rpc-url` fla

This will periodically send `eth_blockNumber` JSON-RPC requests to monitor node health and response times.

### Example with Balance Monitoring

Monitor native token balances for one or more Celestia addresses:

```bash
./ev-metrics monitor \
--header-namespace collect_testnet_header \
--data-namespace collect_testnet_data \
Comment on lines +113 to +114
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The namespaces in this example (collect_testnet_header, collect_testnet_data) are inconsistent with the changes made elsewhere in this README, where they were updated to testnet_header and testnet_data. For consistency, this example should also use the updated namespace names.

Suggested change
--header-namespace collect_testnet_header \
--data-namespace collect_testnet_data \
--header-namespace testnet_header \
--data-namespace testnet_data \

--balance.addresses "celestia1abc...,celestia1def..." \
--balance.consensus-rpc-urls "https://rpc.celestia.org,https://rpc-mocha.pops.one" \
--balance.scrape-interval 30 \
--enable-metrics
```

This will query account balances every 30 seconds with automatic failover between RPC endpoints.

## Prometheus Metrics

When metrics are enabled, the following metrics are exposed:
Expand Down Expand Up @@ -163,3 +183,22 @@ When `--reference-node` and `--full-nodes` are provided:
- **Type**: Gauge
- **Labels**: `chain_id`, `target_endpoint`
- **Description**: Block height difference between reference and target endpoints (positive = target behind, negative = target ahead)

### Balance Monitoring Metrics

When `--balance.addresses` and `--balance.consensus-rpc-urls` are provided:

### `ev_metrics_account_balance`
- **Type**: Gauge
- **Labels**: `chain_id`, `address`, `denom`
- **Description**: Native token balance for Celestia addresses

### `ev_metrics_consensus_rpc_endpoint_availability`
- **Type**: Gauge
- **Labels**: `chain_id`, `endpoint`
- **Description**: Consensus RPC endpoint availability status (1.0 = available, 0.0 = unavailable)

### `ev_metrics_consensus_rpc_endpoint_errors_total`
- **Type**: Counter
- **Labels**: `chain_id`, `endpoint`, `error_type`
- **Description**: Total number of consensus RPC endpoint errors by type
34 changes: 34 additions & 0 deletions cmd/monitor.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import (
"github.com/evstack/ev-metrics/internal/clients/celestia"
"github.com/evstack/ev-metrics/internal/clients/evm"
"github.com/evstack/ev-metrics/internal/clients/evnode"
"github.com/evstack/ev-metrics/pkg/exporters/balance"
"github.com/evstack/ev-metrics/pkg/exporters/drift"
"github.com/evstack/ev-metrics/pkg/exporters/jsonrpc"
"github.com/evstack/ev-metrics/pkg/exporters/verifier"
Expand All @@ -38,6 +39,9 @@ const (
flagFullNodes = "full-nodes"
flagPollingInterval = "polling-interval"
flagJsonRpcScrapeInterval = "jsonrpc-scrape-interval"
flagBalanceAddresses = "balance.addresses"
flagBalanceRpcUrls = "balance.consensus-rpc-urls"
flagBalanceScrapeInterval = "balance.scrape-interval"

metricsPath = "/metrics"
)
Expand All @@ -60,6 +64,9 @@ type flagValues struct {
fullNodes string
pollingInterval int
jsonRpcScrapeInterval int
balanceAddresses string
balanceRpcUrls string
balanceScrapeInterval int
}

func NewMonitorCmd() *cobra.Command {
Expand Down Expand Up @@ -89,6 +96,9 @@ func NewMonitorCmd() *cobra.Command {
cmd.Flags().StringVar(&flags.fullNodes, flagFullNodes, "", "Comma-separated list of full node RPC endpoint URLs for drift monitoring")
cmd.Flags().IntVar(&flags.pollingInterval, flagPollingInterval, 10, "Polling interval in seconds for checking node block heights (default: 10)")
cmd.Flags().IntVar(&flags.jsonRpcScrapeInterval, flagJsonRpcScrapeInterval, 10, "JSON-RPC health check scrape interval in seconds (default: 10)")
cmd.Flags().StringVar(&flags.balanceAddresses, flagBalanceAddresses, "", "Comma-separated celestia addresses to monitor (enables balance checking)")
cmd.Flags().StringVar(&flags.balanceRpcUrls, flagBalanceRpcUrls, "", "Comma-separated consensus rpc urls for balance queries (required if balance.addresses is set)")
cmd.Flags().IntVar(&flags.balanceScrapeInterval, flagBalanceScrapeInterval, 30, "Balance check scrape interval in seconds (default: 30)")

if err := cmd.MarkFlagRequired(flagHeaderNS); err != nil {
panic(err)
Expand Down Expand Up @@ -208,6 +218,30 @@ func monitorAndExportMetrics(_ *cobra.Command, _ []string) error {
exporters = append(exporters, jsonrpc.NewMetricsExporter(flags.chainID, cfg.EvmClient, flags.jsonRpcScrapeInterval, logger))
}

// start balance monitoring if balance.addresses is provided
if flags.balanceAddresses != "" {
if flags.balanceRpcUrls == "" {
return fmt.Errorf("--balance.consensus-rpc-urls is required when --balance.addresses is set")
}

addressList := strings.Split(flags.balanceAddresses, ",")
rpcUrls := strings.Split(flags.balanceRpcUrls, ",")
Comment on lines +227 to +228
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

While splitting comma-separated strings is a common pattern, it can be brittle if users accidentally add extra spaces (e.g., "addr1, addr2"). Using strings.FieldsFunc with a custom splitter can make parsing more robust by trimming whitespace automatically.

Suggested change
addressList := strings.Split(flags.balanceAddresses, ",")
rpcUrls := strings.Split(flags.balanceRpcUrls, ",")
addressList := strings.FieldsFunc(flags.balanceAddresses, func(c rune) bool { return c == ',' })
rpcUrls := strings.FieldsFunc(flags.balanceRpcUrls, func(c rune) bool { return c == ',' })


exporters = append(exporters, balance.NewMetricsExporter(
flags.chainID,
addressList,
rpcUrls,
flags.balanceScrapeInterval,
logger,
))

logger.Info().
Strs("addresses", addressList).
Strs("rpc_urls", rpcUrls).
Int("scrape_interval", flags.balanceScrapeInterval).
Msg("balance monitoring enabled")
}

err = metrics.StartServer(ctx, metricsPath, flags.port, logger, exporters...)

// context.Canceled is expected during graceful shutdown, not an error
Expand Down
116 changes: 114 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ go 1.24.6

require (
connectrpc.com/connect v1.19.1
github.com/cosmos/cosmos-sdk v0.50.10
github.com/ethereum/go-ethereum v1.16.5
github.com/evstack/ev-node v1.0.0-beta.8
github.com/evstack/ev-node/core v1.0.0-beta.3
Expand All @@ -13,51 +14,162 @@ require (
github.com/spf13/cobra v1.10.1
github.com/stretchr/testify v1.11.1
golang.org/x/sync v0.17.0
google.golang.org/grpc v1.75.0
google.golang.org/protobuf v1.36.10
)

require (
cosmossdk.io/api v0.7.6 // indirect
cosmossdk.io/collections v0.4.0 // indirect
cosmossdk.io/core v0.11.1 // indirect
cosmossdk.io/depinject v1.1.0 // indirect
cosmossdk.io/errors v1.0.1 // indirect
cosmossdk.io/log v1.4.1 // indirect
cosmossdk.io/math v1.4.0 // indirect
cosmossdk.io/store v1.1.1 // indirect
cosmossdk.io/x/tx v0.13.7 // indirect
filippo.io/edwards25519 v1.0.0 // indirect
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 // indirect
github.com/99designs/keyring v1.2.1 // indirect
github.com/DataDog/datadog-go v3.2.0+incompatible // indirect
github.com/DataDog/zstd v1.5.5 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/StackExchange/wmi v1.2.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/bgentry/speakeasy v0.1.1-0.20220910012023-760eaf8b6816 // indirect
github.com/bits-and-blooms/bitset v1.20.0 // indirect
github.com/btcsuite/btcd/btcec/v2 v2.3.5 // indirect
github.com/cenkalti/backoff/v4 v4.1.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cockroachdb/errors v1.11.3 // indirect
github.com/cockroachdb/fifo v0.0.0-20240606204812-0bbfbd93a7ce // indirect
github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b // indirect
github.com/cockroachdb/pebble v1.1.5 // indirect
github.com/cockroachdb/redact v1.1.5 // indirect
github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 // indirect
github.com/cometbft/cometbft v0.38.12 // indirect
github.com/cometbft/cometbft-db v0.14.1 // indirect
github.com/consensys/gnark-crypto v0.18.1 // indirect
github.com/cosmos/btcutil v1.0.5 // indirect
github.com/cosmos/cosmos-db v1.1.1 // indirect
github.com/cosmos/cosmos-proto v1.0.0-beta.5 // indirect
github.com/cosmos/go-bip39 v1.0.0 // indirect
github.com/cosmos/gogogateway v1.2.0 // indirect
github.com/cosmos/gogoproto v1.7.0 // indirect
github.com/cosmos/iavl v1.2.2 // indirect
github.com/cosmos/ics23/go v0.11.0 // indirect
github.com/cosmos/ledger-cosmos-go v0.15.0 // indirect
github.com/crate-crypto/go-eth-kzg v1.4.0 // indirect
github.com/crate-crypto/go-ipa v0.0.0-20240724233137-53bbb0ceb27a // indirect
github.com/danieljoos/wincred v1.1.2 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/deckarep/golang-set/v2 v2.6.0 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 // indirect
github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f // indirect
github.com/dgraph-io/badger/v4 v4.5.1 // indirect
github.com/dgraph-io/ristretto/v2 v2.1.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/dvsekhvalnov/jose2go v1.6.0 // indirect
github.com/emicklei/dot v1.6.2 // indirect
github.com/ethereum/c-kzg-4844/v2 v2.1.3 // indirect
github.com/ethereum/go-verkle v0.2.2 // indirect
github.com/fatih/color v1.16.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/filecoin-project/go-jsonrpc v0.8.0 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/getsentry/sentry-go v0.27.0 // indirect
github.com/go-kit/kit v0.13.0 // indirect
github.com/go-kit/log v0.2.1 // indirect
github.com/go-logfmt/logfmt v0.6.0 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 // indirect
github.com/gogo/googleapis v1.4.1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang/snappy v1.0.0 // indirect
github.com/google/btree v1.1.3 // indirect
github.com/google/flatbuffers v24.12.23+incompatible // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/handlers v1.5.1 // indirect
github.com/gorilla/mux v1.8.0 // indirect
github.com/gorilla/websocket v1.5.3 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c // indirect
github.com/hashicorp/go-hclog v1.5.0 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-metrics v0.5.3 // indirect
github.com/hashicorp/go-plugin v1.5.2 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/yamux v0.1.1 // indirect
github.com/hdevalence/ed25519consensus v0.1.0 // indirect
github.com/holiman/uint256 v1.3.2 // indirect
github.com/huandu/skiplist v1.2.0 // indirect
github.com/iancoleman/strcase v0.3.0 // indirect
github.com/improbable-eng/grpc-web v0.15.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/ipfs/go-log/v2 v2.8.1 // indirect
github.com/jmhodges/levigo v1.0.0 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/linxGnu/grocksdb v1.8.14 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mitchellh/go-testing-interface v1.14.1 // indirect
github.com/mtibben/percent v0.2.1 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/oasisprotocol/curve25519-voi v0.0.0-20230904125328-1f23a7beb09a // indirect
github.com/oklog/run v1.1.0 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/petermattis/goid v0.0.0-20240813172612-4fcff4a6cae7 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/procfs v0.17.0 // indirect
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/rs/cors v1.11.1 // indirect
github.com/sagikazarmark/locafero v0.11.0 // indirect
github.com/sasha-s/go-deadlock v0.3.5 // indirect
github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible // indirect
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/spf13/viper v1.21.0 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/supranational/blst v0.3.16-0.20250831170142-f48500c1fdbe // indirect
github.com/syndtr/goleveldb v1.0.1-0.20220721030215-126854af5e6d // indirect
github.com/tendermint/go-amino v0.16.0 // indirect
github.com/tidwall/btree v1.7.0 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/zondax/hid v0.9.2 // indirect
github.com/zondax/ledger-go v1.0.0 // indirect
go.etcd.io/bbolt v1.4.0-alpha.0.0.20240404170359-43604f3112c5 // indirect
go.opencensus.io v0.24.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.43.0 // indirect
golang.org/x/exp v0.0.0-20250911091902-df9299821621 // indirect
golang.org/x/net v0.45.0 // indirect
golang.org/x/sys v0.37.0 // indirect
golang.org/x/term v0.36.0 // indirect
golang.org/x/text v0.30.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250707201910-8d1bb00bc6a7 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250728155136-f173205681a0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
gotest.tools/v3 v3.5.1 // indirect
nhooyr.io/websocket v1.8.6 // indirect
pgregory.net/rapid v1.1.0 // indirect
sigs.k8s.io/yaml v1.4.0 // indirect
)

//replace github.com/evstack/ev-node => /Users/chatton/checkouts/evstack/ev-node
Loading