[BUG] evnode_sequencer_da_inclusion_height metric is not progressing despite active DA submissions

[auricom]

Version Information

ev-node v1.0.0-beta.10

System Information

Docker x64

Network Configuration

No response

Development Environment

No response

Configuration Details

No response

Steps to Reproduce

• Monitor evnode_sequencer_da_inclusion_height over time
• Observe pending blobs and DA submission logs
• Confirm DA endpoint is responsive and returning success
• Note that height does not increase despite successful submissions

Expected Result

The da_inclusion_height should increment as blocks are successfully included in the data availability layer.

Actual Result

The metric evnode_sequencer_da_inclusion_height is currently stuck at 1.3752729e+07 for chain edennet-2. However, there are no observed failures on the DA endpoint (port 7331/da), and the pending blobs metric indicates that blob submissions are not stalled.

This suggests a possible discrepancy between actual DA submission success and the height tracking logic in the sequencer.

Relevant Logs

Additional Information

No response

[julienrbrt]

So you can confirm you see the header and data DA height when you query /evnode.v1.StoreService/GetBlock?
So it is purely a metric issue?

[auricom]

{
  "block": {
    "header": {
      "header": {
        "version": {},
        "height": "16340569",
        "time": "1764602101595933252",
        "lastHeaderHash": "DdCelG2fDjES0S67Hk+pMWtyQfKqHhQWOOvvLFduPHY=",
        "dataHash": "bjQLnP+zepicpUTmu3gKLHiQHT+zNzh2hRGjBhevoB0=",
        "appHash": "Khv637YMocBIcrF4J3/OgLvx8UiJPlE2gqX/Uj2Wh+w=",
        "proposerAddress": "N89c6gm4fP8oW7OxEtHft7b7PFu8BRzrqkPRzVH04dA=",
        "chainId": "edennet-2"
      },
      "signature": "r3uL6Ek9Yh2Gi32JENx74CGU7vxCSmKjEfC/cYaTdsBODy9Ivq5wPMK6OIT2xwXCh5cAj787FEc7ZZkIsycVCQ==",
      "signer": {
        "address": "N89c6gm4fP8oW7OxEtHft7b7PFu8BRzrqkPRzVH04dA=",
        "pubKey": "CAESIENmQztDCdTwd/DMH0NwpSVzbfmh3JogW40tsdYwto1R"
      }
    },
    "data": {
      "metadata": {
        "chainId": "edennet-2",
        "height": "16340569",
        "time": "1764602101595933252",
        "lastDataHash": "eyE7o6tRiHZ7HUeHLBn/6bzjKUP+S2ZnOxePS41HbTc="
      }
    }
  }
}

The Header/DA height does not seem to be present for a blob produced 7mn ago.

At the same time, the metrics endpoint gives evnode_sequencer_da_submitter_pending_blobs{chain_id="edennet-2"} 195 and as the chain is configured with block_time=100ms it means this blob has already been submitted to the DA.

[julienrbrt]

Thanks! Then the issue probably comes from the processDAInclusionLoop directly, and not only metrics related :/
And I suppose we can assume no cache has been cleared on that node right?

[julienrbrt]

@chatton fixed it in evstack/ev-metrics#22.
Do we have a way to clear that queue with those missing height?

EDIT: Thanks @auricom for pointing me to the right metrics. I thought this one was in ev-node.
After a quick look, it looks like this metrics is just never recorded.

ev-node/sequencers/single/sequencer.go

Lines 131 to 141 in 474818d

	// RecordMetrics updates the metrics with the given values.
	// This method is intended to be called by the block manager after submitting data to the DA layer.
	func (c *Sequencer) RecordMetrics(gasPrice float64, blobSize uint64, statusCode coreda.StatusCode, numPendingBlocks uint64, includedBlockHeight uint64) {
	if c.metrics != nil {
	c.metrics.GasPrice.Set(gasPrice)
	c.metrics.LastBlobSize.Set(float64(blobSize))
	c.metrics.TransactionStatus.With("status", fmt.Sprintf("%d", statusCode)).Add(1)
	c.metrics.NumPendingBlocks.Set(float64(numPendingBlocks))
	c.metrics.IncludedBlockHeight.Set(float64(includedBlockHeight))
	}
	}

is never called by anything.

EDIT 2: turns out this is about this DAInclusionHeight and not IncludedBlockHeight (from sequencer package)

[tac0turtle]

was this closed?

[julienrbrt]

Yes