[RFE] Delete stats corresponding to deletes of resources/entities pertaining to tendrl managed cluster

[anmolbabu]

Delete stats corresponding to deletes of resources/entities pertaining to tendrl managed cluster

Graphite maintains stats as wsp whisper files on disk under /var/lib/carbon/whisper/tendrl/<metric_path...>

Now suppose a resource in the cluster managed by tendrl is deleted out-of-band, collectd plugins and even the monitoring-integration stop pushing stats corresponding to it to the graphite. But, the stats that were pushed to graphite until the resource was deleted, which means grafana still sees that resource to be existing as the whisper files corresponding to the metrics for the resource still exist on the disk. Now this means for the delete of the resource to be visible to grafana, the graphite-whisper files on disk needs to be deleted as well.

Possible Approach:

1. Define a basic simple flow in monitoring-integration that takes as a parameter the details of the resource deleted like the cluster-id, resource-type, resource identifier.
2. The flow identifies the resource based on above mentioned parameters, scans through the list of metrics relevant to this resource and deletes files corresponding to the metrics
3. The sds-integration intiates the flow in one of these situations(based on reliability):
   • During the sync cycle, a previously tendrl managed resource is found to be deleted
   • An event corresponding to delete of a previously tendrl managed resource is received.

A basic scenario when this is clearly visible is:

1. Delete of a volume does not delete the corresponding volume and brick dashboards.
2. Removal of brick from a volume does not remove brick dashboard in grafana.
   Other scenarios if any need to be observed and spec'd out.

[anmolbabu]

Cc: @Tendrl/tendrl-core