SOLR-18025 Attempt to fix graceful shutdown of LeaderTragicEventTest

[janhoy]

Analysis:

Root Cause

LeaderTragicEventTest fails during class-level shutdown when Jetty's Server.doStop() exceeds its internal timeout and throws ExecutionException(TimeoutException). After tragic events corrupt cores, shutdown naturally takes longer and can timeout - this is expected behavior, not a test failure. See develocity logs here.

Fix

Added shutdownTimeoutIsError configuration to MiniSolrCloudCluster:

• Default: true - normal tests fail on unexpected timeouts
• LeaderTragicEventTest: false - accepts timeouts as expected outcome

Implementation:

• Added 60s shutdown timeout to the shutdown process (2x Jetty's internal timeout)
• checkForExceptions() treats ExecutionException(TimeoutException) as warning when shutdownTimeoutIsError=false

https://issues.apache.org/jira/browse/SOLR-18025

[dsmiley]

Was an analysis done as to the state of the various threads when the test timed out (I'm assuming test timeout was the ultimate symptom)? Hopefully it would show a clue as to a thread busy or waiting that is preventing the node it lives on from shutting down.

A few weeks ago, I noticed another test (ugh, I forget which) reliably taking a long time to shut down (I forget if it led to a failure or not) and partially root caused it in this way. I have a shelved change to ZkContainer.close() to call shutdownNowAndAwaitTermination (with the "Now" in there, which wasn't there before). I noticed a test trying to shut down had cores that were stuck registering in ZK for some reason. I suppose that's unrelated to the failure here but without seeing the threads -- who knows.

[janhoy]

Was an analysis done as to the state of the various threads when the test timed out

No, I have not dived into the cause of hung nodes. I appreciate that all these failures may be a symptom of a real bug that prevents Solr from gracefully shutting down and giving up control / releasing zk.

I'll mark this as draft, and give some more time to fix the root instead of the symptom then...