Datafeed Retry Fails if Agent is down with LB in Front #827
Description
Bug Report
Steps to Reproduce:
- Setup an LB with an Agent.
- Start the Agent.
- Connect BDK to Agent.
- Stop the Agent.
- See that it stops trying to recover.
Expected Result:
It should continue to retry as this scenario can happen if maintenance is being done on the agent behind the LB or if even only one of the Agents becomes unhealthy and it happens to reach the agent before LB can take it off rotation.
Actual Result:
2025-07-01T10:44:36.332-04:00 ERROR 29488 --- [onyBdk_Datafeed] c.s.b.c.s.d.impl.AbstractDatafeedLoop : Network error occurred while trying to connect to the "AGENT" at the following address: https://agent.symphony.com:443/agent. Error while trying to validate certificate for the trust store. This type of error typically means that your network is using a self-signed certificate.
{}
jakarta.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:534) ~[jersey-apache-connector-3.1.9.jar:na]
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:674) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:709) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:703) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.1.9.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.1.9.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.1.9.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:391) ~[jersey-common-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:703) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:673) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:439) ~[jersey-client-3.1.9.jar:na]
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:345) ~[jersey-client-3.1.9.jar:na]
at com.symphony.bdk.http.jersey2.ApiClientJersey2.getResponse(ApiClientJersey2.java:207) ~[symphony-bdk-http-jersey2-3.1.1.jar:3.1.1]
at com.symphony.bdk.http.jersey2.ApiClientJersey2.invokeAPI(ApiClientJersey2.java:158) ~[symphony-bdk-http-jersey2-3.1.1.jar:3.1.1]
at com.symphony.bdk.gen.api.DatafeedApi.readDatafeedWithHttpInfo(DatafeedApi.java:388) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.gen.api.DatafeedApi.readDatafeed(DatafeedApi.java:321) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.service.datafeed.impl.DatafeedLoopV2.readEvents(DatafeedLoopV2.java:134) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.service.datafeed.impl.AbstractAckIdEventLoop.readAndHandleEvents(AbstractAckIdEventLoop.java:42) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.retry.RetryWithRecovery.executeOnce(RetryWithRecovery.java:105) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.retry.resilience4j.Resilience4jRetryWithRecovery.lambda$execute$1(Resilience4jRetryWithRecovery.java:65) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at io.github.resilience4j.retry.Retry.lambda$decorateCheckedSupplier$1(Retry.java:135) ~[resilience4j-retry-2.2.0.jar:2.2.0]
at io.github.resilience4j.retry.Retry.executeCheckedSupplier(Retry.java:350) ~[resilience4j-retry-2.2.0.jar:2.2.0]
at com.symphony.bdk.core.retry.resilience4j.Resilience4jRetryWithRecovery.execute(Resilience4jRetryWithRecovery.java:65) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.service.datafeed.impl.DatafeedLoopV2.runLoop(DatafeedLoopV2.java:103) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.service.datafeed.impl.AbstractDatafeedLoop.start(AbstractDatafeedLoop.java:90) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.core.service.datafeed.impl.AbstractAckIdEventLoop.start(AbstractAckIdEventLoop.java:19) ~[symphony-bdk-core-3.1.1.jar:3.1.1]
at com.symphony.bdk.spring.service.LoopAsyncLauncherService.uncheckedStart(LoopAsyncLauncherService.java:86) ~[symphony-bdk-core-spring-boot-starter-3.1.1.jar:3.1.1]
at com.symphony.bdk.http.api.tracing.MDCUtils$MdcRunnable.run(MDCUtils.java:59) ~[symphony-bdk-http-api-3.1.1.jar:3.1.1]
at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1719) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1518) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426) ~[na:na]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.14.jar:4.5.14]
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:486) ~[jersey-apache-connector-3.1.9.jar:na]
... 29 common frames omitted
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489) ~[na:na]
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478) ~[na:na]
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160) ~[na:na]
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510) ~[na:na]
... 44 common frames omitted
2025-07-01T10:44:36.338-04:00 INFO 29488 --- [ionShutdownHook] c.s.b.c.s.d.impl.AbstractDatafeedLoop : Stopping the datafeed loop (will happen once the current read is finished)...
2025-07-01T10:44:36.339-04:00 INFO 29488 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2025-07-01T10:44:36.341-04:00 INFO 29488 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2025-07-01T10:44:36.346-04:00 INFO 29488 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
Environment:
BDK 3.1.1 w/ Agent behind an LB.