I recall looking into this early on, when checks were still in beta, and it not working as desired. I should dig up slack history from the internal adobe channel to pull up details of the issues that were had, and see if we're still experiencing it.
If we can move away from PR events and instead rely on check events, then we can have the bot properly respond to "rerun check" UI interactions.