In order to change the border when <input-num> has the focus, that border must be on input-num, not ::part(input). Alternate configurations would involve javascript.
When you mouseover, the first contact is the border, so even if your intention is to directly hover over the spinner buttons, you must cross the border to get there. Depending on how fast you move your mouse, mouseover might fire with its target as this before it fires for the spinner. The event will fire again upon entering the spinner, so it all works out, but the idle image might flash before displaying the correct hover image.
I don't see a reasonable solution for this. Disabling the top, right, and bottom borders for mouseover means checking the event.clientX and y properties and the result is dubious, especially for the top and bottom borders. Other workarounds are equally bad, and this behavior is not awful, it's just a bit clumsy. I'm leaving this open so that others know it is a known issue.