mrtd_bac_keyhandshake is failing

[Maddy5075]

Trying to read the data from US passport with a command
mrtdreader -p=xxxxxxxxx -b=yyyyyy -e=zzzzzz

Log as below

Target found!
debug libnfc.chip.pn53x InDataExchange
debug libnfc.chip.pn53x Timeout value: 500
debug libnfc.bus.i2c TX: 00 00 ff 0f f1 d4 40 01 00 a4 04 0c 07 a0 00 00 02 47 10 01 36 00
debug libnfc.bus.i2c wrote 22 bytes successfully.
debug libnfc.chip.pn53x PN53x ACKed
debug libnfc.driver.pn532_i2c timeout reached with no READY frame.
error libnfc.chip.pn53x Buffer size is too short: 300 available(s), 4294967295 needed
Unable to senddebug libnfc.chip.pn53x InRelease

nfc-list log
debug libnfc.chip.pn53x PN53x ACKed
1 ISO14443B passive target(s) found:
ISO/IEC 14443-4B (106 kbps) target:
PUPI: xx yy zz ww
Application Data: e1 f3 5e 11
Protocol Info: 73 81 a5

debug libnfc.chip.pn53x RFConfiguration

However when the time-out is increased from 500 to 5000, the nfc_initiator_transceive_bytes reports unable to send.
Target found!
debug libnfc.chip.pn53x InDataExchange
debug libnfc.chip.pn53x Timeout value: 5000
debug libnfc.bus.i2c TX: 00 00 ff 0f f1 d4 40 01 00 a4 04 0c 07 a0 00 00 02 47 10 01 36 00
debug libnfc.bus.i2c wrote 22 bytes successfully.
debug libnfc.chip.pn53x PN53x ACKed
debug libnfc.chip.pn53x Chip error: "Timeout" (01), returned error: "RF Transmission Error" (-20))
Unable to senddebug libnfc.chip.pn53x InRelease

Platform used: Rasp pi, PN532 over I2C