Not working with ffmpeg

[strongtz]

First of all, thanks a lot for your great work!
I tried this on rk3568 with mainline kernel (v6.2.10). vainfo works fine, but decoding h264 with ffmpeg doesn't seem to work:

$ LIBVA_DRIVERS_PATH=~/git/libva-v4l2/build/src LIBVA_DRIVER_NAME=v4l2 ffmpeg -v debug -hwaccel vaapi -i ~/Big_Buck_Bunny_1080_10s_30MB.mp4 -f null -

......

[vost#0:0/wrapped_avframe @ 0xaaaae4252b00] Created video stream from input stream 0:0
[AVHWDeviceContext @ 0xaaaae43a0ec0] Trying to use DRM render node for device 0.
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: VA-API version 1.22.0
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: User environment variable requested driver 'v4l2'
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: Trying to open /home/strongtz/git/libva-v4l2/build/src/v4l2_drv_video.so
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: Found init function __vaDriverInit_1_22
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0xaaaae43a0ec0] Initialised VAAPI connection: version 1.22
[AVHWDeviceContext @ 0xaaaae43a0ec0] Format 0x3231564e -> nv12.
[AVHWDeviceContext @ 0xaaaae43a0ec0] VAAPI driver: v4l2.
[AVHWDeviceContext @ 0xaaaae43a0ec0] Driver not found in known nonstandard list, using standard behaviour.

......

[h264 @ 0xaaaae424ddc0] Format vaapi chosen by get_format().
[h264 @ 0xaaaae424ddc0] Format vaapi requires hwaccel h264_vaapi initialisation.
[h264 @ 0xaaaae424ddc0] Considering format 0x3231564e -> nv12.
[h264 @ 0xaaaae424ddc0] Picked nv12 (0x3231564e) as best match for yuv420p.
[AVHWFramesContext @ 0xffff70172d10] Created surface 0.
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: deriveImage 2
[AVHWFramesContext @ 0xffff70172d10] Direct mapping disabled: deriving image does not work: 1 (operation failed).
[AVHWDeviceContext @ 0xaaaae43a0ec0] libva: Failed to create context: No surfaces to be created
[h264 @ 0xaaaae424ddc0] Failed to create decode context: 1 (operation failed).
[h264 @ 0xaaaae424ddc0] Failed to destroy decode context 0: 5 (invalid VAContextID).
[h264 @ 0xaaaae424ddc0] Failed setup for format vaapi: hwaccel initialisation returned error.
[h264 @ 0xaaaae424ddc0] Format vaapi not usable, retrying get_format() without it.
[h264 @ 0xaaaae424ddc0] Format yuv420p chosen by get_format().
[h264 @ 0xaaaae424ddc0] Reinit context to 1920x1088, pix_fmt: yuv420p

I added some debug prints, and the problem seems to be here:
https://github.com/mxsrc/libva-v4l2/blob/master/src/image.cc#L156
https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/hwcontext_vaapi.c#L672

[mxsrc]

Thanks for your input! I am trying to reproduce the error, assuming that my RK3399 is similar enough. So far I am getting a different (albeit still erroneous) behavior. Just to be sure, can you share the file you're trying to decode, so I am sure any difference isn't due to that?

[strongtz]

Sure. It's https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/1080/Big_Buck_Bunny_1080_10s_30MB.mp4

[strongtz]

I'm using Arch Linux though, so the ffmpeg version is quite new. It's v7.1

[mxsrc]

I had a look at the issue and there appears to be a combination of issues, I'll details them here here for transparency mostly, but also development journaling on my side:

1. Currently, libva-v4l2 allocates buffers for the encoded frames at a fixed 1MiB size. This was sufficient for my testing material, but is insufficient for the P-Frames of the video you are decoding, and likely plenty of others. In general, V4L2 can dynamically determine the appropriate size for buffers, but specifically for MPEG2 decoding this unfortunately does not work, or did not work when I first tried to use it. I'm preparing a patchset that will use the dynamic buffer size by default and have an exception for MPEG2.
2. FFmpeg 7.1 changed how it manages its buffer pool. This decouples the libVA surfaces from the decode context which so far we've relied on. I'm figuring out how to solve this in a generic way.

[chewi]

Any news on this front? I just got the same with a different H.264 video using FFmpeg 8.0.1 with jc-kynesim's patches on a Quartz64 (RK3566).

[mxsrc]

I was not able to look into this last year due to a new day job. My new year's resolution is to make some progress on this, so I might have a fix in some time.