hudiy.eu

Recently I was watching an interesting video by MattKC about Nintendo Wii U gamepad video stream reverse engineering and something caught my eye that I think would be helpful for Hudiy (linking a time stamped part of the video): https://youtu.be/si-TUEwC5_M?t=814

Similarly to my experience with Hudiy, openDsh, Crankshaft, OpenAuto Pro on Raspberry Pi, MattKC has found significant latency of h264 decoding on Raspberry Pi. While Hudiy does the best job with performance out of all the android auto emulators I've tried, at least on my setup - the latency is still fairly high.

Fortunately, MattKC was able to find and implement a fix that greatly reduced h264 decode latency, essentially leaving only a tiny 4 frame delay (for which currently the cause is unknown, this doesn't happen on Allwinner based Orange Pi Zero 3).

The video explains the issue and solution nicely.

I'm linking the actual relevant commit here: https://github.com/vanilla-wiiu/vanilla ... 4f46dc7115

And the Raspberry Pi forum post that discovered the fix: https://forums.raspberrypi.com/viewtopi ... 01#p334801

Could this also be applicable to Hudiy?

Thank you for your feedback and for the tip regarding SPS headers. The Android Auto H264 stream uses the Baseline Profile, which does not contain B-frames, so theoretically, the video decoder shouldn't buffer extra frames and introduce delay. Nevertheless, we've already run into a few quirks with the H264 hardware decoder on Raspberry Pi, so we'll test modifying the SPS values to see if it reduces latency. We'll let you know the results soon.

Thanks for the swift reply, hopefully improvements can be made

We ran a benchmark on the Raspberry Pi 4B and it looks like the VUI modifications have no impact on the number of buffered frames in the decoder or on the decode time. Also, the h264 encoder on the phone already sets max_dec_frame_buffering to 1 and num_reorder_frames to 0 by default.

However, we ran the same benchmark on x86_64 with VAAPI and indeed the number of frames buffered by the VAAPI decoder is 0 - which means it doesn't buffer any frames and immediately outputs what it decodes. We observed the exact same zero-buffering behavior with the HEVC decoder on both the Raspberry Pi 4B and Raspberry Pi 5.

Previously, we didn't measure this parameter in our benchmark (only the average frame decode time), so this is a great discovery - thank you! We will dig deeper into the h264 decoder driver on the Raspberry Pi to check if there is any way to decrease the number of buffered frames. For now, the best approach to minimize latency on the Raspberry Pi 4B and 5 is enabling useRpiDrm in main_configuration.json (https://github.com/wiboma/hudiy/blob/ma ... ndroidauto).

Glad to help

MattKC has posted a video on fixing the 4 frame delay: https://youtu.be/R3cpt9a8Cgc