Test program for real time video and audio processing with FFmpeg/libav*
This repository has been archived on 2026-11-17. You can view files and clone it, but you cannot make any changes to its state, such as pushing and creating new issues, pull requests or comments.
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Lucas Almeida 39a7ed3d26
Update README with current repository status
Update the repository README file with the current repository status, as
the next step involves archiving it.  Since this repo no longer has much
purpose, it will be archived for the reasons stated in the README.
2026-11-16 21:16:59 -03:00
src Add main source file 2026-10-15 08:29:23 -03:00
README.md Update README with current repository status 2026-11-16 21:16:59 -03:00

libav_rtsp_test

This small project started as an attempt to create a minimal reproduction of an issue that I'm facing in a larger project (hereinafter referred to simply as "Project"). It ended up shining some light onto what the issue I was experiencing was. In the interest of sharing my findings with everyone, I will share details about this issue.

The scope

I was using libav* for Project and I was having some sync issues with video and audio, in spite of the fact that both were coming from the same source. For context, the task was to receive a live stream containing a video and audio stream in real time, doing some video processing, and then muxing and livestreaming everything again. Because there was no audio processing done in Project, there was no need to transcode the audio stream, so copying audio packets was enough. The video stream, which had to go through this processing stage, had to be re-encoded.

The problem

As mentioned above, there was some noticeable delay in the video stream, causing the video and audio stream to be out of sync. Depending on certain circumstances, the delay would range anywhere from a few milliseconds to several seconds. Project had a very limited constraint, so both had to be as synchronized as possible.

After writing this minimal reproduction code, I narrowed down the issue: it was the video encoder adding the delay. Copying video packets (instead of processing them and re-encoding) confirmed this to be the case.

Conclusion

Plenty of encoders have some tuning options that allow them to take as little time as possible to output encoded packets. libx264 and NVIDIA's NVENC have specific options for real time encoding, which will help eliminate most of the delay. Other encoders, like libopenh264, not so much, sadly.

If you're learning how to work with libav* and you're running into this same problem, hopefully this saves you some time from troubleshooting/trial and error.

This repository will be archived, as it no longer serves any purpose. It will be kept public in the interest of transparency and helping others.

Big thanks to users on the #ffmpeg Libera.Chat IRC channel once again for helping me!