- C 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
Update the repository README file with the current repository status, as the next step involves archiving it. Since this repo no longer has much purpose, it will be archived for the reasons stated in the README. |
||
| src | ||
| README.md | ||
libav_rtsp_test
This small project started as an attempt to create a minimal reproduction of an issue that I'm facing in a larger project (hereinafter referred to simply as "Project"). It ended up shining some light onto what the issue I was experiencing was. In the interest of sharing my findings with everyone, I will share details about this issue.
The scope
I was using libav* for Project and I was having some sync issues with video and audio, in spite of the fact that both were coming from the same source. For context, the task was to receive a live stream containing a video and audio stream in real time, doing some video processing, and then muxing and livestreaming everything again. Because there was no audio processing done in Project, there was no need to transcode the audio stream, so copying audio packets was enough. The video stream, which had to go through this processing stage, had to be re-encoded.
The problem
As mentioned above, there was some noticeable delay in the video stream, causing the video and audio stream to be out of sync. Depending on certain circumstances, the delay would range anywhere from a few milliseconds to several seconds. Project had a very limited constraint, so both had to be as synchronized as possible.
After writing this minimal reproduction code, I narrowed down the issue: it was the video encoder adding the delay. Copying video packets (instead of processing them and re-encoding) confirmed this to be the case.
Conclusion
Plenty of encoders have some tuning options that allow them to take as little time as possible to output encoded packets. libx264 and NVIDIA's NVENC have specific options for real time encoding, which will help eliminate most of the delay. Other encoders, like libopenh264, not so much, sadly.
If you're learning how to work with libav* and you're running into this same problem, hopefully this saves you some time from troubleshooting/trial and error.
This repository will be archived, as it no longer serves any purpose. It will be kept public in the interest of transparency and helping others.
Big thanks to users on the #ffmpeg Libera.Chat IRC channel once again for helping me!