vtdec: Fix a deadlock during ProRes playback

Sometimes a call to negotiate (and thus drain) can happen from the output loop
(via finish_frame()), which will tell VT to output all internal frames, but that won't succeed
if we happen to decide to wait for the queue to empty (because the loop is waiting for draining to finish and
will not make space in the queue!). This commit adds an override for the queue size limit if we're draining/flushing.

This bug could happen for any formats, but was especially obvious for ProRes, which has dpb_size of 0.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/6397>
This commit is contained in:
Piotr Brzeziński 2024-03-18 18:38:41 +01:00 committed by GStreamer Marge Bot
parent 17f92ab400
commit 112951ce55

View file

@ -1181,6 +1181,7 @@ gst_vtdec_session_output_callback (void *decompression_output_ref_con,
GstVtdec *vtdec = (GstVtdec *) decompression_output_ref_con;
GstVideoCodecFrame *frame = (GstVideoCodecFrame *) source_frame_ref_con;
GstVideoCodecState *state;
gboolean push_anyway = FALSE;
GST_LOG_OBJECT (vtdec, "got output frame %p %d and VT buffer %p", frame,
frame->decode_frame_number, image_buffer);
@ -1224,9 +1225,15 @@ gst_vtdec_session_output_callback (void *decompression_output_ref_con,
* to avoid processing too many frames ahead.
* The DPB * 2 size limit is completely arbitrary. */
g_mutex_lock (&vtdec->queue_mutex);
while (gst_queue_array_get_length (vtdec->reorder_queue) >
/* If negotiate() gets called from the output loop (via finish_frame()),
* it can attempt to drain and call VTDecompressionSessionWaitForAsynchronousFrames,
* which will lock up if we decide to wait in this callback, creating a deadlock. */
push_anyway = vtdec->is_flushing || vtdec->is_draining;
while (!push_anyway
&& gst_queue_array_get_length (vtdec->reorder_queue) >
vtdec->dbp_size * 2) {
g_cond_wait (&vtdec->queue_cond, &vtdec->queue_mutex);
push_anyway = vtdec->is_flushing || vtdec->is_draining;
}
gst_queue_array_push_sorted (vtdec->reorder_queue, frame, sort_frames_by_pts,