Lossless mpeg2 transcode segfaults following [09ed2c69]

[stuartm]

Happens immediately and is 100% reproducible.

#0  0x00007ffff73511e6 in av_buffer_realloc (pbuf=pbuf@entry=0x733d90, size=2787) at libavutil/buffer.c:181
        buf = 0x7bb7c0
        tmp = <optimized out>
#1  0x00007ffff62a2a6b in av_grow_packet (pkt=0x733d90, grow_by=2371) at libavcodec/avpacket.c:124
        ret = <optimized out>
        new_size = <optimized out>
#2  0x000000000043bca0 in MPEG2frame::ensure_size (this=0x733d90, size=2755) at mpeg2fix.h:62
No locals.
#3  0x000000000043bcc9 in MPEG2frame::set_pkt (this=0x733d90, newpkt=0x7fffffffb900) at mpeg2fix.h:67
        data = 0x7387b0 "\300\375r"
#4  0x000000000043168a in MPEG2fixup::GetPoolFrame (this=0x7379a0, pkt=0x7fffffffb900) at mpeg2fix.cpp:1287
        f = 0x733d90
        frame_count = 6
        __FUNCTION__ = "GetPoolFrame"
#5  0x0000000000431e9d in MPEG2fixup::GetFrame (this=0x7379a0, pkt=0x7fffffffb900) at mpeg2fix.cpp:1385
        done = true
        tmpFrame = 0x0
        ret = 0
        __FUNCTION__ = "GetFrame"
#6  0x0000000000432303 in MPEG2fixup::FindStart (this=0x7379a0) at mpeg2fix.cpp:1434
        pkt = {buf = 0x737740, pts = 4374470014, dts = 4374470014, data = 0x78cac0 "", size = 2755, stream_index = 0, flags = 0, side_data = 0x0, 
          side_data_elems = 0, duration = 6, destruct = 0x7ffff62a28b0 <dummy_destruct_packet>, priv = 0x0, pos = 32712, convergence_duration = 0}
        found = {{d = 0x7fffede24840 <QMapData::shared_null>, e = 0x7fffede24840 <QMapData::shared_null>}}
        __FUNCTION__ = "FindStart"
#7  0x00000000004364c5 in MPEG2fixup::Start (this=0x7379a0) at mpeg2fix.cpp:2003
        lastPTS = 0
        frame_count = 0
        new_discard_state = 0
        ret = 32767
        af_dlta_cnt = {{d = 0x7fffede24840 <QMapData::shared_null>, e = 0x7fffede24840 <QMapData::shared_null>}}
        pkt = {buf = 0x2e, pts = 128, dts = 2961948932, 
          data = 0x7fffed9e6e24 <qstrtoll(char const*, char const**, int, bool*)+340> "\205\300tHD\211\377\350p\202\373\377\203\370\001D\211\366\031҃\342 \203\302\067)\326@\017\266\306\071D$\004~&I9\354v\211\273\377\377\377\377\353\244f.\017\037\204", size = 7571824, stream_index = 10, 
          flags = -308384220, side_data = 0xa00000000, side_data_elems = 10, duration = 0, destruct = 0x700000000, priv = 0x7fffffffbca8, 
          pos = 140737488338087, convergence_duration = 140737488338160}
        __FUNCTION__ = "Start"
        expectedvPTS = 0
        cutEndPTS = 0
        poq = {offset = {{d = 0x7fffffffbc20, e = 0x7fffffffbc20}}, orig = {{d = 0x7ffff7de6b99 <check_match.9436+89>, 
              e = 0x7ffff7de6b99 <check_match.9436+89>}}, keyList = {{p = {static shared_null = {ref = {_q_value = 274}, alloc = 0, begin = 0, 
                  end = 0, sharable = 1, array = {0x0}}, d = 0x1007fffedb577c0}, d = 0x1007fffedb577c0}}, vid_id = -309030208}
        expectedDTS = 0
        initPTS = 0
        origvPTS = 0
        cutStartPTS = 0
        deltaPTS = 0
        origaPTS = {0, 140737165195429, 0, 20, 140737488338464, 140737174306589, 4235024, 140737488338464, 140737488338464, 140737179854830, 
          140737488338240, 140737179858753, 1, 1411565652, 7226896, 140737301281858, 140737488338320, 140737488338336, 4235024, 140737488338304, 
          140737488338464, 140737179860930, 140733195844957, 140737301142832, 31039281974050141, 7577136, 140737488338480, 140737301142787, 
          140733242240233, 140737488338480, 1, 140737301142712}
        cutState = {{d = 0x7fffede24840 <QMapData::shared_null>, e = 0x7fffede24840 <QMapData::shared_null>}}
        lastRealvPkt = {buf = 0x7fffffffbe3f, pts = 0, dts = 140737488338511, data = 0x7fffffffbe4f "", size = 10, stream_index = 0, flags = 0, 
          side_data = 0x7fffffffbcd0, side_data_elems = -319805920, duration = 32767, destruct = 0x14, priv = 0x409f10 <_start>, 
          pos = 140737488345904, convergence_duration = 0}

[stuartm]

Possibly related to, or caused by, [09ed2c69354f6870667880aee16257b9600b01d8]

[stuartm]

Fixes

[stuartm]

I've attached a patch which fixes a couple of test cases for me. It's by no means a perfect solution, but if it works for others then I can start on a committable version.

[Stuart Morgan <smorgan@…>]

In bf3f16ab4cd2c6e76c57d11de70c427d47d1f49b/mythtv:

Fix a number of mis-uses of AVPacket in mythtranscode

Fixes some leaks, memory issues and segfaults.

We still mis-use AVPacket, but this at least lets lossless transcodes
with cut lists run to completion. Better fixes may come later when we
have more time to work on them.

Fixes #12280

[Stuart Morgan <smorgan@…>]

In 815dc5a3f9c218924519ca1ba63b724c696150bd/mythtv:

More lossless mythtranscode fixes

Lossless (mpeg2) transcodes in master now seem to work perfectly on every file
I've tried included files which have always failed to
transcode. Feedback to the forum, irc, mythtv-dev or mythtv-users list
would be appreciated. I'd like to know whether this resolves long
standing lossless mpeg2 transcoding failures for anyone else. I make no
promise to fix unresolved issues, but it would be good to know if
things have moved in the right direction.

Fixes #12280