_payload_buffer in dtvrecorder.cpp can become huge in size (and eventually resulting in allocation failures)

[erik-mythtv@…]

Hi,

As of recently my mythtv 0.26.1 environment started to crash once it has tried to record a program from a certain DVB-C channel. The issue only seems to happen on one particular channel. Using gdb I've managed to catch the crash:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xa3cffb70 (LWP 18081)]
0x0088c416 in __kernel_vsyscall ()
(gdb) bt
#0  0x0088c416 in __kernel_vsyscall ()
#1  0x02a9ead1 in raise () from /lib/libc.so.6
#2  0x02aa03aa in abort () from /lib/libc.so.6
#3  0x0469db87 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#4  0x0469b9e6 in ?? () from /usr/lib/libstdc++.so.6
#5  0x0469ba23 in std::terminate() () from /usr/lib/libstdc++.so.6
#6  0x0469bb62 in __cxa_throw () from /usr/lib/libstdc++.so.6
#7  0x0469c0f7 in operator new(unsigned int) () from /usr/lib/libstdc++.so.6
#8  0x0183c492 in allocate (this=0x8d1162c, __position=0 '\000', __n=188, __x=@0xa3cfedbf) at /usr/include/c++/4.4.7/ext/new_allocator.h:89
#9  _M_allocate (this=0x8d1162c, __position=0 '\000', __n=188, __x=@0xa3cfedbf) at /usr/include/c++/4.4.7/bits/stl_vector.h:140
#10 std::vector<unsigned char, std::allocator<unsigned char> >::_M_fill_insert (this=0x8d1162c, __position=0 '\000', __n=188, __x=@0xa3cfedbf) at /usr/include/c++/4.4.7/bits/vector.tcc:414
#11 0x01c3586c in insert (this=0x8d11448, tspacket=..., insert=false) at /usr/include/c++/4.4.7/bits/stl_vector.h:851
#12 resize (this=0x8d11448, tspacket=..., insert=false) at /usr/include/c++/4.4.7/bits/stl_vector.h:557
#13 DTVRecorder::BufferedWrite (this=0x8d11448, tspacket=..., insert=false) at dtvrecorder.cpp:303
#14 0x01c35e6d in DTVRecorder::ProcessAVTSPacket (this=0x8d11448, tspacket=...) at dtvrecorder.cpp:1387
#15 0x01c3d669 in DTVRecorder::ProcessVideoTSPacket (this=0x8d11448, tspacket=...) at dtvrecorder.cpp:1334
#16 0x01838139 in MPEGStreamData::ProcessTSPacket (this=0x8d1cbf8, tspacket=...) at mpeg/mpegstreamdata.cpp:1060
#17 0x0182b915 in MPEGStreamData::ProcessData (this=0x8d1cbf8,
    buffer=0xa3f2a008 "G\002Y\034\266\353\063\333Q\261l;\256-gzh}\001\365\003\253-y\232e1P\033_'u\003y\256\234\245\027Q\343\071\252\253\216\063v\240O\f\003$\271j\364\346\f\353\030\276@U\204\001\224h\225\016\272\220\322\\.\002pml!\225\233\361)\315 \342\005\023\350̽%\225\201%];2֙/\226\217tRzW>1\372\227\257o\325P¾\243Zk&\t\006&\264\n\254\016`/W\264\301\310b7'\367\026\231b.\370x\304\062X\017\243\352Ov\306JU\300le\353* \332\034\226]\376\237i\314S՞b1O\241\344\205\030\223\354\240\031k\224I\016\306\v\327G", len=27448) at mpeg/mpegstreamdata.cpp:1024
#18 0x01d09d23 in DVBStreamHandler::RunTS (this=0xa3d31730) at dvbstreamhandler.cpp:267
#19 0x01d0aeea in DVBStreamHandler::run (this=0xa3d31730) at dvbstreamhandler.cpp:118
#20 0x0053c673 in MThreadInternal::run (this=0xb574ced8) at mthread.cpp:79
#21 0x02868e4f in ?? () from /usr/lib/libQtCore.so.4
#22 0x02460a49 in start_thread () from /lib/libpthread.so.0
#23 0x02b56aae in clone () from /lib/libc.so.6

As this isn't a regular crash, but an abort this gave me the idea that something was messed up regarding memory allocation. When I reproduce the issue and observe the memory usage of the mythbackend process I noticed that once the recording got started, the memory usage started to increase over time (and eventually leading to the crash).

When I look at the details of the recording in mythweb I noticed that the size of the recording in question always remained 376 bytes never to increase.

When stepping through the code using gdb (with a breakpoint set on DTVRecorder::BufferedWrite?) I noticed that the _payload_buffer which is used in dtvrecorder.cpp always was becoming larger in size and never got reduced in size:

(gdb) print _payload_buffer
$1 = std::vector of length 4700, capacity 9588 = {71 'G', 66 'B', 90 'Z', 50 '2', 1 '\001', 64 '@', 252 '\374', 62 '>', 230 '\346', 250 '\372', 2 '\002', 241 '\361', 237 '\355', 78 'N', 97 'a', 93 ']', 149 '\225', 54 '6', 47 '/', 
  118 'v', 26 '\032', 223 '\337', 235 '\353', 105 'i', 109 'm', 48 '0', 139 '\213', 211 '\323', 105 'i', 180 '\264', 121 'y', 15 '\017', 100 'd', 71 'G', 182 '\266', 136 '\210', 97 'a', 51 '3', 74 'J', 72 'H', 208 '\320', 180 '\264', 
  102 'f', 199 '\307', 66 'B', 17 '\021', 76 'L', 198 '\306', 216 '\330', 117 'u', 209 '\321', 197 '\305', 231 '\347', 111 'o', 65 'A', 4 '\004', 244 '\364', 247 '\367', 243 '\363', 111 'o', 0 '\000', 237 '\355', 147 '\223', 
  140 '\214', 180 '\264', 185 '\271', 156 '\234', 94 '^', 192 '\300', 140 '\214', 104 'h', 172 '\254', 79 'O', 216 '\330', 154 '\232', 104 'h', 115 's', 172 '\254', 119 'w', 253 '\375', 146 '\222', 125 '}', 134 '\206', 130 '\202', 
  9 '\t', 32 ' ', 54 '6', 58 ':', 206 '\316', 141 '\215', 140 '\214', 131 '\203', 209 '\321', 208 '\320', 219 '\333', 80 'P', 26 '\032', 79 'O', 142 '\216', 32 ' ', 206 '\316', 190 '\276', 125 '}', 101 'e', 236 '\354', 48 '0', 51 '3', 
  76 'L', 240 '\360', 37 '%', 239 '\357', 246 '\366', 237 '\355', 113 'q', 63 '?', 113 'q', 230 '\346', 39 '\'', 124 '|', 39 '\'', 187 '\273', 5 '\005', 100 'd', 54 '6', 66 'B', 54 '6', 126 '~', 5 '\005', 143 '\217', 95 '_', 
  131 '\203', 187 '\273', 60 '<', 149 '\225', 28 '\034', 120 'x', 215 '\327', 73 'I', 231 '\347', 191 '\277', 245 '\365', 222 '\336', 4 '\004', 146 '\222', 106 'j', 43 '+', 13 '\r', 209 '\321', 57 '9', 116 't', 180 '\264', 57 '9', 
  71 'G', 206 '\316', 107 'k', 166 '\246', 50 '2', 21 '\025', 71 'G', 15 '\017', 225 '\341', 177 '\261', 3 '\003', 62 '>', 219 '\333', 10 '\n', 90 'Z', 166 '\246', 156 '\234', 2 '\002', 225 '\341', 49 '1', 38 '&', 158 '\236', 41 ')', 
  50 '2', 141 '\215', 25 '\031', 18 '\022', 214 '\326', 169 '\251', 192 '\300', 63 '?', 12 '\f', 113 'q', 211 '\323', 3 '\003', 132 '\204', 71 'G', 2 '\002', 89 'Y', 17 '\021', 140 '\214', 44 ',', 5 '\005', 142 '\216', 131 '\203', 
  173 '\255', 183 '\267', 3 '\003'...}

(gdb) print _payload_buffer
$2 = std::vector of length 12181648, capacity 19636224 = {71 'G', 66 'B', 90 'Z', 50 '2', 1 '\001', 64 '@', 252 '\374', 62 '>', 230 '\346', 250 '\372', 2 '\002', 241 '\361', 237 '\355', 78 'N', 97 'a', 93 ']', 149 '\225', 54 '6', 
  47 '/', 118 'v', 26 '\032', 223 '\337', 235 '\353', 105 'i', 109 'm', 48 '0', 139 '\213', 211 '\323', 105 'i', 180 '\264', 121 'y', 15 '\017', 100 'd', 71 'G', 182 '\266', 136 '\210', 97 'a', 51 '3', 74 'J', 72 'H', 208 '\320', 
  180 '\264', 102 'f', 199 '\307', 66 'B', 17 '\021', 76 'L', 198 '\306', 216 '\330', 117 'u', 209 '\321', 197 '\305', 231 '\347', 111 'o', 65 'A', 4 '\004', 244 '\364', 247 '\367', 243 '\363', 111 'o', 0 '\000', 237 '\355', 
  147 '\223', 140 '\214', 180 '\264', 185 '\271', 156 '\234', 94 '^', 192 '\300', 140 '\214', 104 'h', 172 '\254', 79 'O', 216 '\330', 154 '\232', 104 'h', 115 's', 172 '\254', 119 'w', 253 '\375', 146 '\222', 125 '}', 134 '\206', 
  130 '\202', 9 '\t', 32 ' ', 54 '6', 58 ':', 206 '\316', 141 '\215', 140 '\214', 131 '\203', 209 '\321', 208 '\320', 219 '\333', 80 'P', 26 '\032', 79 'O', 142 '\216', 32 ' ', 206 '\316', 190 '\276', 125 '}', 101 'e', 236 '\354', 
  48 '0', 51 '3', 76 'L', 240 '\360', 37 '%', 239 '\357', 246 '\366', 237 '\355', 113 'q', 63 '?', 113 'q', 230 '\346', 39 '\'', 124 '|', 39 '\'', 187 '\273', 5 '\005', 100 'd', 54 '6', 66 'B', 54 '6', 126 '~', 5 '\005', 143 '\217', 
  95 '_', 131 '\203', 187 '\273', 60 '<', 149 '\225', 28 '\034', 120 'x', 215 '\327', 73 'I', 231 '\347', 191 '\277', 245 '\365', 222 '\336', 4 '\004', 146 '\222', 106 'j', 43 '+', 13 '\r', 209 '\321', 57 '9', 116 't', 180 '\264', 
  57 '9', 71 'G', 206 '\316', 107 'k', 166 '\246', 50 '2', 21 '\025', 71 'G', 15 '\017', 225 '\341', 177 '\261', 3 '\003', 62 '>', 219 '\333', 10 '\n', 90 'Z', 166 '\246', 156 '\234', 2 '\002', 225 '\341', 49 '1', 38 '&', 158 '\236', 
  41 ')', 50 '2', 141 '\215', 25 '\031', 18 '\022', 214 '\326', 169 '\251', 192 '\300', 63 '?', 12 '\f', 113 'q', 211 '\323', 3 '\003', 132 '\204', 71 'G', 2 '\002', 89 'Y', 17 '\021', 140 '\214', 44 ',', 5 '\005', 142 '\216', 
  131 '\203', 173 '\255', 183 '\267', 3 '\003'...}

(gdb) print _payload_buffer
$3 = std::vector of length 52122060, capacity 78544896 = {71 'G', 66 'B', 90 'Z', 50 '2', 1 '\001', 64 '@', 252 '\374', 62 '>', 230 '\346', 250 '\372', 2 '\002', 241 '\361', 237 '\355', 78 'N', 97 'a', 93 ']', 149 '\225', 54 '6', 
  47 '/', 118 'v', 26 '\032', 223 '\337', 235 '\353', 105 'i', 109 'm', 48 '0', 139 '\213', 211 '\323', 105 'i', 180 '\264', 121 'y', 15 '\017', 100 'd', 71 'G', 182 '\266', 136 '\210', 97 'a', 51 '3', 74 'J', 72 'H', 208 '\320', 
  180 '\264', 102 'f', 199 '\307', 66 'B', 17 '\021', 76 'L', 198 '\306', 216 '\330', 117 'u', 209 '\321', 197 '\305', 231 '\347', 111 'o', 65 'A', 4 '\004', 244 '\364', 247 '\367', 243 '\363', 111 'o', 0 '\000', 237 '\355', 
  147 '\223', 140 '\214', 180 '\264', 185 '\271', 156 '\234', 94 '^', 192 '\300', 140 '\214', 104 'h', 172 '\254', 79 'O', 216 '\330', 154 '\232', 104 'h', 115 's', 172 '\254', 119 'w', 253 '\375', 146 '\222', 125 '}', 134 '\206', 
  130 '\202', 9 '\t', 32 ' ', 54 '6', 58 ':', 206 '\316', 141 '\215', 140 '\214', 131 '\203', 209 '\321', 208 '\320', 219 '\333', 80 'P', 26 '\032', 79 'O', 142 '\216', 32 ' ', 206 '\316', 190 '\276', 125 '}', 101 'e', 236 '\354', 
  48 '0', 51 '3', 76 'L', 240 '\360', 37 '%', 239 '\357', 246 '\366', 237 '\355', 113 'q', 63 '?', 113 'q', 230 '\346', 39 '\'', 124 '|', 39 '\'', 187 '\273', 5 '\005', 100 'd', 54 '6', 66 'B', 54 '6', 126 '~', 5 '\005', 143 '\217', 
  95 '_', 131 '\203', 187 '\273', 60 '<', 149 '\225', 28 '\034', 120 'x', 215 '\327', 73 'I', 231 '\347', 191 '\277', 245 '\365', 222 '\336', 4 '\004', 146 '\222', 106 'j', 43 '+', 13 '\r', 209 '\321', 57 '9', 116 't', 180 '\264', 
  57 '9', 71 'G', 206 '\316', 107 'k', 166 '\246', 50 '2', 21 '\025', 71 'G', 15 '\017', 225 '\341', 177 '\261', 3 '\003', 62 '>', 219 '\333', 10 '\n', 90 'Z', 166 '\246', 156 '\234', 2 '\002', 225 '\341', 49 '1', 38 '&', 158 '\236', 
  41 ')', 50 '2', 141 '\215', 25 '\031', 18 '\022', 214 '\326', 169 '\251', 192 '\300', 63 '?', 12 '\f', 113 'q', 211 '\323', 3 '\003', 132 '\204', 71 'G', 2 '\002', 89 'Y', 17 '\021', 140 '\214', 44 ',', 5 '\005', 142 '\216', 
  131 '\203', 173 '\255', 183 '\267', 3 '\003'...}

(gdb) print _payload_buffer
$4 = std::vector of length 133599944, capacity 157089792 = {71 'G', 66 'B', 90 'Z', 50 '2', 1 '\001', 64 '@', 252 '\374', 62 '>', 230 '\346', 250 '\372', 2 '\002', 241 '\361', 237 '\355', 78 'N', 97 'a', 93 ']', 149 '\225', 54 '6', 
  47 '/', 118 'v', 26 '\032', 223 '\337', 235 '\353', 105 'i', 109 'm', 48 '0', 139 '\213', 211 '\323', 105 'i', 180 '\264', 121 'y', 15 '\017', 100 'd', 71 'G', 182 '\266', 136 '\210', 97 'a', 51 '3', 74 'J', 72 'H', 208 '\320', 
  180 '\264', 102 'f', 199 '\307', 66 'B', 17 '\021', 76 'L', 198 '\306', 216 '\330', 117 'u', 209 '\321', 197 '\305', 231 '\347', 111 'o', 65 'A', 4 '\004', 244 '\364', 247 '\367', 243 '\363', 111 'o', 0 '\000', 237 '\355', 
  147 '\223', 140 '\214', 180 '\264', 185 '\271', 156 '\234', 94 '^', 192 '\300', 140 '\214', 104 'h', 172 '\254', 79 'O', 216 '\330', 154 '\232', 104 'h', 115 's', 172 '\254', 119 'w', 253 '\375', 146 '\222', 125 '}', 134 '\206', 
  130 '\202', 9 '\t', 32 ' ', 54 '6', 58 ':', 206 '\316', 141 '\215', 140 '\214', 131 '\203', 209 '\321', 208 '\320', 219 '\333', 80 'P', 26 '\032', 79 'O', 142 '\216', 32 ' ', 206 '\316', 190 '\276', 125 '}', 101 'e', 236 '\354', 
  48 '0', 51 '3', 76 'L', 240 '\360', 37 '%', 239 '\357', 246 '\366', 237 '\355', 113 'q', 63 '?', 113 'q', 230 '\346', 39 '\'', 124 '|', 39 '\'', 187 '\273', 5 '\005', 100 'd', 54 '6', 66 'B', 54 '6', 126 '~', 5 '\005', 143 '\217', 
  95 '_', 131 '\203', 187 '\273', 60 '<', 149 '\225', 28 '\034', 120 'x', 215 '\327', 73 'I', 231 '\347', 191 '\277', 245 '\365', 222 '\336', 4 '\004', 146 '\222', 106 'j', 43 '+', 13 '\r', 209 '\321', 57 '9', 116 't', 180 '\264', 
  57 '9', 71 'G', 206 '\316', 107 'k', 166 '\246', 50 '2', 21 '\025', 71 'G', 15 '\017', 225 '\341', 177 '\261', 3 '\003', 62 '>', 219 '\333', 10 '\n', 90 'Z', 166 '\246', 156 '\234', 2 '\002', 225 '\341', 49 '1', 38 '&', 158 '\236', 
  41 ')', 50 '2', 141 '\215', 25 '\031', 18 '\022', 214 '\326', 169 '\251', 192 '\300', 63 '?', 12 '\f', 113 'q', 211 '\323', 3 '\003', 132 '\204', 71 'G', 2 '\002', 89 'Y', 17 '\021', 140 '\214', 44 ',', 5 '\005', 142 '\216', 
  131 '\203', 173 '\255', 183 '\267', 3 '\003'...}

I guess there are situations where the _payload_buffer might not be flushed automatically.

I've attached the 376 byte recording to this ticket.

I'm not familiar enough with MPEG and the MythTV code to do more investigations, but hopefully this information can help you guys in finding the real cause of this issue.

[erik-mythtv@…]

376 byte recording

[erik-mythtv@…]

Issue also happens on mythtv 0.27

[jpoet]

Can you please attach the backend log, with "-v record".

Is this H.264 or MPEG2?

[erik-mythtv@…]

mythbackend output with -v record

[erik-mythtv@…]

I just attached the mythbackend output with -v record. The DVB-C channel in question is an MPEG2 stream

[jpoet]

Sorry, please do that log again with "-v record --loglevel debug". Thanks.

[erik-mythtv@…]

mythbackend output with -v record --loglevel debug

[erik-mythtv@…]

I just attached an updated logfile to this ticket

[jpoet]

That log indicates that Myth is not seeing any frames in the MPEG2 stream. It indicates that this channel is encrypted, but also indicates that it is able to decrypt it.

I am attaching a patch for you to try. It is a hack which tells Myth to write the contents of the stream if 10MB are processed without seeing any frames. If this hack works for you, it should allow you to make a larger sample available, so we can see why the frames are not being parsed.

[erik-mythtv@…]

With the patch applied I now see messages in the log like: DTVRec[1]: Frame @ 376 + 10485888 = 10486264 The updated log is now attached to this bug. Eventually the recording is being written to disk, but it is too large to upload (613MB). If I try to open the recording (the file on disk) using mplayer then it fails to play

[erik-mythtv@…]

debug output with Hack-MPEG2-noFrame.patch

[jpoet]

The fact that mplayer will not play the video implies that it also cannot find any frames. My best guess is, that while the logs indicate that Myth is able to decrypt the channel, it is not decrypting it successfully.

You could try running mythtv-setup and re-scanning the channels. You could try contacting your TV provider and asking if they have changed something about how they are broadcasting that channel -- Maybe they have accidentally change some encoding parameter, and are able to fix it at their end.

If you have some other linux utility that allows you to successfully record that channel, then we can revisit why MythTV cannot. As it stands now, however, it does not look like there is anything MythTV can be doing differently that would help.