backend SegFault

[Eric Fontaine <ericrfontaine@…>]

I'm getting a seg fault about a minute after mythbackend starts. I've attached the output of running mythbackend in gdb and included the backtrace at the end. I consistently get this same error. mythtv has been working for a while, although it always has issues like random crashes, but in the past I have always been able to restart mythbackend successfully. I am running on a up-to-date arch linux VM running inside VirtualBox? inside Windows (I don't think the VM is at fault, since has been working for a year+). Let me know what else I can provide to help debug this issue.

[Eric Fontaine <ericrfontaine@…>]

mythbackend output in GDB w/backtrace

[Eric Fontaine <ericrfontaine@…>]

I started with another arch linux VM and reinstalled mythtv, and I got this other segmentation fault on the first time I tried to record a show...I don't know if this is related or not:

2016-05-25 04:15:54.785157 I  TVRec[1]: TuningNewRecorder - CreateRecorder()
2016-05-25 04:15:54.795610 E  RecBase[1](1318B0F6-0): SetStrOption(...recordingtype): Option not in profile.
2016-05-25 04:16:04.701216 I  TVRec[1]: Changing from WatchingLiveTV to None
2016-05-25 04:16:04.715746 E  HDHRSH(1318B0F6-0): UpdateFilters called in wrong tune mode
2016-05-25 04:16:04.746388 N  Finished Recording: Container: MPEG2-TS Video Codec: mpeg2video (720x480 A/R: 2 29.97fps) Audio Codec: ac3
2016-05-25 04:16:04.748311 I  TVRec[1]: FinishedRecording(1002_2016-05-25T08:15:54Z) damaged recq:<RecordingQuality overall_score="0" key="1002_2016-05-25T08:15:54Z" countinuity_error_count="0" packet_count="48547">
    <Gap start="2016-05-25T08:00:00Z" end="2016-05-25T08:15:54Z" duration="954" />
    <Gap start="2016-05-25T08:16:03Z" end="2016-05-25T09:00:00Z" duration="2636" />
</RecordingQuality>

2016-05-25 04:16:04.781386 I  Playback sock(224c500) 'efjz.in' disconnected
2016-05-25 04:16:13.231822 I  Reschedule requested for MATCH 3 0 0 - SaveRule Channel 2 Action News at 4:30AM
2016-05-25 04:16:13.257504 I  Scheduled 1 items in 0.0 = 0.01 match + 0.00 check + 0.00 place
2016-05-25 04:16:20.059217 I  Reschedule requested for MATCH 3 0 0 - DeleteRule Channel 2 Action News at 4:30AM
2016-05-25 04:16:20.086865 I  Scheduled 0 items in 0.0 = 0.01 match + 0.00 check + 0.01 place
2016-05-25 04:16:22.028309 I  Reschedule requested for MATCH 4 0 0 - SaveRule Channel 2 Action News at 4:30AM
2016-05-25 04:16:22.077230 I  Scheduled 18 items in 0.0 = 0.01 match + 0.00 check + 0.03 place
2016-05-25 04:17:25.486794 I  Reschedule requested for MATCH 5 0 0 - SaveRule Good Morning America
2016-05-25 04:17:25.564619 I  Scheduled 44 items in 0.1 = 0.01 match + 0.00 check + 0.06 place
2016-05-25 04:17:32.833828 I  Reschedule requested for MATCH 6 0 0 - SaveRule Channel 2 Action News
2016-05-25 04:17:33.209717 I  Scheduled 142 items in 0.4 = 0.01 match + 0.00 check + 0.35 place
2016-05-25 04:17:40.455158 I  BackendContext: Frontend 'efjz.in' disconnected.
2016-05-25 04:17:40.455174 I  Playback sock(22426e0) 'efjz.in' disconnected
2016-05-25 04:17:40.455974 I  Monitor sock(224be20) 'efjz.in' disconnected
2016-05-25 04:22:00.241795 I  Starting process manager
2016-05-25 04:22:00.241939 I  Starting process signal handler
2016-05-25 04:22:00.250139 I  Starting IO manager (write)
2016-05-25 04:22:00.251285 I  Starting IO manager (read)
2016-05-25 04:22:00.670504 I  MainServer: MainServer::ANN Monitor
2016-05-25 04:22:00.670534 I  MainServer: adding: efjz.in(224c500) as a client (events: 0)
2016-05-25 04:22:00.672906 I  MainServer: MainServer::ANN Monitor
2016-05-25 04:22:00.672926 I  MainServer: adding: efjz.in(2083130) as a client (events: 1)
2016-05-25 04:22:00.736054 I  MainServer: MainServer::ANN Monitor
2016-05-25 04:22:00.736069 I  MainServer: adding: efjz.in(22526f0) as a client (events: 0)
2016-05-25 04:22:00.738009 I  MainServer: MainServer::ANN Monitor
2016-05-25 04:22:00.738022 I  MainServer: adding: efjz.in(2253110) as a client (events: 1)
2016-05-25 04:22:01.876236 I  Monitor sock(224c500) 'efjz.in' disconnected
2016-05-25 04:22:01.876879 I  Monitor sock(2083130) 'efjz.in' disconnected
2016-05-25 04:22:01.916899 E  Preview: Encountered problems running '/usr/bin/mythpreviewgen --size 0x0 --chanid 1002 --starttime 20160525081553 --outfile /var/lib/mythtv/recordings/1002_20160525081553.ts.png' - (128)
2016-05-25 04:22:01.917404 C  Received Segmentation fault: Code 1, PID 16, UID 0, Value 0x0000125f
Segmentation fault (core dumped)

[Eric Fontaine <ericrfontaine@…>]

Chrashed after changing channels

[Jonathan Isom <jeisom@…>]

Hi. I, also on Arch Linux, am getting the same backtrace. I am just running the backend with mythweb and the built-in web server. It will record and display everything fine with mythweb except when I go to the "Recorded Programs" page. If I go to the built-in server or the "Recorded Programs" page it crashes.

[Jonathan Isom <jeisom@…>]

(gdb) backtrace #0 0x00007f03923abd70 in QMetaObject::indexOfClassInfo(char const*) const () from /usr/lib/libQt5Core.so.5 #1 0x00007f039bec40a8 in XmlSerializer::GetContentName?(QString const&, QMetaObject const*, QMetaProperty const*) () from /usr/lib/libmythupnp-0.28.so.0 #2 0x00007f039bec3624 in XmlSerializer::AddProperty?(QString const&, QVariant const&, QMetaObject const*, QMetaProperty const*) () from /usr/lib/libmythupnp-0.28.so.0 #3 0x00007f039bec2935 in Serializer::Serialize(QVariant const&, QString const&) () from /usr/lib/libmythupnp-0.28.so.0 #4 0x00007f039be9f96f in ServiceHost::FormatResponse?(HTTPRequest*, QVariant) () from /usr/lib/libmythupnp-0.28.so.0 #5 0x00007f039be9ed1f in ServiceHost::ProcessRequest?(HTTPRequest*) () from /usr/lib/libmythupnp-0.28.so.0 #6 0x00007f039be53e4d in HttpServer::DelegateRequest?(HTTPRequest*) () from /usr/lib/libmythupnp-0.28.so.0 #7 0x00007f039be55077 in ?? () from /usr/lib/libmythupnp-0.28.so.0 #8 0x00007f039ba313f8 in ?? () from /usr/lib/libmythbase-0.28.so.0 #9 0x00007f039ba2ddcd in ?? () from /usr/lib/libmythbase-0.28.so.0 #10 0x00007f03921cc1d8 in ?? () from /usr/lib/libQt5Core.so.5 #11 0x00007f03946ec474 in start_thread () from /usr/lib/libpthread.so.0 #12 0x00007f03915c669d in clone () from /usr/lib/libc.so.6

[Jonathan Isom <jeisom@…>]

gdb output Current fixes/0.28 as of today

[stuartm]

Can you both include the version of QT you are using? There is a known QT bug in 5.6 which results in a similar/identical backtrace.

[xtragb@…]

Sounds like a duplicate of ticket#12668.

[Jonathan Isom <jeisom@…>]

I have Qt 5.6 installed. Any easy work around?

[xtragb@…]

Fairly easy...I posted a workaround for Arch Linux users here ​https://bbs.archlinux.org/viewtopic.php?pid=1631996#p1631996.

[Jonathan Isom <jeisom@…>]

At least in my case, it is the Qt 5.6 bug. Thanks.

[PlanetEater <zkr@…>]

I think this is a real bug in myth, seemingly masked by some magic performed by gcc versions prior to gcc 6. Pull request for the proposed fix: ​https://github.com/MythTV/mythtv/pull/122

[xtragb@…]

backtrace after applying patch from commit 2dd09d2fddb69d6807c89e8efe997e303cdfb203

[xtragb@…]

Replying to PlanetEater <zkr@…>:

I think this is a real bug in myth, seemingly masked by some magic performed by gcc versions prior to gcc 6. Pull request for the proposed fix: ​https://github.com/MythTV/mythtv/pull/122

Didn't work for me. New backtrace attached.

[PlanetEater <zkr@…>]

That's odd...

• How do You trigger the crash?
• You are on Arch, right? Could You rebuild with debug symbols?
  (Put options=('debug' 'strip') into the PKGBUILD, then install both resulting packages)

[xtragb@…]

The crash was triggered just by starting mythbackend - but the good news is I reinstalled qt5 and mythtv (built with your patch) and now I can no longer reproduce it. The problem must have been specific to my install.

BTW, the backtrace I attached yesterday was produced with a debug build of mythtv.

[Jonathan Isom <jeisom@…>]

Doing the same action results in this now:

#0  0x00007f190b840840 in QMetaObject::indexOfClassInfo(char const*) const () from /usr/lib/libQt5Core.so.5
#1  0x00007f19150d7f6e in XmlSerializer::GetContentName(QString const&, QMetaObject const*, QMetaProperty const*) () from /usr/lib/libmythupnp-0.28.so.0
#2  0x00007f19150d74ea in XmlSerializer::AddProperty(QString const&, QVariant const&, QMetaObject const*, QMetaProperty const*) () from /usr/lib/libmythupnp-0.28.so.0
#3  0x00007f19150d6811 in Serializer::Serialize(QVariant const&, QString const&) () from /usr/lib/libmythupnp-0.28.so.0
#4  0x00007f19150b38a7 in ServiceHost::FormatResponse(HTTPRequest*, QVariant) () from /usr/lib/libmythupnp-0.28.so.0
#5  0x00007f19150b2c57 in ServiceHost::ProcessRequest(HTTPRequest*) () from /usr/lib/libmythupnp-0.28.so.0
#6  0x00007f1915067d2d in HttpServer::DelegateRequest(HTTPRequest*) () from /usr/lib/libmythupnp-0.28.so.0
#7  0x00007f1915068f57 in ?? () from /usr/lib/libmythupnp-0.28.so.0
#8  0x00007f1914c44498 in ?? () from /usr/lib/libmythbase-0.28.so.0
#9  0x00007f1914c40e6d in ?? () from /usr/lib/libmythbase-0.28.so.0
#10 0x00007f190b65cd78 in ?? () from /usr/lib/libQt5Core.so.5
#11 0x00007f190b39a484 in start_thread () from /usr/lib/libpthread.so.0
#12 0x00007f190a8386dd in clone () from /usr/lib/libc.so.6

[PlanetEater <zkr@…>]

I can't see any difference between this backtrace and the one You submitted 3 weeks ago. Did You rebuild mytbackend from source, after applying the patch I submitted? My pull request is still waiting to be accepted, so the fix is not yet in the official repo.

[Jonathan Isom <jeisom@…>]

I built this from fit this morning. A version of the patch appears to already be applied.

[Jonathan Isom <jeisom@…>]

Sorry about double. Now using Qt 5.7 from arch. I did not rebuild it with the GCC flag change yet.

[xtragb@…]

If you build mythtv with the patch from PlanetEater? applied, you should no longer need the modified Qt build. I am using qt5-base 5.6.1-2 straight from the Extra repo on Arch and I am no longer seeing any segfaults.

[Jonathan Isom <jeisom@…>]

I am pulling from: ​git://github.com/MythTV/mythtv.git#branch=fixes/0.28

Let me know if this is the wrong repo.

Here is the code in that repo and below it from the pull request. I got the crash from the above repo. Obviously the code looks different, but should not change how it functions. If there is a crash with one and not the other then it would be a compiler error.

void XmlSerializer::AddProperty( const QString       &sName,
                                 const QVariant      &vValue,
                                 const QMetaObject   *pMetaParent,
                                 const QMetaProperty *pMetaProp )
{
    m_pXmlWriter->writeStartElement( sName );

    if ((pMetaProp != NULL) &&
        (pMetaProp->isEnumType() || pMetaProp->isFlagType()))
    {
        RenderEnum ( sName, vValue, pMetaProp );
    }
    else
        RenderValue( GetContentName( sName, pMetaParent, pMetaProp ), vValue );

    m_pXmlWriter->writeEndElement();
}

From pull request:

void XmlSerializer::AddProperty( const QString       &sName, 
                                 const QVariant      &vValue,
                                 const QMetaObject   *pMetaParent,
                                 const QMetaProperty *pMetaProp )
{
    m_pXmlWriter->writeStartElement( sName );

    if (pMetaProp == NULL)
    {
        // Do nothing
    }
    else if (pMetaProp->isEnumType() || pMetaProp->isFlagType())
    {
        RenderEnum ( sName, vValue, pMetaProp );
    }
    else
        RenderValue( GetContentName( sName, pMetaParent, pMetaProp ), vValue );

    m_pXmlWriter->writeEndElement();
}

I will try the pull request version, but I don't expect a difference. It also may be that Qt 5.7 from arch testing introduced another bug relate to the compiler.

[PlanetEater <zkr@…>]

My pull request is still pending, so the fix is only in my forked repo yet. And I submitted the PR against the master branch, so You have to apply the patch manually anyway, or wait until it finds its way into the fixes/0.28 branch of the official repo.
Here is the link to the patch from my repo, in raw format: ​https://github.com/PlanetEater/mythtv/commit/2dd09d2.patch

[PlanetEater <zkr@…>]

Replying to Jonathan Isom <jeisom@…>:

Here is the code in that repo and below it from the pull request. I got the crash from the above repo. Obviously the code looks different, but should not change how it functions. If there is a crash with one and not the other then it would be a compiler error.

No, You're wrong: in the original code, GetContentName() gets called when pMetaProp is NULL, in the modified version it doesn't.

[Jonathan Isom <jeisom@…>]

Ok. I see it now. Sorry about that. I just tested it. It fixes the issue for me using Qt 5.7 arch testing version. I wonder if this version below might be easier to understand what is going on. It functionally is the same this time.

Thanks

    if (pMetaProp != NULL) && (pMetaProp->isEnumType() || pMetaProp->isFlagType())
    {
        RenderEnum ( sName, vValue, pMetaProp );
    }
    else if (pMetaProp != NULL)
        RenderValue( GetContentName( sName, pMetaParent, pMetaProp ), vValue );

[PlanetEater <zkr@…>]

Yes, that would be functionally correct too. BUT we don't like redundancy, do we? :)
I redid the fix using nested if() statements, I hope the differences are now more self-explaining. Updated link to the patch: ​https://github.com/PlanetEater/mythtv/commit/36e180f.patch

[stuartm]

See also #12821 and possibly #12813

[ericrfontaine@…]

"Can you both include the version of QT you are using? There is a known QT bug in 5.6 which results in a similar/identical backtrace."

Sorry I haven't been monitoring this email. According to ​https://archive.archlinux.org/packages/q/qt5-base/ then than means 3 weeks ago I must have been using qt 5.6 (probably 5.6.0rc-2). I did give up and reinstall mythtv on another arch system, which worked for a time, and then eventually starting experiencing other segfaults (and these segfaults were on arch's latest qt 5.7). I came to the conclusion that mythtv doesn't work with the latest qt versions, so I eventually gave up with arch+mythtv and went to mythbuntu for stability, since my other family members need a stable myth system. Anyway, I want to suggest the devs test myth with latest qt libraries. Also, fyi there is an arch repository ​https://wiki.archlinux.org/index.php/Unofficial_user_repositories#qt-debug which has all the debugging symbols for the latest qt version, so you don't need to compile with debug symbols.

[jacksawild@…]

I've applied PlanetEater?'s patch using qt5.7 and my segfaults are no longer reproducible.

[Zoltán Karcagi <zkr@…>]

In c3a7929853026d98fb85bf76a79061d6b02f9b47/mythtv:

Error: Processor CommitTicketReference failed

GIT backend not available

[Bill Meek]

Some observations. For curl <be>:6544/Myth/GetHostName:

0.27 correctly gets: <?xml version="1.0" encoding="UTF-8"?><String>mc5-27</String>

0.28 gets: curl: (52) Empty reply from server + the expected segv.

master, with the fix gets: <?xml version="1.0" encoding="UTF-8"?><String/>

No hostname in the response, just the empty <String/>.

In servicehost.cpp:547 vValue=mc5-29 as it should be for my test. Just started trying to find where the hostname, in this case, is lost.

Run on a freshly installed Arch with Qt 5.7.0.

Adding -H Accept:application/json to the command works on all 3 versions.

[Bill Meek]

Replying to PlanetEater <zkr@…>:

• How do You trigger the crash?

Just type: curl <backend>:6544/Myth/GetHostName (without any patches and not while doing anything important, like recording.)

[Bill Meek]

Eric, PlanetEater?,

Please revert the patch and use this. Try to cause the failure with the command in the previous comment. The output should have <String>yourHostName</String> it it. With the existing patch, you'll see <String/>.

diff --git a/mythtv/libs/libmythupnp/serializers/xmlSerializer.cpp b/mythtv/libs/libmythupnp/serializers/xmlSerializer.cpp
index 9851e61..eeacd61 100644
--- a/mythtv/libs/libmythupnp/serializers/xmlSerializer.cpp
+++ b/mythtv/libs/libmythupnp/serializers/xmlSerializer.cpp
@@ -324,9 +324,12 @@ QString XmlSerializer::GetContentName( const QString        &sName,
 {
     // Try to read Name or TypeName from classinfo metadata.
 
-    int nClassIdx = pMetaObject->indexOfClassInfo( sName.toLatin1() );
+    int nClassIdx = -1;
 
-    if (nClassIdx >=0)
+    if ( pMetaObject )
+        nClassIdx = pMetaObject->indexOfClassInfo( sName.toLatin1() );
+
+    if (nClassIdx >=0 )
     {
         QString     sOptionData = pMetaObject->classInfo( nClassIdx ).value();
         QStringList sOptions    = sOptionData.split( ';' );

pMetaObject->indexOfClassInfo( sName.toLatin1() returns -1 "normally".

[PlanetEater <zkr@…>]

Applied the patch on top of the fixes/0.28 branch.

• The curl command correctly shows the backend hostname, no crash.
• Thumbnail generation in mythweb still triggers a crash, but now it's a different one:

Reading symbols from /usr/bin/mythbackend...Reading symbols from /usr/lib/debug/usr/bin/mythbackend.debug...done.
done.
[New LWP 31383]
[New LWP 31412]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `/usr/bin/mythbackend --logpath /var/log/mythtv'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0xb1db3d80 in malloc_consolidate () from /usr/lib/libc.so.6
[Current thread is 1 (Thread 0xacac4000 (LWP 31383))]
(gdb) bt
#0  0xb1db3d80 in malloc_consolidate () from /usr/lib/libc.so.6
#1  0xb1db3cc0 in malloc_consolidate () from /usr/lib/libc.so.6
#2  0xb1db4bd0 in _int_free () from /usr/lib/libc.so.6
#3  0xb21dee8c in ?? () from /usr/lib/libQt5Core.so.5
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) 

[Bill Meek]

Thanks for getting back. mythweb also uses the Services API, including:

<be>:6544/Content/GetPreviewImage?RecordedId=<n>

Please try that from a browser and expect an image. It works on my Arch test host.

Did the earlier patch prevent the SEGV reported in the comment above?

[Stuart Auchterlonie]

Bill,

You seem to be making good progress on this one, so reassigning to you.

Regards Stuart

[PlanetEater <zkr@…>]

Replying to bmeek:

Thanks for getting back. mythweb also uses the Services API, including:

<be>:6544/Content/GetPreviewImage?RecordedId=<n>

Please try that from a browser and expect an image. It works on my Arch test host.

Did the earlier patch prevent the SEGV reported in the comment above?

You Welcome. The earlier patch prevented a different crash, not the one above.

The good news is that since a couple of mythbackend restarts, I could not trigger a BE crash at all. The direct url test method you suggested always produces a picture in the browser, regardless of whether it existed before or had to be generated first. Thumbnail generation via mythweb seems to be working as well.

The bad news is that I have just found oom crash entries in the logs for other processes, including mariadb. These crashes started to appear one day before I installed the modified backend with your patch applied, so they probably more related to a system upgrade happened that day than to mythtv, but I'm still looking into this.

[PlanetEater <zkr@…>]

Mariadb and mythtv are the top memory consumers on my system, and after downgrading them to their previous versions I'm still getting those oom crashes. Consequently, I'm fairly convinced mythtv isn't the cause of them. I suggest reverting my patch and applying bmeek's instead.

[Bill Meek <bmeek@…>]

In 0dd75a68f2c60a7a8300a031880bd72c6d2d4bfa/mythtv:

Error: Processor CommitTicketReference failed

GIT backend not available

[Bill Meek <bmeek@…>]

In 38de43efa92603e2ac524b9a72951c15caf2af4e/mythtv:

Error: Processor CommitTicketReference failed

GIT backend not available

[PlanetEater <zkr@…>]

@bmeek: I cherry-picked your commit above on top of fixes/0.28, and the segfault is back. Looks like your original fix in comment31 did not make it into xmlSerializer.cpp. The actually committed changes for that file touch a different chunk of code.

[Bill Meek <bmeek@…>]

In eaf81bc3438e71f9623b59199b9ad1e7683db780/mythtv:

Error: Processor CommitTicketReference failed

GIT backend not available

[Bill Meek <bmeek@…>]

In 8eb43db5ba274c6105f61a5fe91f94ea62ac10fe/mythtv:

Error: Processor CommitTicketReference failed

GIT backend not available

[Bill Meek <bmeek@…>]

In 7c91ab6bc635c0299c029e6a0c148b290396b695/mythtv:

Error: Processor CommitTicketReference failed

GIT backend not available

[Stuart Auchterlonie]

The fixes for this issue appear to have been committed to fixes/0.28 Therefore closing.