Opened 4 years ago

Closed 3 years ago

#12501 closed Bug Report - Hang/Deadlock (Upstream Bug)

MythBackend Locks Up

Reported by: compnerd@… Owned by:
Priority: minor Milestone:
Component: MythTV - General Version: Master Head
Severity: medium Keywords: mythbackend deadlock 0.28
Cc: Ticket locked: no

Description

Mythbackend has been locking up recently. I didn't have verbose logging going on the last couple of times, so I will attach that at a later time (once it happens again). I don't have steps to reproduce, it just seems to happen at random. Sometimes it takes 5 minutes between killing the previous process to over a day.

It completely stops responding to the API, frontends, mythweb, etc. The process is not using 100% which is why I think this is just a deadlock and not a resource problem.

I was able to get two backtraces during the lock up (attach method). Those are attached. This has occurred for about a month, give or take, so I think it is around the last 2 or 3 releases from this PPA: https://launchpad.net/~mythbuntu/+archive/ubuntu/0.28

Attachments (5)

mythbackend_version.txt (1.1 KB) - added by compnerd@… 4 years ago.
MythBackend? Version
gdb.txt (1.5 MB) - added by compnerd@… 4 years ago.
GDB Backtrace
gdb-2.txt (1.5 MB) - added by compnerd@… 4 years ago.
GDB Backtrace
gdb-3.txt (1.5 MB) - added by compnerd@… 4 years ago.
GDB Backtrace with Log
mythbackend.log.gz (1.7 MB) - added by compnerd@… 4 years ago.
Corresponding mythbackend logs

Change History (17)

Changed 4 years ago by compnerd@…

Attachment: mythbackend_version.txt added

MythBackend? Version

Changed 4 years ago by compnerd@…

Attachment: gdb.txt added

GDB Backtrace

Changed 4 years ago by compnerd@…

Attachment: gdb-2.txt added

GDB Backtrace

comment:1 Changed 4 years ago by compnerd@…

Happend again. This time with verbose mythbackend logs. Once I finished gdb debugging and let it sit for a minute, XBMC started playing again without me restarting mythtv.

Changed 4 years ago by compnerd@…

Attachment: gdb-3.txt added

GDB Backtrace with Log

Changed 4 years ago by compnerd@…

Attachment: mythbackend.log.gz added

Corresponding mythbackend logs

comment:2 Changed 4 years ago by paulh

Can you reproduce this without using third party scripts and clients?

comment:3 Changed 4 years ago by paulh

Priority: blockerminor

comment:4 Changed 4 years ago by compnerd@…

That's my entire environment. I don't have native MythTV frontends in use. I will shut down all my frontends today and see if it can be reproduced by the time I return this evening, but that's about all I can do.

comment:5 Changed 4 years ago by jlbarriere68@…

Hi, reading your log file I guess you use pvr.mythtv client. It seems you have one client with the addon enabled which is not uptodate with the protocol version and it retry to connect every couple of second. Certainly not related with the issue but it could help to stop filling your log file. You should upgrade all clients with the latest version to support the protocol 87. And then you will have only ONE notification "... speaks protocol version 75 but we speak 87!" logged at the client startup. br

comment:6 Changed 4 years ago by compnerd@…

They're all up to date (or somewhere in the past two or three minor versions if not). I'll get the actual version numbers when I can later this evening.

However, I can say that is how pvr.mythtv connects to MythTV. It starts with the lowest version it supports (currently 75) and works its way up from there (max is 87). I may submit a bug report/patch to janbar to get that changed to be in reverse. That might work better in the long-run, however the API also parses the version response from MythTV and uses the appropriate commands for that API (if there is anything special that is needed).

Version table: https://github.com/janbar/pvr.mythtv/blob/8e5ba2cdf3d8f5ad5a622c2f0be82ae4d2536f1e/lib/cppmyth/src/proto/mythprotobase.cpp#L40

Loop to connect to MythTV: https://github.com/janbar/pvr.mythtv/blob/8e5ba2cdf3d8f5ad5a622c2f0be82ae4d2536f1e/lib/cppmyth/src/proto/mythprotobase.cpp#L299

comment:7 Changed 4 years ago by compnerd@…

To update as promised:

  1. Currently I'm running pvr.mythtv 2.6.2 across all Kodi systems (4 of them). I will be upgrading to 2.7.2 today.
  1. No lockups since my first post this morning, roughly 11 hours ago

comment:8 Changed 4 years ago by compnerd@…

Also, diving into the pvr.mythtv logs, it negotiates up to version 87 (or whatever max the backend is):

17:14:41 T:2492   DEBUG: AddOnLog: MythTV PVR Client: (CPPMyth)__connectAddr: connected to socket(0B8BADA0)
17:14:41 T:2492   DEBUG: AddOnLog: MythTV PVR Client: (CPPMyth)Myth::ProtoBase::SendCommand: MYTH_PROTO_VERSION 75 SweetRock
17:14:41 T:10332   DEBUG: CRecentlyAddedJob::UpdateMusic() - Running RecentlyAdded home screen update
17:14:41 T:2492   DEBUG: AddOnLog: MythTV PVR Client: (CPPMyth)Myth::ProtoBase::RcvMessageLength: 13
17:14:41 T:2492   DEBUG: AddOnLog: MythTV PVR Client: (CPPMyth)Myth::ProtoBase::OpenConnection: asked for version 75, got version 87
17:14:41 T:2492   DEBUG: AddOnLog: MythTV PVR Client: (CPPMyth)__connectAddr: connected to socket(0B8BADA0)
17:14:41 T:2492   DEBUG: AddOnLog: MythTV PVR Client: (CPPMyth)Myth::ProtoBase::SendCommand: MYTH_PROTO_VERSION 87 (ノಠ益ಠ)ノ彡┻━┻_No_entiendo!)
17:14:41 T:2492   DEBUG: AddOnLog: MythTV PVR Client: (CPPMyth)Myth::ProtoBase::RcvMessageLength: 13
17:14:41 T:2492   DEBUG: AddOnLog: MythTV PVR Client: (CPPMyth)Myth::ProtoBase::OpenConnection: asked for version 87, got version 87
17:14:41 T:2492   DEBUG: AddOnLog: MythTV PVR Client: (CPPMyth)Myth::ProtoBase::OpenConnection: agreed on Version 87 protocol

So those Myth warnings should be able to be ignored.

comment:9 in reply to:  6 Changed 4 years ago by George Nassas <gnassas@…>

Replying to compnerd@…:

However, I can say that is how pvr.mythtv connects to MythTV. It starts with the lowest version it supports (currently 75) and works its way up from there (max is 87). I may submit a bug report/patch to janbar to get that changed to be in reverse.

Instead of that approach take a look at https://www.mythtv.org/wiki/MYTH_PROTO_VERSION_(Myth_Protocol). You send the backend a version you support and it either accepts it or sends a rejection with the version it supports. If you understand that protocol version then announce with it otherwise your client is SOL and shuts down.

Hammering the way it's doing now is unnecessary.

comment:10 Changed 4 years ago by compnerd@…

It isn't my code. That being said, that is what it does. It tries the first version (75) and gets a rejection. Then it checks if that version (in my case 87) which succeeds just fine.

That is what the PVR logs are showing above.

So far tonight we've had no lockups. There was also a minor version update to MythTV today on the repos I applied.

comment:11 Changed 4 years ago by compnerd@…

I'm sorry to leave this hanging for so many days. I haven't had that much time to mess with the system.

Since the upgrade of the frontends and one of the dailies from 0.28, I haven't had this problem. It has been 10 days without an incident.

This can be closed.

comment:12 Changed 3 years ago by paulh

Resolution: Upstream Bug
Status: newclosed
Note: See TracTickets for help on using tickets.