Opened 10 years ago

Closed 10 years ago

Last modified 10 years ago

#7046 closed defect (invalid)

libmysql deadlock

Reported by: David Asher <asherml@…> Owned by: danielk
Priority: major Milestone: 0.22
Component: MythTV - General Version: head
Severity: medium Keywords:
Cc: Ticket locked: no

Description

Ok, so I'm not exactly sure which happened first here -- in fact I think there might be 2 bugs. I was watching a show on my remote frontend. When it was over I did my usual jump to the end to get the popup for whether to delete the show. I pressed the button to tell it to delete. At this point the frontend core dumped with an index out of range ASSERT. Now looking at the logs it looks like that was related to losing connection to the backend. Looking at the backend logs it looks like 3-4 minutes prior, the backend was starting recording of a new show. That recording never fully started up -- there was no recording file in the location the log says it was going to use. In addition the backend appeared to be non-responsive to both mythweb and an attempt to restart mythfrontend.

I have logs from both the frontend and backend, and gdb backtraces of the frontend core dump, the restarted frontend (which never fully started and had to be killed), and the non-responsive backend.

Let me know if there's anything else I can do to help.

Attachments (5)

gdb.txt.crash (17.6 KB) - added by David Asher <asherml@…> 10 years ago.
backtrace of frontend core dump (ASSERT)
gdb.txt.be.gz (9.3 KB) - added by David Asher <asherml@…> 10 years ago.
backtrace of non-responsive backend
gdb.txt.newfe (16.3 KB) - added by David Asher <asherml@…> 10 years ago.
backtrace of restarted frontend which never fully restarted
mythfrontend.log.crash (4.9 KB) - added by David Asher <asherml@…> 10 years ago.
log from frontend crash (ASSERT)
mythbackend.log.crash.gz (12.1 KB) - added by David Asher <asherml@…> 10 years ago.
log from non-responsive backend

Download all attachments as: .zip

Change History (17)

Changed 10 years ago by David Asher <asherml@…>

Attachment: gdb.txt.crash added

backtrace of frontend core dump (ASSERT)

Changed 10 years ago by David Asher <asherml@…>

Attachment: gdb.txt.be.gz added

backtrace of non-responsive backend

Changed 10 years ago by David Asher <asherml@…>

Attachment: gdb.txt.newfe added

backtrace of restarted frontend which never fully restarted

Changed 10 years ago by David Asher <asherml@…>

Attachment: mythfrontend.log.crash added

log from frontend crash (ASSERT)

Changed 10 years ago by David Asher <asherml@…>

Attachment: mythbackend.log.crash.gz added

log from non-responsive backend

comment:1 Changed 10 years ago by David Asher <asherml@…>

BTW, the frontend log is "important, general, playback" and the backend log is "all, nodatabase, noupnp"

comment:2 Changed 10 years ago by David Asher <asherml@…>

I forgot to mention I'm running trunk r21784.

comment:3 Changed 10 years ago by danielk

(In [21939]) Refs #7046. Fixes segfault portion of this bug report.

comment:4 Changed 10 years ago by danielk

Milestone: unknown0.22
Owner: changed from Isaac Richards to danielk
Priority: minormajor
Status: newassigned
Version: unknownhead

Note: the hang is possibly a dupe of #7046.

comment:5 Changed 10 years ago by danielk

Oops: The hang is possibly a dupe of #7078, not itself :)

comment:6 Changed 10 years ago by danielk

David A, janneg started looking at this problem. At first blush it appears we died within the mysql library while we were in MSqlDatabase::prepareQuery() which is locked by a static lock. Once we die with that lock in place 99% of mythtv db queries can no longer complete which pretty much stops the application dead in it's tracks.

comment:7 Changed 10 years ago by danielk

Summary: Mythfrontend crash and Mythbackend becomes non-responsivelibmysql deadlock

Changing summary.

While it looks like the underlying problem is with libmysql we haven't eliminated the possibility that we're somehow triggering it or could deal with it in a better way. Hence I'm leaving the ticket open for now.

comment:8 Changed 10 years ago by danielk

David A, what distro are you using, and what package revision of libmysql?

comment:9 Changed 10 years ago by danielk

Status: assignedinfoneeded

comment:10 Changed 10 years ago by David Asher <asherml@…>

Distro is Mythbuntu 9.04 (Jaunty)

There appears to be a libmysqlclient16 as well, but I'm using libmysqlclient15off:

$ aptitude show libmysqlclient15off Package: libmysqlclient15off State: installed Automatically installed: no Version: 5.1.30really5.0.75-0ubuntu10.2 Priority: optional Section: libs Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss@…> Uncompressed Size: 4317k Depends: mysql-common (>= 5.1.30really5.0.75-0ubuntu10.2), libc6 (>= 2.4), zlib1g (>= 1:1.1.4) Conflicts: libmysqlclient15, mysql-doc-5.0 (< 5.0.56-0ubuntu1) Replaces: libmysqlclient15, mysql-doc-5.0 (< 5.0.56-0ubuntu1) Description: MySQL database client library

MySQL is a fast, stable and true multi-user, multi-threaded SQL database server. SQL (Structured Query Language) is the most popular database query language in the world. The main goals of MySQL are speed, robustness and ease of use.

This package includes the client library.

Homepage: http://dev.mysql.com/

comment:11 Changed 10 years ago by danielk

Resolution: invalid
Status: infoneededclosed

From all appearances this is at it's root a libmysql problem so I'm marking it invalid as a MythTV ticket. But the libmysql deadlock should be reported to Canonical.

comment:12 Changed 10 years ago by cpinkham

(In [22561]) Make MSqlQuery's prepareLock non-static as discussed on IRC.

This lock was originally put in a long time ago to hande the fact that QSqlQuery::prepare() was not thread safe in Qt versions prior to v3.3.2. The Qt v4 version is thread safe. We have seen issues with deadlocks in mythbackend due to this lock, so we are removing the static so that different threads do not lock each other trying to prepare a query. If there are no issues seen after the lock is made non-static, we'll commit a larger patch which removes the prepareLock totally.

The issue causing the deadlock currently is with the recorded table and in IRC, we've been able to reproduce this with current trunk. Here is the description of the one situation we've been able to reproduce with 2 concurrent recordings.

Thread #1 calls programinfo.cpp's insert_program. insert_program gets a MySQL table write lock on the recorded table. insert_program then performs a select on the recorded table.

While Thread #1 is performing the select, Thread #2 issues a MSqlQuery::prepare() call in order to update the recorded table. Thread #2 gets our MSqlQuery::prepareLock. MSqlQuery::prepare() in turn calls QSqlQuery::prepare which calls the MySQL prepare function. MySQL sees that the recorded table is locked for write and locks Thread #2 waiting for the write lock to be released.

Meanwhile, Thread #1 finishes its select on the recorded table and goes to perform an insert to insert the information for the new recording. It tries to MSqlQuery::parepare() the query, but locks trying to get the prepareLock lock. Now we have a deadlock. Thread #1 is waiting on Thead #2 to unlock prepareLock and Thread #2 is waiting on Thread #1 to unlock the recorded table.

This does update the binary api version so make clean. It might not have been totally necessary, but you had to recompile anyway because of my previous [22560] commit.

Refs #7046

Note: See TracTickets for help on using tickets.