Opened 10 years ago

Closed 9 years ago

Last modified 9 years ago

#9274 closed Bug Report (Invalid)

[HANG] backend lockups

Reported by: brian@… Owned by:
Priority: minor Milestone: unknown
Component: MythTV - General Version: 0.24-fixes
Severity: medium Keywords:
Cc: Ticket locked: no

Description (last modified by Kenni Lund [kenni a kelu dot dk])

I have been experiencing lockups in the backend. When this happens, the frontends are no longer able to get a list of recordings from the backend (it spins for about 20-30 seconds and then shows an empty list) and mythweb reports errors trying to open any page.

The only way I have found to resolve this is to restart the backend.

During the lockup last night I was able to grab a threaded stack trace:

(moved to an attachment called "stack_trace_1.txt")

Attachments (6)

stack_trace_1.txt (77.0 KB) - added by Kenni Lund [kenni a kelu dot dk] 10 years ago.
stack_trace_2.txt (132.0 KB) - added by Kenni Lund [kenni a kelu dot dk] 10 years ago.
mythbackend-ThreadStacktrace.txt (94.3 KB) - added by brian@… 10 years ago.
stack trace: incident #2
mythfrontend.log (44.3 KB) - added by brian@… 10 years ago.
frontend log: incident #2
mythbackend.log (22.4 KB) - added by brian@… 10 years ago.
backend log: incident #2
gdb.txt (86.0 KB) - added by Alex Tomlins <alex@…> 9 years ago.

Download all attachments as: .zip

Change History (17)

comment:1 Changed 10 years ago by brian@…

Hit another lockup this evening. 7 programs that were supposed to record (including the wife's favorite: Survivor) failed to record because of this. She is not going to be at all happy. This MythTV thing is going to get kicked to the curb pretty quickly after many years of service due to a complete lack of reliability that has come with this 0.24 release. :-(

Anyway, here's this evening's stack trace:

(moved to stack_trace_2.txt)

I'd be much appreciative if we get an analysis on this. I am happy to help where I can.

Last edited 10 years ago by Kenni Lund [kenni a kelu dot dk] (previous) (diff)

Changed 10 years ago by Kenni Lund [kenni a kelu dot dk]

Attachment: stack_trace_1.txt added

Changed 10 years ago by Kenni Lund [kenni a kelu dot dk]

Attachment: stack_trace_2.txt added

comment:2 Changed 10 years ago by Kenni Lund [kenni a kelu dot dk]

Description: modified (diff)
Priority: blockerminor
Severity: highmedium

Please don't post 2500 lines of stacktraces inside a ticket, it's impossible to get an overview of the comments etc. of the ticket. And don't change the priority/severity of the ticket, as described in the TicketHowTo.

comment:3 Changed 10 years ago by beirdo

Please attach a copy of the frontend and backend logs at the point where this dies so we have a better idea of how to debug this.

comment:4 Changed 10 years ago by beirdo

Status: newinfoneeded_new

comment:5 Changed 10 years ago by brian@…

Funny you should ask since this just crapped out on me, yet again tonight. I will attach logs.

Changed 10 years ago by brian@…

stack trace: incident #2

Changed 10 years ago by brian@…

Attachment: mythfrontend.log added

frontend log: incident #2

Changed 10 years ago by brian@…

Attachment: mythbackend.log added

backend log: incident #2

comment:6 Changed 10 years ago by brian@…

It's probably worth mentioning that this problem seems to only occur while a particular front-end in my network is watching a recording. The other/main front-end does not seem to be able to tickle this problem.

The troublesome frontend is running branches/release-0-24-fixes [27334] (i.e. per http://code.mythtv.org/trac/attachment/ticket/9274/mythfrontend.log) and the front-end that does not seem to tickle this issue is running branches/release-0-24-fixes [27321].

Given how close those revs are though, I don't hold much hope that that's simply the problem.

comment:7 Changed 9 years ago by beirdo

The line:

2010-12-19 22:51:27.731 DevRdB(/dev/dvb/adapter0/frontend0) Error: Driver buffers overflowed

in your backend log sure doesn't look promising.

Changed 9 years ago by Alex Tomlins <alex@…>

Attachment: gdb.txt added

comment:8 Changed 9 years ago by Alex Tomlins <alex@…>

I'm seeing a similar problem. I've added a gdb trace I obtained when it occurred.

comment:9 Changed 9 years ago by beirdo

Summary: backend lockups[HANG] backend lockups

comment:10 Changed 9 years ago by robertm

Resolution: Invalid
Status: infoneeded_newclosed

Closing as this particular hang can no longer occur (it has been replaced by all new hangs).

comment:11 Changed 9 years ago by brian@…

Is this fixed in 0.24-fixes or only in trunk? If the latter, that's great, except for those of us struggling with this issue in a production environment (and suffering plummeting WAF every time it happens).

Cheers, b.

Note: See TracTickets for help on using tickets.