[HANG] backend lockups

[brian@…]

I have been experiencing lockups in the backend. When this happens, the frontends are no longer able to get a list of recordings from the backend (it spins for about 20-30 seconds and then shows an empty list) and mythweb reports errors trying to open any page.

The only way I have found to resolve this is to restart the backend.

During the lockup last night I was able to grab a threaded stack trace:

(moved to an attachment called "stack_trace_1.txt")

[brian@…]

Hit another lockup this evening. 7 programs that were supposed to record (including the wife's favorite: Survivor) failed to record because of this. She is not going to be at all happy. This MythTV thing is going to get kicked to the curb pretty quickly after many years of service due to a complete lack of reliability that has come with this 0.24 release. :-(

Anyway, here's this evening's stack trace:

(moved to stack_trace_2.txt)

I'd be much appreciative if we get an analysis on this. I am happy to help where I can.

[kenni]

Please don't post 2500 lines of stacktraces inside a ticket, it's impossible to get an overview of the comments etc. of the ticket. And don't change the priority/severity of the ticket, as described in the TicketHowTo.

[beirdo]

Please attach a copy of the frontend and backend logs at the point where this dies so we have a better idea of how to debug this.

[brian@…]

Funny you should ask since this just crapped out on me, yet again tonight. I will attach logs.

[brian@…]

stack trace: incident #2

[brian@…]

frontend log: incident #2

[brian@…]

backend log: incident #2

[brian@…]

It's probably worth mentioning that this problem seems to only occur while a particular front-end in my network is watching a recording. The other/main front-end does not seem to be able to tickle this problem.

The troublesome frontend is running branches/release-0-24-fixes [27334] (i.e. per ​http://code.mythtv.org/trac/attachment/ticket/9274/mythfrontend.log) and the front-end that does not seem to tickle this issue is running branches/release-0-24-fixes [27321].

Given how close those revs are though, I don't hold much hope that that's simply the problem.

[beirdo]

The line:

2010-12-19 22:51:27.731 DevRdB(/dev/dvb/adapter0/frontend0) Error: Driver buffers overflowed

in your backend log sure doesn't look promising.

[Alex Tomlins <alex@…>]

I'm seeing a similar problem. I've added a gdb trace I obtained when it occurred.

[robertm]

Closing as this particular hang can no longer occur (it has been replaced by all new hangs).

[brian@…]

Is this fixed in 0.24-fixes or only in trunk? If the latter, that's great, except for those of us struggling with this issue in a production environment (and suffering plummeting WAF every time it happens).

Cheers, b.