Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#13150 closed Bug Report - Crash (fixed)

Backend spawns threads until it crashes when user jobs active

Reported by: mythwiki@… Owned by: Jonatan Lindblad
Priority: minor Milestone: 30.0
Component: MythTV - General Version: 0.28.1
Severity: medium Keywords:
Cc: Ticket locked: no

Description

I'm using LinHES, and the past two upgrades have had mythbackends that crash. There seems to be a system that spawns a new thread roughly every two seconds. Screen output when running in gdb is:

[New Thread 0x7ffd6bd37700 (LWP 30169)]
2017-10-12 11:54:23.158482 I  MainServer: MainServer::ANN Monitor
2017-10-12 11:54:23.158490 I  MainServer: adding: mythtv.i.cneufeld.ca(ca1920) as a client (events: 0)
2017-10-12 11:54:23.728318 I  Monitor sock(ca1920) 'mythtv.i.cneufeld.ca' disconnected
[New Thread 0x7ffd6b536700 (LWP 30225)]
2017-10-12 11:54:24.941168 I  MainServer: MainServer::ANN Monitor
2017-10-12 11:54:24.941177 I  MainServer: adding: mythtv.i.cneufeld.ca(ca19f0) as a client (events: 0)
2017-10-12 11:54:25.517690 I  Monitor sock(ca19f0) 'mythtv.i.cneufeld.ca' disconnected

Eventually the system runs out of file descriptors for pipes and the backend crashes. In normal operation, there seems to be some housekeeping routine that is reaping the threads. However, when there is a long-running user job (mine can run for hours), the threads pile up and the backend crashes. It then is restarted by the system, and so resubmits the long job with the same jobqueue ID.

My current version is v0.28.1-44-gebd69ec There is a backtrace with some details at https://pastebin.com/mpN9eNcA

I've built a backend with debugging symbols, and can test patches.

Change History (8)

comment:1 Changed 3 years ago by Jonatan Lindblad

Can you try reverting 019044db324fc630581def44e0d02a570dd184ed and see if that helps?

comment:2 in reply to:  1 Changed 3 years ago by mythwiki@…

Replying to natanojl:

Can you try reverting 019044db324fc630581def44e0d02a570dd184ed and see if that helps?

Reverting that change prevents the runaway thread spawning. Threads are consistently reaped, and the thread count stays around 30-40. I'll be applying this reversion change to the binary running on my machine.

Thank you.

comment:3 Changed 3 years ago by mythwiki@…

To follow up: I've been running in production for four days. I've gone from about 30 crashes a day in regular use (not running my scripts), to none.

This bug never damaged any recordings, I suspect that the reaping routine runs regularly when a recording is going on.

Thank you again.

comment:4 Changed 3 years ago by Jonatan Lindblad

Owner: set to Jonatan Lindblad
Status: newassigned

comment:5 Changed 3 years ago by Jonatan Lindblad <jlindblad@…>

Resolution: fixed
Status: assignedclosed

In b1e5ddeb92ae93b6eb692aa7863d80be89aaf1e9/mythtv:

MainServer?: Reinstate the event about disconnected sockets

Fixes #13150

comment:6 Changed 3 years ago by Stuart Auchterlonie

Milestone: needs_triage30.0
Version: Unspecified0.28.1

comment:7 Changed 3 years ago by Jonatan Lindblad <jlindblad@…>

In 31fdace083ab295d3c1bf4ed3543f6d1dfa050a6/mythtv:

MainServer?: Reinstate the event about disconnected sockets

Fixes #13150

(cherry picked from commit b1e5ddeb92ae93b6eb692aa7863d80be89aaf1e9)

comment:8 Changed 3 years ago by Jonatan Lindblad <jlindblad@…>

In 08197560e23fbe4177a822d2fa25e9040de3f8b7/mythtv:

MainServer?: Reinstate the event about disconnected sockets

Fixes #13150

(cherry picked from commit b1e5ddeb92ae93b6eb692aa7863d80be89aaf1e9)

Note: See TracTickets for help on using tickets.