socket errors cause back-end to become unresponsive (trunk 21894)

[databubble]

I've been experiencing various socket error after the back-end has been running for a few hours, which once they occur makes the back-end unresponsive. Occurs in listing recordings, listing scheduled/upcoming recording screens, etc. Have been having problems for about a week.... only just managed to capture with logging, etc.

MythTV Version : 21894M MythTV Branch : trunk Network Protocol : 48 Library API : 0.22.20090916-2 QT Version : 4.5.2 Options compiled in:

linux release using_alsa using_pulse using_backend using_directfb using_dvb using_firewire using_frontend using_hdpvr using_iptv using_lirc using_mheg using_opengl_video using_opengl_vsync using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_xvmc using_xvmc_vld using_bindings_perl using_bindings_python using_opengl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live using_mheg

[databubble]

frontend log with -v socket,network,extra

[databubble]

backtrace

[databubble]

first 1000 lines of mythbackend.log

[databubble]

last 5000 lines of mythbackend.log

[databubble]

output from configure

[danielk]

databubble, the backtrace is missing symbols. Can you try to recreate this with a mythbackend compiled with --compile-type=profile ? Since this is difficult to reproduce I'll try my best to diagnose without that info.

[anonymous]

I have re-compiled with --compile-type=debug still with r21894 and the back-end is running now. Some quick poking, and it hasn't triggered the problem yet.... and that log keeps getting bigger and bigger. As soon as it does fall over (usually a few hours) I'll do my best to extract better quality information for you.

[anonymous]

Replying to anonymous:

I have re-compiled with --compile-type=debug still with r21894 and the back-end is running now. Some quick poking, and it hasn't triggered the problem yet.... and that log keeps getting bigger and bigger. As soon as it does fall over (usually a few hours) I'll do my best to extract better quality information for you.

I can confirm the same issue with:

MythTV Version : 21583-openglvdpau2-nv180 MythTV Branch : branches/release-0-21-fixes Network Protocol : 40 Library API : 0.21.20080304-1 Options compiled in:

linux profile using_oss using_alsa using_pulse using_jack using_backend using_directfb using_dvb using_firewire using_frontend using_glx_proc_addr_arb using_hdhomerun using_iptv using_ivtv using_joystick_menu using_libfftw3 using_lirc using_opengl_video using_opengl_vsync using_v4l using_x11 using_xrandr using_xv using_xvmc using_xvmc_vld using_xvmcw using_bindings_perl using_bindings_python using_opengl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live

I have yet to find a way to trigger it, seems random but destroy WAF.

[anonymous]

I can confirm the same issue with:

MythTV Version : 21583-openglvdpau2-nv180 MythTV Branch : branches/release-0-21-fixes

r21583 is much too old to be much help. Much socket stuff has been improved around r21726, possibly even after that. Of course it was only improved on the current trunk (soon to be 0.22). 0.21-fixes contains none of the current fixes, and likely won't.

[databubble]

I tried creating another backtrace, but I´m not sure it's any better than the first. Although, I did recompile with debug after a make distclean:

Please include all output in bug reports. MythTV Version : 21894M MythTV Branch : trunk Network Protocol : 48 Library API : 0.22.20090916-2 QT Version : 4.5.2 Options compiled in:

linux debug using_alsa using_pulse using_backend using_directfb using_dvb using_firewire using_frontend using_hdpvr using_iptv using_lirc using_mheg using_opengl_video using_opengl_vsync using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_xvmc using_xvmc_vld using_bindings_perl using_bindings_python using_opengl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live using_mheg

One thing I can say is that the back-end got itself into it's unresponsive state without the front-end running at all. If I check back, it looks like the backend had a problem around 7:30p, or after it had been running for 8 hours, and made 8 successful recordings. If failed on trying to record "Hannah Montana" (who can blame it?) leaving it empty in the recordings list. Thereafter, nothing more appears in the recordings list and it was unresponsive to the front-end when I finally connected to it 6 hours later.

BTW, I get the following 4 messages repeated a LOT in the log... this seems to be normal (HD-PVR). I've stripped them out.

2009-09-17 19:33:26.859 [h264 @ 0x7f3bed4b07c0]decode_slice_header error 2009-09-17 19:33:26.861 [h264 @ 0x7f3bed4b07c0]no frame! 2009-09-17 19:33:26.862 AFD Error: Unknown decoding error 2009-09-17 19:33:26.864 [h264 @ 0x7f3bed4b07c0]B picture before any references, skipping

[databubble]

Attempt at backtrace after compiling with -debug

[databubble]

Backend log file from time around second freeze

[David Asher <asherml@…>]

This backend hang sounds a lot like the one I reported in #7046. There are backtraces and logs there.

[davideshay@…]

mythbackend.log from similar issues -- associated debug output next

[davideshay@…]

gdb output from same event, I think...

[davideshay@…]

I'm seeing very similar issues to those reported as well. I've attempted to put put backtraces out here, but I haven't really done that before, so I hope they are correct.

mythbackend version: Please include all output in bug reports. MythTV Version : 21949 MythTV Branch : trunk Network Protocol : 48 Library API : 0.22.20090918-1 QT Version : 4.5.1 Options compiled in:

linux debug using_oss using_alsa using_arts using_jack using_backend using_dvb using_firewire using_frontend using_hdho

merun using_hdpvr using_iptv using_ivtv using_joystick_menu using_lirc using_mheg using_opengl_video using_opengl_vsync using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_xvmc using_bindings_perl using_bindings_python using_open gl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live using_mheg

I can provide frontend info as well. I don't have gdb data for that but could run additional tests if necessary. Let me know.

[nickj-fox@…]

Similar errors here as well:

MythTV Version : 21954 MythTV Branch : trunk Network Protocol : 48 Library API : 0.22.20090918-1 QT Version : 4.5.2 Options compiled in:

linux profile using_oss using_alsa using_pulse using_jack using_backend using_dvb using_firewire using_frontend using_glx_proc_addr_arb using_hdhomerun using_hdpvr using_iptv using_ivtv using_joystick_menu using_libfftw3 using_lirc using_mheg using_opengl_video using_opengl_vsync using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_xvmc using_xvmc_vld using_xvmcw using_bindings_perl using_bindings_python using_opengl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live using_mheg

[danielk]

databubble, your backtraces are still missing debugging info.

davideshay, I do not see any evidence of a deadlock in that backtrace. All threads appear to be in valid wait states.

nickj-fox, please do not report "me too" -- it is not helpful. A backtrace with debugging symbols that shows a deadlock would be helpful. (I don't expect you to know if it displays a deadlock or not, just follow the TicketHowTo link on how to generate a full backtrace.)

[davideshay@…]

danielk, perhaps posted under wrong ticket. Symptom I am getting timeouts on the frontend like this:

2009-09-21 07:03:40.446 MythSocket?(25aae80:55): readStringList: Error, timed out after 30000 ms. 2009-09-21 07:03:40.447 Preview Error: Remote Preview failed due to communications error.

Not always remote preview. Right now I always get the error when entering watch recordings, and nearly always get a pause when exiting a show.

At some point, the front end gets confused and thinks shows are missing in Watch Recordings and I have to at least exit that screen and go back in.

Thought based on the email in mythtv-users to post on this ticket, maybe not. I see that since my backend wasn't completely unresponsive that a backtrace did no good. Do you want -v socket logs on front and back simultaneously or what else would help to track this down?

[databubble]

Sorry, but I need some help with getting the backtrace. I did a "make distclean" and then ran configure with --compile-type=profile (also tried --compile-type=debug), and installed with "checkconfig" and indeed profile is listed as a compile option...

MythTV Version   : 21992M
MythTV Branch    : trunk
Network Protocol : 48
Library API      : 0.22.20090919-1
QT Version       : 4.5.2
Options compiled in:
 linux profile using_alsa using_pulse using_backend using_directfb using_dvb using_firewire using_frontend using_hdpvr using_iptv using_lirc using_mheg using_opengl_video using_opengl_vsync using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_xvmc using_xvmc_vld using_bindings_perl using_bindings_python using_opengl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live using_mheg

But whenever I try to connect to the process (as root), I get:

Reading symbols from /usr/bin/mythbackend...(no debugging symbols found)...done.                                                                              
Attaching to program: /usr/bin/mythbackend, process 28932                      
Reading symbols from /usr/lib/libmythtv-0.22.so.0...(no debugging symbols found)...done.                                                                      
Loaded symbols for /usr/lib/libmythtv-0.22.so.0                                
Reading symbols from /usr/lib/libmythavformat-0.22.so.0...(no debugging symbols found)...done.                                                                
Loaded symbols for /usr/lib/libmythavformat-0.22.so.0                          
Reading symbols from /usr/lib/libmythavutil-0.22.so.0...(no debugging symbols found)...done.                                                                  
Loaded symbols for /usr/lib/libmythavutil-0.22.so.0                            
Reading symbols from /usr/lib/libmythavcodec-0.22.so.0...(no debugging symbols found)...done.                                                                 
Loaded symbols for /usr/lib/libmythavcodec-0.22.so.0                           
Reading symbols from /usr/lib/libmythswscale-0.22.so.0...(no debugging symbols found)...done.                                                                 
Loaded symbols for /usr/lib/libmythswscale-0.22.so.0                           
Reading symbols from /usr/lib/libmythupnp-0.22.so.0...(no debugging symbols found)...done.                                                                    
Loaded symbols for /usr/lib/libmythupnp-0.22.so.0                              
Reading symbols from /usr/lib/libmyth-0.22.so.0...(no debugging symbols found)...done.                                                                        
Loaded symbols for /usr/lib/libmyth-0.22.so.0                                  
Reading symbols from /usr/lib/libmythui-0.22.so.0...(no debugging symbols found)...done.                                                                      
Loaded symbols for /usr/lib/libmythui-0.22.so.0                                
Reading symbols from /usr/lib/libmythdb-0.22.so.0...(no debugging symbols found)...done.                                                                      
Loaded symbols for /usr/lib/libmythdb-0.22.so.0                                
Reading symbols from /usr/lib/libmythlivemedia-0.22.so.0...(no debugging symbols found)...done.    

Shouldn't the symbols be there? What am I doing wrong?

Sorry, I have searched and tried to sort this out myself. I just want to extract the backtrace so this ticket can move forward.

Phil

[databubble]

It appears that the patch in #7132 may resolve the issue I was seeing with the back-end stopping recording and becoming unresponsive. I've been running 14 hours and made about 18 recordings with an HD-PVR, and so far so good. I wasn't able to run that long previously. I'll let it go for another day, and if it's still good then we can close this ticket as a duplicate of 7132.

[danielk]

Dup #7132.