Opened 10 years ago

Closed 10 years ago

#7078 closed defect (duplicate)

socket errors cause back-end to become unresponsive (trunk 21894)

Reported by: databubble Owned by: danielk
Priority: major Milestone: 0.22
Component: MythTV - General Version: head
Severity: medium Keywords: socket erros
Cc: Ticket locked: no

Description

I've been experiencing various socket error after the back-end has been running for a few hours, which once they occur makes the back-end unresponsive. Occurs in listing recordings, listing scheduled/upcoming recording screens, etc. Have been having problems for about a week.... only just managed to capture with logging, etc.

MythTV Version : 21894M MythTV Branch : trunk Network Protocol : 48 Library API : 0.22.20090916-2 QT Version : 4.5.2 Options compiled in:

linux release using_alsa using_pulse using_backend using_directfb using_dvb using_firewire using_frontend using_hdpvr using_iptv using_lirc using_mheg using_opengl_video using_opengl_vsync using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_xvmc using_xvmc_vld using_bindings_perl using_bindings_python using_opengl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live using_mheg

Attachments (9)

mythfrontend.log (8.8 KB) - added by databubble 10 years ago.
frontend log with -v socket,network,extra
backtrace.txt (53.1 KB) - added by databubble 10 years ago.
backtrace
mythbackend1-log.zip (47.2 KB) - added by databubble 10 years ago.
first 1000 lines of mythbackend.log
mythbackend2-log.zip (192.5 KB) - added by databubble 10 years ago.
last 5000 lines of mythbackend.log
configure.out (2.7 KB) - added by databubble 10 years ago.
output from configure
bt_try2.txt (46.0 KB) - added by databubble 10 years ago.
Attempt at backtrace after compiling with -debug
backend2.log (70.8 KB) - added by databubble 10 years ago.
Backend log file from time around second freeze
mythback.log (6.5 KB) - added by davideshay@… 10 years ago.
mythbackend.log from similar issues -- associated debug output next
mythgdb.txt.gz (3.1 KB) - added by davideshay@… 10 years ago.
gdb output from same event, I think…

Download all attachments as: .zip

Change History (23)

Changed 10 years ago by databubble

Attachment: mythfrontend.log added

frontend log with -v socket,network,extra

Changed 10 years ago by databubble

Attachment: backtrace.txt added

backtrace

Changed 10 years ago by databubble

Attachment: mythbackend1-log.zip added

first 1000 lines of mythbackend.log

Changed 10 years ago by databubble

Attachment: mythbackend2-log.zip added

last 5000 lines of mythbackend.log

Changed 10 years ago by databubble

Attachment: configure.out added

output from configure

comment:1 Changed 10 years ago by danielk

Owner: changed from Isaac Richards to danielk
Status: newassigned

comment:2 Changed 10 years ago by danielk

Status: assignedinfoneeded

databubble, the backtrace is missing symbols. Can you try to recreate this with a mythbackend compiled with --compile-type=profile ? Since this is difficult to reproduce I'll try my best to diagnose without that info.

comment:3 Changed 10 years ago by anonymous

I have re-compiled with --compile-type=debug still with r21894 and the back-end is running now. Some quick poking, and it hasn't triggered the problem yet.... and that log keeps getting bigger and bigger. As soon as it does fall over (usually a few hours) I'll do my best to extract better quality information for you.

comment:4 in reply to:  3 Changed 10 years ago by anonymous

Replying to anonymous:

I have re-compiled with --compile-type=debug still with r21894 and the back-end is running now. Some quick poking, and it hasn't triggered the problem yet.... and that log keeps getting bigger and bigger. As soon as it does fall over (usually a few hours) I'll do my best to extract better quality information for you.

I can confirm the same issue with:

MythTV Version : 21583-openglvdpau2-nv180 MythTV Branch : branches/release-0-21-fixes Network Protocol : 40 Library API : 0.21.20080304-1 Options compiled in:

linux profile using_oss using_alsa using_pulse using_jack using_backend using_directfb using_dvb using_firewire using_frontend using_glx_proc_addr_arb using_hdhomerun using_iptv using_ivtv using_joystick_menu using_libfftw3 using_lirc using_opengl_video using_opengl_vsync using_v4l using_x11 using_xrandr using_xv using_xvmc using_xvmc_vld using_xvmcw using_bindings_perl using_bindings_python using_opengl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live

I have yet to find a way to trigger it, seems random but destroy WAF.

comment:5 Changed 10 years ago by anonymous

I can confirm the same issue with:

MythTV Version : 21583-openglvdpau2-nv180 MythTV Branch : branches/release-0-21-fixes

r21583 is much too old to be much help. Much socket stuff has been improved around r21726, possibly even after that. Of course it was only improved on the current trunk (soon to be 0.22). 0.21-fixes contains none of the current fixes, and likely won't.

comment:6 Changed 10 years ago by databubble

I tried creating another backtrace, but I´m not sure it's any better than the first. Although, I did recompile with debug after a make distclean:

Please include all output in bug reports. MythTV Version : 21894M MythTV Branch : trunk Network Protocol : 48 Library API : 0.22.20090916-2 QT Version : 4.5.2 Options compiled in:

linux debug using_alsa using_pulse using_backend using_directfb using_dvb using_firewire using_frontend using_hdpvr using_iptv using_lirc using_mheg using_opengl_video using_opengl_vsync using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_xvmc using_xvmc_vld using_bindings_perl using_bindings_python using_opengl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live using_mheg

One thing I can say is that the back-end got itself into it's unresponsive state without the front-end running at all. If I check back, it looks like the backend had a problem around 7:30p, or after it had been running for 8 hours, and made 8 successful recordings. If failed on trying to record "Hannah Montana" (who can blame it?) leaving it empty in the recordings list. Thereafter, nothing more appears in the recordings list and it was unresponsive to the front-end when I finally connected to it 6 hours later.

BTW, I get the following 4 messages repeated a LOT in the log... this seems to be normal (HD-PVR). I've stripped them out.

2009-09-17 19:33:26.859 [h264 @ 0x7f3bed4b07c0]decode_slice_header error 2009-09-17 19:33:26.861 [h264 @ 0x7f3bed4b07c0]no frame! 2009-09-17 19:33:26.862 AFD Error: Unknown decoding error 2009-09-17 19:33:26.864 [h264 @ 0x7f3bed4b07c0]B picture before any references, skipping

Changed 10 years ago by databubble

Attachment: bt_try2.txt added

Attempt at backtrace after compiling with -debug

Changed 10 years ago by databubble

Attachment: backend2.log added

Backend log file from time around second freeze

comment:7 Changed 10 years ago by David Asher <asherml@…>

This backend hang sounds a lot like the one I reported in #7046. There are backtraces and logs there.

Changed 10 years ago by davideshay@…

Attachment: mythback.log added

mythbackend.log from similar issues -- associated debug output next

Changed 10 years ago by davideshay@…

Attachment: mythgdb.txt.gz added

gdb output from same event, I think...

comment:8 Changed 10 years ago by davideshay@…

I'm seeing very similar issues to those reported as well. I've attempted to put put backtraces out here, but I haven't really done that before, so I hope they are correct.

mythbackend version: Please include all output in bug reports. MythTV Version : 21949 MythTV Branch : trunk Network Protocol : 48 Library API : 0.22.20090918-1 QT Version : 4.5.1 Options compiled in:

linux debug using_oss using_alsa using_arts using_jack using_backend using_dvb using_firewire using_frontend using_hdho

merun using_hdpvr using_iptv using_ivtv using_joystick_menu using_lirc using_mheg using_opengl_video using_opengl_vsync using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_xvmc using_bindings_perl using_bindings_python using_open gl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live using_mheg

I can provide frontend info as well. I don't have gdb data for that but could run additional tests if necessary. Let me know.

comment:9 Changed 10 years ago by nickj-fox@…

Similar errors here as well:

MythTV Version : 21954 MythTV Branch : trunk Network Protocol : 48 Library API : 0.22.20090918-1 QT Version : 4.5.2 Options compiled in:

linux profile using_oss using_alsa using_pulse using_jack using_backend using_dvb using_firewire using_frontend using_glx_proc_addr_arb using_hdhomerun using_hdpvr using_iptv using_ivtv using_joystick_menu using_libfftw3 using_lirc using_mheg using_opengl_video using_opengl_vsync using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_xvmc using_xvmc_vld using_xvmcw using_bindings_perl using_bindings_python using_opengl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live using_mheg

comment:10 Changed 10 years ago by danielk

databubble, your backtraces are still missing debugging info.

davideshay, I do not see any evidence of a deadlock in that backtrace. All threads appear to be in valid wait states.

nickj-fox, please do not report "me too" -- it is not helpful. A backtrace with debugging symbols that shows a deadlock would be helpful. (I don't expect you to know if it displays a deadlock or not, just follow the TicketHowTo link on how to generate a full backtrace.)

comment:11 Changed 10 years ago by davideshay@…

danielk, perhaps posted under wrong ticket. Symptom I am getting timeouts on the frontend like this:

2009-09-21 07:03:40.446 MythSocket?(25aae80:55): readStringList: Error, timed out after 30000 ms. 2009-09-21 07:03:40.447 Preview Error: Remote Preview failed due to communications error.

Not always remote preview. Right now I always get the error when entering watch recordings, and nearly always get a pause when exiting a show.

At some point, the front end gets confused and thinks shows are missing in Watch Recordings and I have to at least exit that screen and go back in.

Thought based on the email in mythtv-users to post on this ticket, maybe not. I see that since my backend wasn't completely unresponsive that a backtrace did no good. Do you want -v socket logs on front and back simultaneously or what else would help to track this down?

comment:12 Changed 10 years ago by databubble

Sorry, but I need some help with getting the backtrace. I did a "make distclean" and then ran configure with --compile-type=profile (also tried --compile-type=debug), and installed with "checkconfig" and indeed profile is listed as a compile option...

MythTV Version   : 21992M
MythTV Branch    : trunk
Network Protocol : 48
Library API      : 0.22.20090919-1
QT Version       : 4.5.2
Options compiled in:
 linux profile using_alsa using_pulse using_backend using_directfb using_dvb using_firewire using_frontend using_hdpvr using_iptv using_lirc using_mheg using_opengl_video using_opengl_vsync using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_xvmc using_xvmc_vld using_bindings_perl using_bindings_python using_opengl using_vdpau using_ffmpeg_threads using_libavc_5_3 using_live using_mheg

But whenever I try to connect to the process (as root), I get:

Reading symbols from /usr/bin/mythbackend...(no debugging symbols found)...done.                                                                              
Attaching to program: /usr/bin/mythbackend, process 28932                      
Reading symbols from /usr/lib/libmythtv-0.22.so.0...(no debugging symbols found)...done.                                                                      
Loaded symbols for /usr/lib/libmythtv-0.22.so.0                                
Reading symbols from /usr/lib/libmythavformat-0.22.so.0...(no debugging symbols found)...done.                                                                
Loaded symbols for /usr/lib/libmythavformat-0.22.so.0                          
Reading symbols from /usr/lib/libmythavutil-0.22.so.0...(no debugging symbols found)...done.                                                                  
Loaded symbols for /usr/lib/libmythavutil-0.22.so.0                            
Reading symbols from /usr/lib/libmythavcodec-0.22.so.0...(no debugging symbols found)...done.                                                                 
Loaded symbols for /usr/lib/libmythavcodec-0.22.so.0                           
Reading symbols from /usr/lib/libmythswscale-0.22.so.0...(no debugging symbols found)...done.                                                                 
Loaded symbols for /usr/lib/libmythswscale-0.22.so.0                           
Reading symbols from /usr/lib/libmythupnp-0.22.so.0...(no debugging symbols found)...done.                                                                    
Loaded symbols for /usr/lib/libmythupnp-0.22.so.0                              
Reading symbols from /usr/lib/libmyth-0.22.so.0...(no debugging symbols found)...done.                                                                        
Loaded symbols for /usr/lib/libmyth-0.22.so.0                                  
Reading symbols from /usr/lib/libmythui-0.22.so.0...(no debugging symbols found)...done.                                                                      
Loaded symbols for /usr/lib/libmythui-0.22.so.0                                
Reading symbols from /usr/lib/libmythdb-0.22.so.0...(no debugging symbols found)...done.                                                                      
Loaded symbols for /usr/lib/libmythdb-0.22.so.0                                
Reading symbols from /usr/lib/libmythlivemedia-0.22.so.0...(no debugging symbols found)...done.    

Shouldn't the symbols be there? What am I doing wrong?

Sorry, I have searched and tried to sort this out myself. I just want to extract the backtrace so this ticket can move forward.

Phil

comment:13 Changed 10 years ago by databubble

It appears that the patch in #7132 may resolve the issue I was seeing with the back-end stopping recording and becoming unresponsive. I've been running 14 hours and made about 18 recordings with an HD-PVR, and so far so good. I wasn't able to run that long previously. I'll let it go for another day, and if it's still good then we can close this ticket as a duplicate of 7132.

comment:14 Changed 10 years ago by danielk

Resolution: duplicate
Status: infoneededclosed

Dup #7132.

Note: See TracTickets for help on using tickets.