Opened 9 years ago

Closed 8 years ago

#12408 closed Bug Report - General (Fixed)

mythtv fails to compile with Gentoo gcc 4.9

Reported by: rich0@… Owned by: Karl Egly
Priority: minor Milestone: 0.27.6
Component: MythTV - General Version: 0.27-fixes
Severity: medium Keywords:
Cc: Ticket locked: no

Description

See Gentoo bug: https://bugs.gentoo.org/show_bug.cgi?id=516692

mythtv apparently is missing some object files in the link stage which causes build problems in gcc 4.9:

Apart from mythavtest needs videobuffers.o,

  • mythtranscode also needs videobuffers.o and
  • mythtv-setup needs dtvmultiplex.o, dtvconfparser.so and dtvconfparserhelpers.o

I haven't personally reproduced this but probably could do so if necessary. Feel free to reach out to me with questions/etc. There was quite a bit of investigation as it was originally suspected to be a gcc issue, and escalated with them.

This was most recently tested with 0.27.4 at git commit 3b4390396bf09dfe3741508ecf7fc71a004abd01 on the fixes branch.

Attachments (2)

mythtv-0.27.5-modified-makefiles.tar.gz (10.9 KB) - added by mike.desimone@… 8 years ago.
Modified makefiles to workaround Gentoo link bug
mythtv-0.27.5-visibility.patch (1.0 KB) - added by anonymous 8 years ago.
Patch fixing symbol visibility in shared object file for gcc-4.9. The patch was tested on my Gentoo system, but might not work with other compilers.

Download all attachments as: .zip

Change History (25)

comment:1 Changed 9 years ago by stuartm

Component: MythTV - MythfilldatabaseMythTV - General
Owner: changed from stuartm to Isaac Richards
Status: newassigned

comment:2 Changed 9 years ago by stuartm

Owner: Isaac Richards deleted
Status: assignednew

comment:3 Changed 9 years ago by Frank Phillips <fphillips81@…>

FWIW, I build latest 0.27-fixes all the time with gcc-4.9.2 on Arch Linux.

[fp@viron mythtv-fixes-git]$ gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/lto-wrapper Target: x86_64-unknown-linux-gnu Configured with: /build/gcc/src/gcc-4.9-20150304/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --with-system-zlib --enable-cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-cloog-backend=isl --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --disable-multilib --disable-werror --enable-checking=release Thread model: posix gcc version 4.9.2 20150304 (prerelease) (GCC)

Our gcc doesn't have any relevant patches. Let me know if you want me to check some things for you.

comment:4 Changed 9 years ago by paulh

Resolution: Works for me
Status: newclosed

Work for me with 4.9.2

comment:5 in reply to:  4 Changed 8 years ago by cardoe@…

Resolution: Works for me
Status: closednew

Replying to paulh:

Work for me with 4.9.2

I'm at a loss as to how a very clear issue of missing object files to the linking operation can be marked as "works for me" and closed. Frank Phillip's example shows that he is using lto when building. As the GCC maintainers explained in the bug (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63745) which introduces its own cans of worms but happens to work in this case for MythTV. But if you read they're analysis for the issue you will see that MythTV's build system is underlinking mythtranscode and mythtv-setup.

comment:6 Changed 8 years ago by Stuart Auchterlonie

Any gentoo user able to provide a patch that fixes the underlinking?

It'll take some time here to build something to the point where I could even test for this.

Regards Stuart

comment:7 Changed 8 years ago by Cédric Schieli <cschieli@…>

I'd be happy to test a/o patch, but I was not able to reproduce this on my Gentoo boxes with gcc 4.9.3 installed. Are there any specific flags I should set to trigger this underlinking ?

Regards, Cédric

comment:8 Changed 8 years ago by mike.desimone@…

Having this exact problem with these Gentoo builds of MythTV:

  • 0.27.5_p20150627
  • 0.27.5_p20150904-r1
  • 0.27.5_p20150904-r2

Gentoo's gcc 4.9.3. CFLAGS="-march=amdfam10 -O2 -pipe". Here's the link command and its output:

x86_64-pc-linux-gnu-g++ -Wl,-O1 -o mythavtest main.o commandlineparser.o     -L../../libs/libmyth -L../../libs/libmythtv -L../../external/FFmpeg/libavutil -L../../external/FFmpeg/libavcodec -L../../external/FFmpeg/libavformat -L../../external/FFmpeg/libswscale -L../../external/FFmpeg/libswresample -L../../libs/libmythbase -L../../libs/libmythui -L../../libs/libmythupnp -L../../libs/libmythmetadata -L../../libs/libmythservicecontracts -L../../libs/libmythprotoserver -lmythswscale -lmythavformat -lmythavcodec -lmythavutil -lmythswresample -lmythtv-0.27 -lmythupnp-0.27 -lmythbase-0.27 -lmythui-0.27 -lmyth-0.27 -lmythmetadata-0.27 -lmythservicecontracts-0.27 -lmythprotoserver-0.27 -L../../libs/libmythfreemheg -lmythfreemheg-0.27 -L../../external/libhdhomerun -lmythhdhomerun-0.27 -lXext -lXinerama -lXxf86vm -lXv -lXrandr -lX11 -lxml2 -lcrypto -lass -lfftw3_threads -lfftw3f -lfftw3 -lasound -lfreetype -lxvidcore -lx264 -lvpx -lvorbisenc -lvorbis -ltheoraenc -ltheoradec -logg -lmp3lame -lfaac -lm -ludev -luuid -pthread -lbz2 -lz -ldl -lraw1394 -liec61883 -lavc1394 -lrom1394 -L/var/tmp/portage/media-tv/mythtv-0.27.5_p20150904-r2/work/mythtv-0.27.5/mythtv/external/zeromq/src/.libs -lmythzmq -L/var/tmp/portage/media-tv/mythtv-0.27.5_p20150904-r2/work/mythtv-0.27.5/mythtv/external/nzmqt/src -lmythnzmqt -L/var/tmp/portage/media-tv/mythtv-0.27.5_p20150904-r2/work/mythtv-0.27.5/mythtv/external/qjson/lib -lmythqjson -L/usr/lib64 -L/usr/lib64/qt4 -lGL -lQtSql -L/usr/lib64/mysql -lQtXml -lQtOpenGL -lQtGui -lQtNetwork -lQtCore -lpthread 
main.o: In function `VideoOutput::StartDisplayingFrame()':
/var/tmp/portage/media-tv/mythtv-0.27.5_p20150904-r2/work/mythtv-0.27.5/mythtv/programs/mythavtest/../../libs/libmythtv/videooutbase.h:199: undefined reference to `VideoBuffers::StartDisplayingFrame()'
main.o: In function `VideoOutput::DoneDisplayingFrame(VideoFrame_*)':
/var/tmp/portage/media-tv/mythtv-0.27.5_p20150904-r2/work/mythtv-0.27.5/mythtv/programs/mythavtest/../../libs/libmythtv/videooutbase.h:203: undefined reference to `VideoBuffers::DoneDisplayingFrame(VideoFrame_*)'
main.o: In function `VideoOutput::StartDisplayingFrame()':
/var/tmp/portage/media-tv/mythtv-0.27.5_p20150904-r2/work/mythtv-0.27.5/mythtv/programs/mythavtest/../../libs/libmythtv/videooutbase.h:199: undefined reference to `VideoBuffers::StartDisplayingFrame()'
main.o: In function `VideoOutput::DoneDisplayingFrame(VideoFrame_*)':
/var/tmp/portage/media-tv/mythtv-0.27.5_p20150904-r2/work/mythtv-0.27.5/mythtv/programs/mythavtest/../../libs/libmythtv/videooutbase.h:203: undefined reference to `VideoBuffers::DoneDisplayingFrame(VideoFrame_*)'
collect2: error: ld returned 1 exit status
Makefile:109: recipe for target 'mythavtest' failed
make[2]: *** [mythavtest] Error 1
make[2]: Leaving directory '/var/tmp/portage/media-tv/mythtv-0.27.5_p20150904-r2/work/mythtv-0.27.5/mythtv/programs/mythavtest'
Makefile:58: recipe for target 'sub-mythavtest-make_default' failed
make[1]: *** [sub-mythavtest-make_default] Error 2
make[1]: Leaving directory '/var/tmp/portage/media-tv/mythtv-0.27.5_p20150904-r2/work/mythtv-0.27.5/mythtv/programs'
Makefile:67: recipe for target 'programs' failed
make: *** [programs] Error 2

Installed MythTV is 0.27_p20140321, and was probably built before the compiler was upgraded to the 4.9 series.

Going to attempt a patch.

Changed 8 years ago by mike.desimone@…

Modified makefiles to workaround Gentoo link bug

comment:9 Changed 8 years ago by mike.desimone@…

This is odd. It complains about undefined reference to VideoBuffers::StartDisplayingFrame, but it's right there in the linked-in library:

# nm libmythtv-0.27.so | c++filt | grep VideoBuffers::StartDisplayingFrame
0000000000548000 t VideoBuffers::StartDisplayingFrame()

I've attached a tarball with the three modified Makefiles. All they do is add the aforementioned object files directly to the link line. This is a workaround, and not a real fix. This shouldn't be necessary since these objects are in the linked libraries. Unless, of course, whatever GCC's linker is doing ("devirtualization", I hear) is something it can't do across shared library boundaries.

Let me know if you want anything else tried.

comment:10 Changed 8 years ago by cardoe@…

Mike,

Can you provide a diff of what you changed rather than the changed Makefiles? Those makefiles are auto-generated so we'd need to distill the changes into the qmake files.

As far as the underlying issue, I bet it has something to do with the fact that its a static symbol and there's multiple callers and in fact the caller in question that's problematic is inside of a header file. I'm going to go with the GCC guys here who are saying that 4.9 is closer to the proper behavior while 4.8 and lower was not. Another workaround is to force people to use -flto.

comment:11 Changed 8 years ago by mike.desimone@…

Well, under the caveat that what I changed was a cheat... I can't work up an actual diff right now, but all I did was add the lines in the OBJS = block that start with ../libs/libmythtv. So grep should show you exactly what I did. Which was literally the mods described in the OP. If this isn't good enough, I can probably cough up a diff tomorrow night.

That said, it's basically feeding the linker two copies of the same thing (one in the object file, another in libmythtv) so while it builds, I can't say for sure it works. IMHO -fno-devirtualization or -flto are better workarounds.

Over in embedded land, I've seen a lot of things "break" in 4.9 and later that worked in 4.7 or earlier. One of them was even a bug that just happened to work close enough to right to escape detection. (Note: be careful when porting libraries from C-only developers.)

comment:12 Changed 8 years ago by Karl Egly

Are you trying to work around a broken compiler?

According to GCC's documentation (at https://gcc.gnu.org/onlinedocs/gccint/WHOPR.html ) the whole program optimization / link time optimization is supposed to work with C++ programs using shared libraries with inline functions in headers. So to me this is clearly a bug in their compiler / linker.

The referenced GCC bug (at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63745 ) appears to state the opposite in contradiction with that documentation.

I'm happy to apply a patch that moves the relevant function's implementation from the header to the implementation file if that helps Gentoo users.

comment:13 Changed 8 years ago by cardoe@…

dekarl,

The issue doesn't occur with -flto or -fwhole-program. The issue occurs only without that flag set. It appears that a number of distros are now using -flto by default which is why people aren't seeing that crop up there.

Changed 8 years ago by anonymous

Patch fixing symbol visibility in shared object file for gcc-4.9. The patch was tested on my Gentoo system, but might not work with other compilers.

comment:14 Changed 8 years ago by Karl Dietz <dekarl@…>

In 1c46add12bf49864c1e2c60b43cbb106cf0eb665/mythtv:

Fix symbol visibility for mythtranscode and mythtv-setup

Hopefully that fixes linking with newer GCCs

Refs #12408
https://gcc.gnu.org/onlinedocs/gccint/WHOPR.html
https://gcc.gnu.org/wiki/Visibility

comment:15 Changed 8 years ago by Karl Egly

Good catch. I've rewritten it to use our macro. Looking forward to a success report.

comment:16 Changed 8 years ago by tomegrigg@…

dekarl,

I think maybe your fix needs s/mythexp.h/mythtvexp.h/ and s/MPUBLIC/MTV_PUBLIC/

Otherwise works for me backported to a custom build of 0.26 (don't ask) with gcc 4.9.3 on x86-64 (not Gentoo)

comment:17 Changed 8 years ago by Karl Dietz <dekarl@…>

In b16ebfc0cd5ea9eef8fe997005c940bf95ecde4b/mythtv:

Fix symbol visibility

We split MPUBLIC into one M<name of library>_PUBLIC per library
in 26ea6674c8fd533b9e87d24ffcde5d654dd09a27

Pointed out in https://code.mythtv.org/trac/ticket/12408#comment:16

Refs #12408

comment:18 Changed 8 years ago by Karl Dietz <dekarl@…>

In 0739981fc27896bff5125b415a9b5ae6b1989f51/mythtv:

Fix symbol visibility for mythtranscode and mythtv-setup

Hopefully that fixes linking with newer GCCs

Refs #12408
https://gcc.gnu.org/onlinedocs/gccint/WHOPR.html
https://gcc.gnu.org/wiki/Visibility

(cherry picked from commit 1c46add12bf49864c1e2c60b43cbb106cf0eb665)

Conflicts:

mythtv/libs/libmythbase/mythversion.h
mythtv/libs/libmythtv/dtvmultiplex.h
mythtv/libs/libmythtv/videobuffers.h

comment:19 Changed 8 years ago by Karl Dietz <dekarl@…>

In 68e51d6fd4fa25e985b21cb3a5b5ca6fce9aca9d/mythtv:

Fix symbol visibility

We split MPUBLIC into one M<name of library>_PUBLIC per library
in 26ea6674c8fd533b9e87d24ffcde5d654dd09a27

Pointed out in https://code.mythtv.org/trac/ticket/12408#comment:16

Refs #12408

(cherry picked from commit b16ebfc0cd5ea9eef8fe997005c940bf95ecde4b)

Conflicts:

mythtv/libs/libmythtv/dtvmultiplex.h
mythtv/libs/libmythtv/videobuffers.h

comment:20 Changed 8 years ago by Karl Egly

Milestone: unknown0.27.6
Status: newinfoneeded_new

I pushed the visibility fixes to fixes/0.27 please update and see if it actually fixes the issue as seen on Gentoo.

comment:21 Changed 8 years ago by Karl Egly

Owner: set to Karl Egly

comment:22 Changed 8 years ago by cardoe@…

I've tested this in Gentoo and this works for me now. I've pushed out a new version in Gentoo that contains this update for all users.

comment:23 Changed 8 years ago by Stuart Auchterlonie

Resolution: Fixed
Status: infoneeded_newclosed

Based on the previous comment, I'm going to close this as fixed

Note: See TracTickets for help on using tickets.