qmake build system has bad dependencies which causes --as-needed and future binutils to fail to link

[cardoe@…]

The build system has dependencies on libraries incorrect and poorly laid out. This causes anyone compiling with --as-needed to have missing symbols and libraries aren't built in the correct order for resolving dependencies since apparently qmake respects order rather then dependencies.

The following patch is for release-0_20-fixes, it cleans this whole mess up.

[nigel]

Hi Doug. I have never used '--as-needed' (it is not an option in my ld), but yes, the dependencies need fixing. Sadly, your re-ordering of SUBDIRS causes link failure on Mac OS X:

rm -f libmythui-0.20.0.20.0.dylib libmythui-0.20.dylib libmythui-0.20.0.dylib libmythui-0.20.0.20.dylib
g++ -headerpad_max_install_names -Wl,-search_paths_first -prebind -dynamiclib -seg1addr 0xCC000000 -compatibility_version       0.20 -current_version   0.20.0 -install_name    libmythui-0.20.0.dylib -o libmythui-0.20.0.20.0.dylib mythmainwindow.o mythpainter.o mythimage.o myththemebase.o mythpainter_qt.o xmlparsebase.o mythscreenstack.o mythscreentype.o mythgesture.o mythuitype.o mythuiimage.o mythuitext.o mythuistatetype.o mythlistbutton.o mythfontproperties.o mythuibutton.o myththemedmenu.o mythdialogbox.o moc_mythmainwindow.o moc_mythuitype.o moc_mythlistbutton.o moc_myththemedmenu.o moc_mythdialogbox.o  -L/Users/nigel/.osx-packager/src/qt-mac-free-3.3.6/lib -L/usr/lib -L/Users/nigel/.osx-packager/build/lib -framework Carbon -framework OpenGL -lqt-mt 
ld: warning prebinding disabled because dependent library: libqt-mt.3.dylib is not prebound
ld: Undefined symbols:
__ZN11AppleRemote11setListenerEPNS_8ListenerE
__ZN11AppleRemote13stopListeningEv
__ZN11AppleRemote14startListeningEv
__ZN11AppleRemote19isListeningToRemoteEv
__ZN11AppleRemote7runLoopEv
__ZN11AppleRemote8instanceEv
__ZN11MythContext10GetSettingERK7QStringS2_
__ZN11MythContext11GetHostNameEv
__ZN11MythContext11ThemeWidgetEP7QWidget
__ZN11MythContext13GetMediumFontEv
__ZN11MythContext13GetNumSettingERK7QStringi
__ZN11MythContext13verbose_mutexE
__ZN11MythContext16ResetScreensaverEv
__ZN11MythContext17GetScreenIsAsleepEv
__ZN11MythContext17GetScreenSettingsERiS0_RfS0_S0_S1_
__ZN11MythContext18DoResetScreensaverEv
__ZN11MythContext20DoDisableScreensaverEv
__ZN11MythContext20DoRestoreScreensaverEv
__ZN11MythContext7DBErrorERK7QStringRK9QSqlQuery
__ZN12MediaMonitor15GetMediaMonitorEv
__ZN12MediaMonitor15ValidateAndLockEP15MythMediaDevice
__ZN12MediaMonitor25MonitorRegisterExtensionsEjRK7QString
__ZN12MediaMonitor6UnlockEP15MythMediaDevice
__ZN19AppleRemoteListenerC1EP7QObject
__ZN9MSqlQuery7InitConEv
__ZN9MSqlQuery7prepareERK7QString
__ZN9MSqlQueryC1ERK14_MSqlQueryInfo
__ZN9MSqlQueryD1Ev
__ZTI10MythDialog
_gContext
_print_verbose_messages
__ZN11MythContext14LoadScaleImageE7QStringb
__ZN11MythContext11GetThemeDirEv
__ZN11MythContext8qtconfigEv
__ZN8Settings10GetSettingE7QStringS0_
__ZN11MythContext18GetThemeSearchPathEv
__ZN11MythContext11GetLanguageEv
__ZN11MythContext21GetLanguageAndVariantEv
_globalFontMap
__Z11myth_systemRK7QStringi
__ZN10MythDialog4execEv
__ZN11MythContext10FindPluginERK7QString
__ZN11MythContext10GetConfDirEv
__ZN11MythContext10SetSettingERK7QStringS2_
__ZN11MythContext11GetShareDirEv
__ZN11MythContext11SaveSettingERK7QStringS2_
__ZN11MythContext13GetMainWindowEv
__ZN11MythContext15GetMenuThemeDirEv
__ZN11MythContext16getPluginManagerEv
__ZN11MythContext21SendReceiveStringListER11QStringListbb
__ZN12MediaMonitor19ChooseAndEjectMediaEv
__ZN17MythPluginManager10run_pluginERK7QString
__ZN17MythPluginManager13config_pluginERK7QString
__ZN18MythPasswordDialogC1E7QStringPbS0_P14MythMainWindowPKcb
__ZN3LCD12switchToMenuEP8QPtrListI11LCDMenuItemE7QStringb
__ZN3LCD12switchToTimeEv
__ZN3LCD3GetEv
/usr/bin/libtool: internal link edit command failed
make[2]: *** [libmythui-0.20.0.20.0.dylib] Error 1
make[1]: *** [sub-libmythui] Error 2

because there are actually circular dependencies. I am not a C++ expert, so I have been thinking about this for several months (since libmythui started being used) and can only think of three fixes:

1. Re-engineer the classes so that MythDialog is moved into libmythui, and there is yet another source directory and library for all the fundamental Myth stuff (like MSqlQuery, MythContext, MythMediaMonitor, MythPluginManager), or

2. Change the build system to make one big library, or

3. Use late binding on Mac OS X

Only the first option is a proper fix

[anonymous]

nigel:

I'm well aware that there are circular dependencies. I've been saying that as far back as 0.18 or 0.17. There's A LOT of code that needs to get cleaned up. I've posted this here for Linux users with new binutils that use --as-needed since this will work on there systems. There is no guarentee it will work on other systems. I highlighted the Isaac the circular depends the other night.

[candrews@…]

Is there any news on this ticket? I believe multiple distros now use --as-needed, so I assume this could be becoming a bigger issue, and the cause of more distro provided patches.

[nigel]

I am working on it in #4264

[nigel]

Hmm. I just hacked a build machine's Qt to use ld with --as-needed, deleted all libs and did a build of both 0-20-fixes and current SVN head. No linker errors:

% make
...
cd libmyth && make -f Makefile
make[2]: Entering directory `/home/nigel/mythtv/libs/libmyth'
rm -f libmyth-0.20.so.0.20.0 libmyth-0.20.so libmyth-0.20.so.0 libmyth-0.20.so.0.20
g++ -shared -v -Wl,--as-needed,--verbose -Wl,-soname,libmyth-0.20.so.0 -o libmyth-0.20.so.0.20.0
...
GNU ld version 2.17 Debian GNU/Linux
...
attempt to open /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../lib/crtn.o succeeded
/usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../lib/crtn.o
ln -s libmyth-0.20.so.0.20.0 libmyth-0.20.so
ln -s libmyth-0.20.so.0.20.0 libmyth-0.20.so.0
ln -s libmyth-0.20.so.0.20.0 libmyth-0.20.so.0.20
make[2]: Leaving directory `/home/nigel/mythtv/libs/libmyth'

I have looked at the binutils README, but there don't seem to be any real changes with --as-needed in 2.18 (except for a helper script). Am I going mad? Or does --as-needed do nothing different?

[nigel]

I tested again, on a 64bit machine which has binutils 2.18. Again, no problems in SVN head or 0-20-fixes.

[candrews@…]

I have this line in my /usr/qt/3/mkspecs/linux-g++/qmake.conf file: QMAKE_LFLAGS_RELEASE=-Wl,--hash-style=both,-z,relro,--enable-new-dtags,-O1,-z,now,--as-needed

My machine is an AMD64 running GCC 4.2.2, binutils 2.18, and qt 3.3.8.

When I build mythtv 15429, this is the error I get: g++ -c -pipe -fomit-frame-pointer -O3 -Wall -Wno-switch -Wpointer-arith -Wredundant-decls -Wno-non-virtual-dtor -DSTDC_CONSTANT_MACROS -D_REENTRANT -DMMX -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -DPREFIX=\"/usr\" -DLIBDIR=\"/usr/lib64\" -DUSING_X11 -DUSING_XV -DUSING_IVTV -DUSING_XVMC -DUSING_XVMC_VLD -DUSING_XRANDR -DUSING_OPENGL_VSYNC -DUSING_ALSA -DUSING_OSS -DQT_NO_DEBUG -DQT_THREAD_SUPPORT -DQT_SHARED -DQT_TABLET_SUPPORT -I/usr/qt/3/mkspecs/linux-g++ -I. -I/usr/include -I/usr/include -I../.. -I../../libs -I../../libs/libmyth -I../../libs/libmythtv -I../../libs/libavutil -I../../libs/libavformat -I../../libs/libavcodec -I../../libs/libmythupnp -I../../libs/libmythui -I../../libs/libmythlivemedia -I/usr/qt/3/include -I/usr/X11R6/include -I/usr/X11R6/include -o manualschedule.o manualschedule.cpp g++ -c -pipe -fomit-frame-pointer -O3 -Wall -Wno-switch -Wpointer-arith -Wredundant-decls -Wno-non-virtual-dtor -DSTDC_CONSTANT_MACROS -D_REENTRANT -DMMX -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -DPREFIX=\"/usr\" -DLIBDIR=\"/usr/lib64\" -DUSING_X11 -DUSING_XV -DUSING_IVTV -DUSING_XVMC -DUSING_XVMC_VLD -DUSING_XRANDR -DUSING_OPENGL_VSYNC -DUSING_ALSA -DUSING_OSS -DQT_NO_DEBUG -DQT_THREAD_SUPPORT -DQT_SHARED -DQT_TABLET_SUPPORT -I/usr/qt/3/mkspecs/linux-g++ -I. -I/usr/include -I/usr/include -I../.. -I../../libs -I../../libs/libmyth -I../../libs/libmythtv -I../../libs/libavutil -I../../libs/libavformat -I../../libs/libavcodec -I../../libs/libmythupnp -I../../libs/libmythui -I../../libs/libmythlivemedia -I/usr/qt/3/include -I/usr/X11R6/include -I/usr/X11R6/include -o programrecpriority.o programrecpriority.cpp ../../libs/libmyth/libmyth-0.20.so: undefined reference to `XmlConfiguration::XmlConfiguration?(QString const&)' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `MythXMLClient::~MythXMLClient()' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `UPnp::UPnp()' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `SSDPCache::Find(QString const&)' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `SSDPCache::Find(QString const&, QString const&)' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `MythXMLClient::MythXMLClient(QUrl const&, bool)' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `UPnp::SetConfiguration?(Configuration*)' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `UPnp::Initialize(int, HttpServer?*)' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `UPnpDeviceDesc::Retrieve(QString&, bool)' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `UPnp::g_SSDPCache' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `SSDP::PerformSearch?(QString const&)' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `HttpServer::HttpServer?(int)' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `UPnp::g_UPnpDeviceDesc' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `MythXMLClient::GetConnectionInfo?(QString const&, DatabaseParams?*, QString&)' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `UPnp::g_pConfig' ../../libs/libmyth/libmyth-0.20.so: undefined reference to `UPnp::g_pSSDP' collect2: ld returned 1 exit status make[2]: * [mythtv] Error 1 make[2]: Leaving directory `/var/tmp/portage/media-tv/mythtv-0.21_pre15429/work/mythtv-0.21/programs/mythtv' make[1]: * [sub-mythtv] Error 2 make[1]: * Waiting for unfinished jobs....

If I remove the "--as-needed" from the QMAKE_LDFLAGS_RELEASE line in qmake.conf, mythtv builds and runs fine.

[nigel]

Hooray. Someone actually showed me the real errors they are getting!
From the patches, I had assumed the problems were in the library linking (closure), not the program linking.
I'm not sure this will be fixable, because --as-needed just seems to be broken on the link line. Check out this output after I pruned LIBS down to -lGL -lmyth -lmythui:

/usr/bin/ld --eh-frame-hdr -m elf_i386 --hash-style=both -dynamic-linker /lib/ld-linux.so.2 -o mythtv /usr/lib/gcc/i486-linux-gnu/4.1.3/../../../../lib/crt1.o /usr/lib/gcc/i486-linux-gnu/4.1.3/../../../../lib/crti.o /usr/lib/gcc/i486-linux-gnu/4.1.3/crtbegin.o -L/usr/share/qt3/lib -L../../libs/libmyth -L../../libs/libmythui -L/usr/lib/gcc/i486-linux-gnu/4.1.3 -L/usr/lib/gcc/i486-linux-gnu/4.1.3 -L/usr/lib/gcc/i486-linux-gnu/4.1.3/../../../../lib -L/lib/../lib -L/usr/lib/../lib --as-needed -v main.o -lmyth-0.20.2 -lGL -lmythui-0.20.2 -lqt-mt -lXext -lX11 -lpthread -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/i486-linux-gnu/4.1.3/crtend.o /usr/lib/gcc/i486-linux-gnu/4.1.3/../../../../lib/crtn.o
GNU ld (GNU Binutils for Debian) 2.18
main.o: In function `main':
main.cpp:(.text+0x3c86): undefined reference to `TV::InitKeys()'
main.cpp:(.text+0x3c9c): undefined reference to `TV::TV()'
main.cpp:(.text+0x3cad): undefined reference to `TV::Init(bool)'
main.cpp:(.text+0x431e): undefined reference to `ProgramInfo::ProgramInfo()'
main.cpp:(.text+0x43e1): undefined reference to `TV::Playback(ProgramInfo*)'
main.cpp:(.text+0x43ff): undefined reference to `TV::GetState() const'
main.cpp:(.text+0x449b): undefined reference to `TV::LiveTV(bool, bool)'
../../libs/libmythui/libmythui-0.20.2.so: undefined reference to `glClearDepth'
../../libs/libmythui/libmythui-0.20.2.so: undefined reference to `glMatrixMode'
../../libs/libmythui/libmythui-0.20.2.so: undefined reference to `glHint'...
% objdump -T /usr/lib/libGL.so | grep glClearDepth
0004ace0 g    DF .text  00000000  Base        glClearDepth

libGL is parsed first, it contains the symbols, but 'ld --as-needed' doesn't find it. I'm at a loss (again).

[nigel]

OK. I have worked it out. Linux ld --as-needed uses the dependencies that are compiled into the libraries that it is parsing (like Solaris's ld -R, except that they are used at runtime). Several checkins on the way!

[nigel]

(In [15472]) First of several Linux 'ld --as-needed' patches. Thanks to Doug (cardoe at gentoo.org). See #2402.

[nigel]

(In [15476]) Reorder libs to get around ld --as-needed problems. See #2402. Basically, build libmythui earlier, and libmythtv later.

[nigel]

(In [15477]) Also rebuild libmythtv if libmythui changes. Closes #2402

[anonymous]

this cannot be marked as fixed, problem still there in last svn revisions