MacOS X - Qt binaries log spam

[Craig Treleaven <ctreleaven@…>]

MythFrontend writes hundreds of log entries per second complaining about 2 sets of Qt binaries. Sample below; attached file has log from starting the frontend and immediately exiting--2,300 line.

The main frontend functions seem to work properly (set up recording, play recording, etc) but the log spam makes it difficult to identify other problems.

On Mac OS X, you might be loading two sets of Qt binaries into the same process. Check that all plugins are compiled against the right Qt binaries. Export DYLD_PRINT_LIBRARIES=1 and check that only one set of binaries are being loaded.
QObject::moveToThread: Current thread (0x884a70) is not the object's thread (0x80a680).
Cannot move to target thread (0x80a680)

[Raymond Wagner]

attaching as a file, as the text says to

[Nigel]

Can't reproduce with a fresh build on 10.6.5, but last time I encountered that, moving the built libraries out of the way silenced the errors. i.e.:

mv .osx-packager/build .osx-packager/build.old

I think it was caused by DYLD_LIBRARY_PATH containing the non-packaged libraries, and the runtime system had two sets of identical libraries being observed.

[Craig Treleaven <ctreleaven@…>]

Thanks, Nigel! I confirm that this workaround solved the problem for me.

[Nigel]

export DYLD_PRINT_LIBRARIES=1 reveals that these problems are due to Qt plugins:

2011-05-10 21:54:16.279 LIRC, Error: Failed to connect to Unix socket '/var/run/lirc/lircd'
                        eno: No such file or directory (2)
dyld: loaded: /System/Library/Extensions/IOHIDFamily.kext/Contents/PlugIns/IOHIDLib.plugin/Contents/MacOS/IOHIDLib
2011-05-10 21:54:16.289 UDPListener: bound to port 6948
2011-05-10 21:54:16.342 Desktop video mode: 1280x800 60.000 Hz
2011-05-10 21:54:16.355 Enabled verbose msgs:  important general
2011-05-10 21:54:16.358 Loading en_us translation for module mythfrontend
dyld: loaded: /Volumes/Backups/MythTV/.osx-packager/build/plugins/imageformats/libqgif.dylib
dyld: loaded: /Volumes/Backups/MythTV/.osx-packager/build/lib/libQtGui.4.dylib
dyld: loaded: /Volumes/Backups/MythTV/.osx-packager/build/lib/libQtCore.4.dylib
objc[95787]: Class QMacSoundDelegate is implemented in both /Volumes/Backups/MythTV/MythFrontend.app/Contents/MacOS/../Frameworks/QtGui.framework/QtGui and /Volumes/Backups/MythTV/.osx-packager/build/lib/libQtGui.4.dylib. One of the two will be used. Which one is undefined.

QObject::moveToThread: Current thread (0x4481a0) is not the object's thread (0x500b00).
Cannot move to target thread (0x500b00)

On Mac OS X, you might be loading two sets of Qt binaries into the same process. Check that all plugins are compiled against the right Qt binaries. Export DYLD_PRINT_LIBRARIES=1 and check that only one set of binaries are being loaded.
dyld: loaded: /Volumes/Backups/MythTV/.osx-packager/build/plugins/imageformats/libqico.dylib
QObject::moveToThread: Current thread (0x4481a0) is not the object's thread (0x500b00).
Cannot move to target thread (0x500b00)

i.e. moving .osx-packager/build/plugins out of the way removes the warning.
The only plugins that we load are the image ones, and ​http://code.mythtv.org/trac/ticket/8898 says that QImage uses QCoreApplication::libraryPaths() to locate its plugins. The doc for that class now hints that qt.conf can override the defaults.

[Nigel]

QCoreApplication::libraryPaths() for me reports ("/Volumes/Backups?/MythTV/.osx-packager/build/plugins", "/Volumes/Backups?/MythTV/MythFrontend.app/Contents/MacOS").
Qt 4.6's QLibraryInfoPrivate::findConfiguration() does not locate qt.conf because QCoreApplication::instance() is false, so the Q_OS_MAC is never called. Bypassing that if() still fails - bundleRef set, urlRef isn't. Still trying to work out why.

[mythtv@…]

This issue is being caused by Qt loading its plugins from the .osx-packager/build/plugins directory due to a misconfigured plugins path, as returned by ​QLibraryInfo::Location(QLibraryInfo::PluginsPath). It appears to become more of an issue with later versions of Qt, as with 4.8 at least it results in a application crash for me.

The documentation covering ​plugins as part of Deploying an Application on Mac OS X suggests the use of a ​qt.conf file in order to adjust the plugins path to be within the application bundle. However, placing a qt.conf file within <app_bundle>/Contents/Resources, as suggested, is not picked up. The reason for this is that the code within QLibraryInfoPrivate::findConfiguration(), whose role is to locate and parse the qt.conf file during initialisation, contains a check for whether the QCoreApplication has been initialised:

    if (!QFile::exists(qtconfig) && QCoreApplication::instance()) {
        ...
    }

and the iso8859_codecs global static array of ​QTextCodec objects, which are implemented as plugins, in mythtv/libs/libmythtv/mpeg/dvbdescriptors.cpp is initialised before reaching main() and consequently before the QApplication instance has been initialised.

The attached patches move the iso8859_codecs array into the decode_text() function within mythtv/libs/libmythtv/mpeg/dvbdescriptors.cpp, as this appears to be the only function that requires the array and through relocating its initialisation is deferred until the function is called, which will be after the QApplication has been initialised. This change allows the QLibraryInfoPrivate::findConfiguration() function to be called later and consequently it behaves as expected and a ​qt.conf file placed within <app_bundle>/Contents/Resources will be correctly detected. The osx-packager.pl script is then modified to copy the plugins from .osx-packager/build/plugins to <app_bundle>/Contents/Resources and to rewrite the embedded dynamic library references to refer to the copies of Qt libraries within the bundle, and an empty <app_bundle>/Contents/Resources/qt.conf file is generated to indicate to Qt that it should not be using the installation prefix path it was compiled with.

The loading of the plugins from .osx-packager/build/plugins and consequently the processing of the embedded dynamic library references to the version of Qt libraries within .osx-packager/build/lib when combined with the rewritten references to Qt libraries within the packaged executables within the application bundle are the root cause of "loading two sets of Qt binaries" errors. Copying the plugins into the application bundle and rewriting their dynamic library references and then actually loading them is what ensures only one set of Qt binaries are ever loaded.

[Nigel]

Peter, many thanks for your analysis and patches. I'm still digesting all that (and might defer on the dvbdescriptors.cpp changes), but don't understand the packager change - copying $PREFIX/plugins.
.osx-packager/build/plugins contains lots of Qt plugins that we don't need, and about 50 lines later, the ones we do use (imageformats/libqgif.dylib and libqjpeg.dylib) are copied.
Haven't tested yet, but ignoring the path difference (Contents/MacOS/imageformats vs Contents/Resources?/plugins/imageformats), wouldn't just Syscall("touch $res/qt.conf") suffice?

[Stuart Auchterlonie]

Nigel,

Don't worry about the packaging side of this.

There is a lot of work being done (primarily by jyavenard) to make mythtv build against Qt install as a framework as downloaded from Nokia, rather than building it ourselves.

As side effect of this is a cleanup of the packaging.

We intended to switch the osx buildbot over to this new schema once 0.25 is out.

Regards Stuart Auchterlonie

[JYA]

I'm almost done with the rewrite. The new code let you compile in all mac supported architecture, as well as full universal (32/64) or Intel/PPC whatever

The error about Qt loading to set of libraries is due to a bug still existing in Qt 4.8 which I've logged into QTBUG-24541. Even if you properly set a qt.conf, it won't read it. Note that the patch I've provided in QTBUG-24541 isn't sufficient to fully fix the problem. There's another problem in which the running directory isn't properly calculated, so the relative path put in qt.conf do not work

Basically, Qt always ignores the Qt plugins found in the application bundle, even if a qt.conf file was properly corrected. So what occurs is that the MythFrontend application is loaded, then Qt loads the plugins found in the Qt system directory or wherever it was first installed. As this has dependencies on the set of Qt libraries outside the bundle, they too are loaded and you end up with some nastiness. Usually it crashes.

There are currently only three ways to fix it. 1- Move where the original Qt were stored (if using osx-packager.pl it will be .osx-packager) 2- Set an environment variable: export QT_PLUGIN_PATH=~/Work/mythtv/MythFrontend.app/Contents/PlugIns/ Where ~/Work/mythtv is the installation where the MythFrontend.app is installed. 3-In MythFrontend.app/Content/MacOs create a link to ../Resources/Plugins?

The best is to start it from the command line with

QT_PLUGIN_PATH=`pwd` MythFrontend.app/Content/MacOs/MythFrontend

It's not pretty :(

If you apply the patch I posted in QTBUG-24541 You could created a qt.conf file in: MythFrontend.app/Content/Resources/qt.conf

containing the following info:

[Paths]
Prefix = /Home/Users/jyavenard/Work/mythtv/MythFrontend.app/Content
Plugins = Plugins

no more crash after that...

I have resolved most of the issues in the new packager, you still get the warning, but nowhere near as bad, and more importantly: it doesn't crash

[JYA]

Oh, and I forgot about the dependencies not being properly recalculated form the plugins folder.

I wish I had read this bug earlier, I could have saved you a lot of time as I've sorted them all

[mythtv@…]

Hi Nigel,

No problem.

The dvbdescriptors.cpp changes are key, without them Qt will not detect the presence of a qt.conf file and will not modify its paths accordingly. I entirely understand your reticence over this change, perhaps you could get someone more familiar with the dvbdescriptors.cpp code to review/make it?

libqgif.dylib and libqjpeg.dylib are not the only Qt plugins required. We are definitely using the text codecs, which live beneath .osx-packager/build/plugins/codecs. I am unsure as to whether any other plugins are required; But I would suggest the drawbacks associated with the possibility of someone introducing new code that requires the services of hitherto unused plugins outweighs any potential benefits associated with not including them all within the application bundles.

Any plugins we do load must be processed by osx-bundler.pl in order to have the references to the Qt dynamic libraries that are embedded within them rewritten from pointing to the Qt libraries in .osx-packager/build/lib so that they point to the libraries within <app_bundle>/Contents/Resources. Failure to do this results in the application, which has been processed by osx-bundler.pl, loading <app_bundle>/Contents/Resources/qtcore.dylib, then later a plugin which references .osx-packager/build/lib/qtcore.dylib. The dynamic linker does not understand that these are the same libraries and therefore loads two copies of qtcore.dylib into memory. Resulting in two sets of global variables, Qt threads running, etc. which ultimately leads to lots of log spam and probably some form a application crash.

The presence of an empty <app_bundle>/Contents/Resources/qt.conf in an application bundle containing Qt libraries when QLibraryInfo is initialised after QCoreApplication causes Qt to adjust its default paths as described in ​Using qt.conf. Having reviewed it I realise that I have not validated that the plugins copied to <app_bundle>/Contents/Resources/plugins by my changes to osx-packager.pl are actually being loaded. All I have validated is that the changes to dvbdescriptors.cpp are required to get Qt to recognise the presence of <app_bundle>/Contents/Resources/qt.conf and that with a qt.conf present, Qt will not load the plugins from .osx-packager/build/plugins therefore preventing the log spam/crash.

Re-reading the ​qt.conf documentation suggests that we both may be wrong. I now believe that the plugins should all be placed beneath <app_bundle>/Contents/MacOS/plugins (i.e. the image format plugins belong beneath <app_bundle>/Contents/MacOS/plugins/imageformats), because <app_bundle>/Contents/MacOS is the equivalent of QCoreApplication::applicationDirPath()? But we can put the plugins wherever we wish as long as we add a Plugins = <directory> entry beneath the [Paths] section within the qt.conf file.

Indeed Syscall("touch $res/qt.conf") would probably suffice, my proposal is based upon a combination of “when in Perl, do as Perl” and the fact that we may wish to actually write some information into the qt.conf file in the future (e.g. customise the Plugins path).

Thanks

Peter

[mythtv@…]

Jyavenard,

I suspect your bug report ​QTBUG-24541 will be rejected. I believe that the bug lies within the dvbdescriptors.cpp code in that it is not valid to attempt to load plugins before a QCoreApplication has been inialised. Whilst I am unable to explain exactly why this is, I can well imagine that some parts of QCoreApplication must be initialised prior to loading plugins.

I also disagree with your logic regarding embedding a qt.conf using the Qt resource system. I suspect that this implementation has been chosen in order to allow an application developer to choose to explicitly ignore the presence of a qt.conf file as some form of basic security measure.

But thanks for looking into all of this. I'm looking forward to having a fully operational MythTV install on my Mac.

Peter

[Github]

Delay first call to QTextCodec until after main QApplication has been created

Calling QTextCode before QApplication has been created, causes Qt to ignore local plugins and use system-wide instead. Not a problem on linux because they are one and the same, but on mac they are different: resulting in loading two sets of Qt libraries in RAM, ultimately leading to a crash

Fixes #9447

Signed-off-by: Jean-Yves Avenard <jyavenard@…>

Branch: master Changeset: 4392c16c6bec8556981824ce1a5655198f017040