Network Control socket fails to listen on second address

[Paul Saunders <darac@…>]

As of 0.25, the Network Control socket (port 6456) doesn't work. At least, this is the immediately obvious problem. Looking at the logs, I can see that it successfully listens on 127.0.0.1, but then fails listening to the next address in the list (for the attached log, that is 192.168.123.1, but the same occurs on another machine where the second, failing address is public).

With the frontend still running, I can see that:

1. there is nothing already listening on that port (sudo netstat -nlp | grep 6456 returns nothing) and
2. I can listen on that port (nc -l -p 6456 and even 'nc -l -s 192.168.123.1 -p 6456` start up and print commands from my remote control).

I'm not quite sure how to proceed, though. I can see there is a problem here, but without knowing why MythFrontend is not listening, it's a bit difficult to find the root cause.

[Raymond Wagner]

Waiting on logs presumed to be forthcoming. Also, the default port is 6546, not 6456. You were likely checking netstat for the wrong port. Is there anything already listening on that?

[Paul Saunders <darac+mythtv@…>]

Frontend Log (Debug Level)

[Paul Saunders <darac+mythtv@…>]

Mythfrontend --version

[Paul Saunders <darac@…>]

Sorry. My laptop crashed between creating the ticket and attaching the logs.

6456 was a typo here in the ticket. I have actually checked port 6546.

[Raymond Wagner]

Just a few side comments first. The logs report version "v0.25pre". You're running a pre-release and should upgrade, but there haven't been any changes to the listen servers in several weeks, and that's not likely to cause your problems. More importantly, that's not a version string MythTV would generate itself. MythTV would have included a specific commit hash, as well as a counter of commits since that tag, likely in the 4000+ range. If you are getting MythTV from a third party packager, it would be beneficial to know who, so we could contact them and get a more descriptive string put in there.

Line 96 indicates you're running your backend against an internet facing IP address. Generally this is advised against for security reasons, and just simple lack of bandwidth needed to make such configuration worthwhile. Since you don't have an IP forced for that host, MythTV scans your available addresses, and automatically selects where to listen. You'll note on line 34 that it refused to listen on the non-RFC1918 address. On the other hand, considering you have multiple internet facing addresses, as well as multiple internal subnets, you presumably know what you're doing in that regard.

[Raymond Wagner]

Replying to wagnerrp:

The logs report version "v0.25pre". That's not a version string MythTV would generate itself.

Scratch that, it appears that IS the string the build scripts produce now...

[Raymond Wagner]

In f4d7bffcc71904d84b06046213ba453fa2aa2865/mythtv:

Add logging information as to why MythTV would fail to listen on a port.

Refs #10633

[Raymond Wagner]

In 1a671d0b252d2b33bd1e96258e65a15d52a5fc7d/mythtv:

Add logging information as to why MythTV would fail to listen on a port.

Refs #10633
(cherry picked from commit f4d7bffcc71904d84b06046213ba453fa2aa2865)

[beirdo]

Replying to wagnerrp:

Replying to wagnerrp:

The logs report version "v0.25pre". That's not a version string MythTV would generate itself.

Scratch that, it appears that IS the string the build scripts produce now...

That means that his build was made from the tarball. I have reuploaded the tarball since with the proper value set in VERSION, and fixed the release script so it doesn't happen again. If it's a git build, then the only time you see the tag as the version is when you are right on that tag, but then the branch would also be filled out.

It might be a good idea to redownload the tarball to remove confusion later. If this was packaged by someone, please let us know which package so we can let them know as well.

[Raymond Wagner]

Paul Saunders:

Could you test again with the above changes to get an explanation of the error? My only guess is you've got something else already listening on those ports.

[Paul Saunders <darac@…>]

OK. For the record, the build I initially reported the bug against was the Debian-Multimedia package, version 0.25-0.3.

I've now re-built from git commit 1a671d0 and re-run mythfrontend again. Perhaps a little surprisingly, the error now is:

Failed listening on TCP 192.168.123.1:6546 - Error 10: Operation on socket is not supported

I can attach the full log if you like, but I suspect that's the bit you're interested in.

[martin_ginkel@…]

I have exactly the same problem on mythbuntu with official debs,

2:0.25.0+fixes.20120506.1c4066e.

The problem does not exist on opensuse (my other frontend) on a 2.6.39 kernel. Maybe this is related to newer kernels and different socket handling. I have not seen anything special in strace.

[martin_ginkel@…]

OK after some debugging I found a strange but simple solution:

On my myth box I have the environment variables http_proxy=​http://my.squid.box:3128 enabled. The Squid is working, but apparently QT uses these always for all tcp sockets. After unsetting the variables, mythfrontend listens as expected.

The code for this is somewhere deep in QTcpServer::listen(), I have no Idea why this only fails for the network control port. But on my laptop it always worked, until I set the same environment variables, then the network control socket failed as well.

Solution:

export http_proxy=

[martin_ginkel@…]

Did ab bit of research about proxy network operation in QT. Apparently they introduced the QNetworkProxy in 4.1 and allowed to set an application-global proxying of all network connections.

​http://doc.qt.nokia.com/4.7-snapshot/qnetworkproxy.html#setApplicationProxy

I still don't see, how this is directly related to listening servers but apparently the QTcpServer works with the DefaultProxy? proxy type by default. This means it respects the global application setting.

If the application setting is set to global QNetworkProxy::HttpProxy? then all connections (not only HTTP-outgoing) are supposed to tunnel through a transparent HTTP proxy.

Eventually one could set the Proxy mode of the PrivTcpServer? in ServerPool? to use the NoProxy? mode. For the special network non-HTTP ports it does not make any sense to use Proxies automatically.

[bjoernv@…]

I have this bug in Mythfrontend 0.26 too.

[stuartm]

Paul, it would be helpful if you could confirm that you were using a proxy and whether setting the http_proxy environmental variable to an empty string works for you?

[JYA]

http_proxy is required and necessary when opening some media with FFmpeg

so either you use it, or you don't

you can't simply disable http_proxy in a blank manner.

[martin_ginkel@…]

Replying to jyavenard:

http_proxy is required and necessary when opening some media with FFmpeg so either you use it, or you don't you can't simply disable http_proxy in a blank manner.

It might be useful for client connections like ffmpeg talking to a http server in the internet, but in this case you try to establish a *server* socket for the frontend -- listening for remote control clients.

This should never be proxied from the server side, because it stops working in this case (obviously).

Summary: use http_proxy when building client http connects, ignore it for any server socket.

[bjoernv@…]

Clearing http_proxy didn't worked for me, maybe because MythFrontend reads KDE's proxy settings and not http_proxy directly.

But disabling proxies with networkControl->setProxy(QNetworkProxy::NoProxy?) works. I found similar code here:

mythtv/libs/libmythprotoserver/mythsocketmanager.cpp
mythtv/programs/mythbackend/mainserver.cpp
mythtv/programs/mythbackend/mediaserver.cpp

So please try this patch:

diff --git a/mythtv/programs/mythfrontend/main.cpp b/mythtv/programs/mythfrontend/main.cpp
index 7cac474..6bcfc50 100644
--- a/mythtv/programs/mythfrontend/main.cpp
+++ b/mythtv/programs/mythfrontend/main.cpp
@@ -1704,6 +1704,8 @@ int main(int argc, char **argv)
     {
         int port = gCoreContext->GetNumSetting("NetworkControlPort", 6546);
         networkControl = new NetworkControl();
+        // work-around #10633 (https://code.mythtv.org/trac/ticket/10633)
+        networkControl->setProxy(QNetworkProxy::NoProxy);
         if (!networkControl->listen(port))
             LOG(VB_GENERAL, LOG_ERR,
                 QString("NetworkControl failed to bind to port %1.")

[Karl Egly]

Maybe we should not use proxies for our server sockets at all. (Ignoring the fact that there may be obscure use cases when one might want to open listening sockets via a Socks5 proxy.)

Did you by any chance set the http_proxy for other things, e.g. Socks5, too?

Although I do not understand why a proxy without ListeningCapability? is used for listening... ​http://doc.qt.io/qt-4.8/qnetworkproxy.html#Capability-enum

Here is the relevant code. It falls through to "use default proxy which will raise an error". Its not clear why raising an error is preferable to not using a proxy if there is no listening capable proxy available. ​http://code.qt.io/cgit/qt/qtbase.git/tree/src/network/socket/qtcpserver.cpp#n134

There is another open bug about QTcpServer failing when a HTTP proxy is set. So I'm tempted to close this a upstream bug. See ​https://bugreports.qt.io/browse/QTBUG-48194

Setting it to infoneeded_new instead as a hack to work around that bug in QT has already been posted. Does that hack fix it for you?

[Karl Egly]

Reschedule all tickets planned for, but not solved in time for, 0.27.6 to 0.27.7.

[cas@…]

This bug still exists in MythTV 29 and is now preventing mythbackend from starting.

I just had to add unset http_proxy https_proxy to my /etc/default/mythtv-backend script (sourced by /etc/init.d/mythtv-backend) to stop mythbackend from insanely trying to use my squid proxy to listen on port 6549.

There is no reason for any program to ever try to use a http_proxy to listen on a port. That makes no sense at all. The only reason to ever use a http proxy is when you're trying to fetch something. i.e. a client requesting stuff, not a server listening.

[Peter Bennett]

In MythTV 29 there is a new feature in mythtv-setup, general, Host Address Backend Setup, called "listen on all IP addresses". Have you tried that? That causes it to only listen on the generic ip addresses 0.0.0.0 and [0:0]. That should be the preferred setup option in future.

If that does not work, please paste here the backend log where it lists the ip addresses and ports being listened on.

[cas@…]

yep, it was already set to that.

Here's the log extract from when it died. I've also included the version & build info lines.

2017-08-03 02:27:15.440228 C [27769/27769] thread_unknown mythcommandlineparser.cpp:2619 (ConfigureLogging) - mythbackend version: fixes/29 [v29.0-b4e4ddcf97] www.mythtv.org
2017-08-03 02:27:15.440289 C [27769/27769] thread_unknown mythcommandlineparser.cpp:2621 (ConfigureLogging) - Qt version: compile: 5.7.1, runtime: 5.7.1
2017-08-03 02:27:15.440295 N [27769/27769] thread_unknown mythcommandlineparser.cpp:2623 (ConfigureLogging) - Enabled verbose msgs: none system general
...
2017-08-03 02:27:15.990051 I [27769/27769] CoreContext mythsystemunix.cpp:948 (Fork) - Managed child (PID: 27777) has started! * command=ping -t 1 -c 1  proxy.mydomain  >/dev/null 2>&1, timeout=0
2017-08-03 02:27:15.990200 I [27769/27774] SystemSignalManager mythsystemunix.cpp:509 (run) - Starting process signal handler
2017-08-03 02:27:15.990262 I [27769/27773] SystemManager mythsystemunix.cpp:276 (run) - Starting process manager
2017-08-03 02:27:15.990323 I [27769/27775] SystemIOHandlerR mythsystemunix.cpp:92 (run) - Starting IO manager (read)
2017-08-03 02:27:15.990347 I [27769/27776] SystemIOHandlerW mythsystemunix.cpp:92 (run) - Starting IO manager (write)
2017-08-03 02:27:16.010485 I [27769/27773] SystemManager mythsystemunix.cpp:354 (run) - Managed child (PID: 27777) has exited! command=ping -t 1 -c 1  proxy.mydomain  >/dev/null 2>&1, status=0, result=0
2017-08-03 02:27:16.047744 I [27769/27769] CoreContext schemawizard.cpp:120 (Compare) - Current MythTV Schema Version (DBSchemaVer): 1348
...

2017-08-03 02:27:20.857412 I [27769/27769] CoreContext housekeeper.cpp:706 (Start) - Starting HouseKeeper.
2017-08-03 02:27:20.873567 I [27769/27769] CoreContext serverpool.cpp:422 (listen) - Listening on TCP 0.0.0.0:6544
2017-08-03 02:27:20.874681 I [27769/27769] CoreContext serverpool.cpp:422 (listen) - Listening on TCP [::]:6544
2017-08-03 02:27:20.874735 I [27769/27769] CoreContext serverpool.cpp:422 (listen) - Listening on TCP 0.0.0.0:6554
2017-08-03 02:27:20.874779 I [27769/27769] CoreContext serverpool.cpp:422 (listen) - Listening on TCP [::]:6554
2017-08-03 02:27:20.874858 E [27769/27769] CoreContext serverpool.cpp:437 (listen) - Failed listening on TCP 0.0.0.0:6549 - Error 10: Operation on socket is not supported
2017-08-03 02:27:20.874878 I [27769/27769] CoreContext serverpool.cpp:454 (listen) - No IPv4 support on this system. Disabling MythTV IPv4.
2017-08-03 02:27:20.874915 E [27769/27769] CoreContext serverpool.cpp:437 (listen) - Failed listening on TCP [::]:6549 - Error 10: Operation on socket is not supported
2017-08-03 02:27:20.874928 I [27769/27769] CoreContext serverpool.cpp:463 (listen) - No IPv6 support on this system. Disabling MythTV IPv6.
2017-08-03 02:27:20.874942 E [27769/27769] CoreContext mediaserver.cpp:111 (Init) - MediaServer: WebSocketServer Create Error
2017-08-03 02:27:20.889206 I [27769/27769] CoreContext main_helpers.cpp:687 (run_backend) - Main::Registering HttpStatus Extension
2017-08-03 02:27:20.890239 C [27769/27769] CoreContext main_helpers.cpp:700 (run_backend) - Backend exiting, MainServer initialization error.
2017-08-03 02:27:23.856569 I [27769/27769] CoreContext bonjourregister.cpp:28 (~BonjourRegister) - Bonjour: De-registering service '' on ''

What's really weird is that it doesn't try to use the proxy to listen on 6544 or 6554, it only does that for port 6549.

After I unset http_proxy, it works. Doesn't try to ping my squid proxy server, either.

It makes sense for myth to use a proxy if/when it ever needs to fetch data from the net. or to pass the http_proxy env var to any external scripts/programs it might call to do that. but not when listening on a socket.

[Stuart Auchterlonie]

Something definitely isn't right here. A quick look at the code and I can see a very odd looking else here

​https://github.com/MythTV/mythtv/blob/master/mythtv/libs/libmythbase/serverpool.cpp#L425

and it's been there for a while.

[Peter Bennett]

There are many types of proxy, some of them support listening. See ​http://doc.qt.io/qt-5/qnetworkproxy.html#Capability-enum .

MythTV uses a proxy for all socket listening if there is one. I don't want to simply prevent proxies from being used for listening. People may use a proxy for tunneling or other things. So it looks like it successfully sets up your proxy for other ports but not for port 6549. Is it possible another application is using the proxy for port 6549?

There is one bug I can fix - this is where it assumes that there is "No IPV4 support" and "No IPV6 support" and therefore shuts down. If I fix this logic, the backend will start successfully, but port 6549 will not work.

It uses port 6549 for SSDP, so if that port is not available the thing that will be affected is the SSDP, part of UPNP discovery. That affects the ability of frontends to find a backend without a config.xml file, and possibly the ability for consumer devices to access videos from the backend, so it may not be important to you.

[Peter Bennett <pbennett@…>]

In 951e601107aa4974fd8e6284071c17611a96283a/mythtv:

Networking: Improve check for IPV4 and IPV6 support

Under certain conditions the system is incorrectly finding that
there is no support for IPV4 and IPV6. This adds some more logic
to the process.

Refs #10633

[Peter Bennett <pbennett@…>]

In 9ea488fbe28abfe9fd05c2055620435cd8843078/mythtv:

Networking: Improve check for IPV4 and IPV6 support

Under certain conditions the system is incorrectly finding that
there is no support for IPV4 and IPV6. This adds some more logic
to the process.

Refs #10633

(cherry picked from commit 951e601107aa4974fd8e6284071c17611a96283a)

[Peter Bennett]

Replying to stuarta:

Something definitely isn't right here. A quick look at the code and I can see a very odd looking else here

​https://github.com/MythTV/mythtv/blob/master/mythtv/libs/libmythbase/serverpool.cpp#L425

and it's been there for a while.

I have removed the dangling "else". It was wrong but would not have caused a problem unless we listened on port 0, which I don't think we do.

[cas@…]

The problem is caused by the fact that you're trying to listen on a port using a http proxy.

That makes no sense. There are no circumstances where that makes any sense at all. It is not a thing that can or should be done. It does not and can not work.

The solution is to NOT do that.

The fact that you your response includes "People may use a proxy for tunnelling or other things" indicates that you do not know the subject matter well enough to make a judgement on this - it looks like code for "i don't know but maybe we did this sometime for some good reason that someone, somewhere knows". I can understand a reluctance to not mess with something you're not an expert on but there isn't any good reason for doing this, not one. This is a bug, nothing more. If anyone did need to do anything like that, there is no way they ever would or could do it via a web proxy.

BTW, did you notice that doc.qt.io link you posted ONLY mentioned binding to a port for a socks5 proxy? Not for http or ftp proxies? That it even says "supports only outgoing TCP connections" for them? That's because web & ftp proxies don't support that. They are for clients, not servers of any kind. It's a good idea to use the proxy for outgoing requests (http/https/ftp or whatever), not for listening on a port for incoming connections.

But you don't have to believe me (I've been doing network admin on unix systems for over 30 years, but you don't know me or my knowledge or my level of expertise). Find a recognised networking expert you know and trust, or ask on Stack Exchange or something.

If anyone does need or want to set up a socks proxy or ssh port forwarding or vpn or tunnel or whatever else for whatever weird network configuration they have or need, that's something that can and should be done outside of an application like MythTV. Do one thing and do it well. That one thing for MythTV is to be a great, full featured media scheduling, recording, and playback application. Leave networking in all its weird and wonderful special cases to networking tools - that is their speciality and they will do it far better than MythTV ever can or will. This is out of scope for a media application.

Worse, by using the http_proxy env var in this way, you are making it unnecessarily more difficult for those who do have unusual or complex networking needs. Having to unset http_proxy just to get mythbackend running is certainly making it difficult for those who run myth behind any kind of firewall and need any data-fetching scripts called by myth (e.g. to fetch from IMDB or other video info sources) to use the local web proxy.

[martin_ginkel@…]

@cas

Nice comment! I fully support your explanation, It is totally BS to use any http_proxy for binding a server socket. And I explained the solution 5 years ago.

Best regards Martin

[Peter Bennett <pbennett@…>]

In a1823f3c390c02fb6d5c2c4ff969d811e7d2729d/mythtv:

Refs #10633 - Remove some incorrect logic around IPV6 checking.

[Peter Bennett <pbennett@…>]

In d087884e403d7f9479f6efc07dd8a9d971f996be/mythtv:

Refs #10633 - Remove some incorrect logic around IPV6 checking.

(cherry picked from commit a1823f3c390c02fb6d5c2c4ff969d811e7d2729d)

[Peter Bennett]

You are right, I am not an expert on this. I am just doing my best :)

It looks like MythTV searches the http_proxy environment and uses whatever it finds as application proxy. It seems a mistake to use that for listening. It is valid to use that for http downloads. This code has been there for a long time, so I am reluctant to mess with it. I will consult the other MythTV developers to see whether that should be removed from the listen logic.

[Peter Bennett <pbennett@…>]

In c6efa1b869f5a89497db14a0c48ee5001ca7746b/mythtv:

Refs #10633 - Set default proxy for listening ports to NoProxy?.

The code was using http proxy as the default for listening ports, and
in most cases overriding it to NoProxy?. Http proxies do not make
sense for listening ports, so default to NoProxy? unless the code
specifically requests otherwise when creating the server socket.

[Peter Bennett <pbennett@…>]

In d35436d9405eb31e9e2aeef3c2fad20c6e613da6/mythtv:

Refs #10633 - Set default proxy for listening ports to NoProxy?.

The code was using http proxy as the default for listening ports, and
in most cases overriding it to NoProxy?. Http proxies do not make
sense for listening ports, so default to NoProxy? unless the code
specifically requests otherwise when creating the server socket.

(cherry picked from commit c6efa1b869f5a89497db14a0c48ee5001ca7746b)

[Peter Bennett]

Let us know if this fixes the problem so I can close the ticket