mythmetadatalookup not using proxy

[brian@…]

Using 0.25.3-fixes from 20130104 at 010e576 I'm finding that even though mythmetadatalookup's environment has an "http_proxy=" set, mythmetadatalookup is is ignoring that and still trying to fetch from the Internet directly.

[Raymond Wagner]

'mythmetadatalookup' does not access the internet in any capacity. It calls external grabber scripts which in turn access the internet. Which grabber script in particular are you referring to?

[brian@…]

Replying to wagnerrp:

'mythmetadatalookup' does not access the internet in any capacity. It calls external grabber scripts which in turn access the internet. Which grabber script in particular are you referring to?

That's not what strace is telling me here. I've followed the cloning of process/threads up the tree from the process that is making the direct connections. i.e.

connect(29, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2400:cb00:2048:1::be5d:fc6e", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28 <unfinished ...>

The process tree leading down to 24472 from the initial process is as follows:

32679 <... clone resumed> child_stack=0xae2f83e4, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|C+++ LONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xae2f8ba8, {entry_number:6, base_addr:0xae2f8b40, limit+++ :1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}, child_tidptr=0xae2f8ba+++ 8) = 313
\_313   <... clone resumed> child_stack=0xadaf73e4, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|C+++ LONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xadaf7ba8, {entry_number:6, base_addr:0xadaf7b40, limit+++ :1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}, child_tidptr=0xadaf7ba+++ 8) = 314
  \_314   <... clone resumed> child_stack=0xacaf53e4, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|C+++ LONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xacaf5ba8, {entry_number:6, base_addr:0xacaf5b40, limit+++ :1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}, child_tidptr=0xacaf5ba+++ 8) = 320
    \_320   connect(29, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "2400:cb00:2048:1::be5d:fc6e", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28 <unfinished ...>
      320   <... connect resumed> )           = 0
      ...
      320   write(29, "GET /banners/seasonswide/79349-6"..., 264 <unfinished ...>
      320   <... write resumed> )             = 264

None of those intervening processes invokes an execve() so they presumably are threads. One of those threads very much appears to be doing a direct grab.

[Raymond Wagner]

Can anyone else who uses a proxy confirm this? The way I'm reading this, proxies and the http_proxy attribute should be handled automatically by the Python HTTP libraries.

​http://code.mythtv.org/cgit/mythtv/tree/mythtv/bindings/python/MythTV/ttvdb/tvdb_api.py#n364 ​http://docs.python.org/2/library/urllib2.html#urllib2.build_opener ​http://docs.python.org/2/library/urllib2.html#urllib2.ProxyHandler

Brian. The ttvdb.py script is called by mythmetadatalookup, which is run by mythbackend through the JobQueue?. Are you certain mythbackend is being run with the http_proxy attribute in its environment?

[Raymond Wagner]

Just to clarify, this actually is MML, not the grabber script, and is caused by the image downloads after the grabber script provides the URLs to them.

[brian@…]

Replying to wagnerrp:

Can anyone else who uses a proxy confirm this? The way I'm reading this, proxies and the http_proxy attribute should be handled automatically by the Python HTTP libraries.

Yes, that is my understanding also.

Brian. The ttvdb.py script is called by mythmetadatalookup, which is run by mythbackend through the JobQueue?. Are you certain mythbackend is being run with the http_proxy attribute in its environment?

In my case (which is the case I filed this bug for) I am running from the shell prompt with:

$ env | grep -i proxy
http_proxy=http://proxy:3128/
$ mythmetadatalookup --refresh-all-artwork

I'm happy to carry out any tests to confirm/deny that ttvdb.py is the culprit since I'm not entirely convinced that that tool is the culprit.

[brian@…]

Anything I can do to progress this issue?

[Stuart Auchterlonie]

Raymond, sphery,

is this still an issue, or can we close this out?

[Stuart Auchterlonie]

We have moved all bug tracking to github [1]

If you continue to have this issue, please open a new issue at github, referencing this ticket.

[1] - ​https://github.com/MythTV/mythtv/issues