Opened 6 years ago

Closed 4 years ago

#13187 closed Bug Report - General (Fixed)

HLSPlaylist doesn't follow 302 redirects correctly

Reported by: Steven Ellis <support@…> Owned by: jpoet
Priority: minor Milestone: 31.0
Component: MythTV - HTTP Streaming Version: v29-fixes
Severity: medium Keywords:
Cc: Ticket locked: no

Description

When using HLS to IPTV stream NZ channels the base URL for the playlist re-directs to the correct location via a 302.

This re-direct can't be permanent as the streaming URL changes over time.

The redirected playlist provides the correct TS streams but HLSStream tries to use the original base playlist domain rather than the domain provided via re-direction.

Change History (14)

comment:1 Changed 6 years ago by Steven Ellis <support@…>

Sample M3u8 file for importing channels into MythTV

#EXTM3U

#EXTINF:-1,401 - TVNZ 1
#EXTMYTHTV:xmltvid=tv.9
http://iptv.matthuisman.nz/nz/tv.9.m3u8

#EXTINF:-1,402 - TVNZ 2
#EXTMYTHTV:xmltvid=tv.10
http://iptv.matthuisman.nz/nz/tv.10.m3u8

#EXTINF:-1,403 - Three
#EXTMYTHTV:xmltvid=tv.11
http://iptv.matthuisman.nz/nz/tv.11.m3u8

#EXTINF:-1,411 - The Edge TV
#EXTMYTHTV:xmltvid=tv.51
http://iptv.matthuisman.nz/nz/tv.51.m3u8

#EXTINF:-1,413 - DUKE
#EXTMYTHTV:xmltvid=tv.51
http://iptv.matthuisman.nz/nz/tv.55.m3u8

Tracing a request for the tv.55.m3u8 channel via curl

curl -L -O -v http://iptv.matthuisman.nz/nz/tv.55.m3u8
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 108.61.96.150...
* TCP_NODELAY set
* Connected to iptv.matthuisman.nz (108.61.96.150) port 80 (#0)
> GET /nz/tv.55.m3u8 HTTP/1.1
> Host: iptv.matthuisman.nz
> User-Agent: curl/7.51.0
> Accept: */*
> 
< HTTP/1.1 302 Moved Temporarily
< Server: nginx
< Date: Mon, 04 Dec 2017 02:57:05 GMT
< Content-Type: text/html
< Content-Length: 154
< Connection: keep-alive
< Location: https://tvnzioslive01-i.akamaihd.net/hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8
< 
* Ignoring the response-body
{ [154 bytes data]
* Curl_http_done: called premature == 0
100   154  100   154    0     0   1333      0 --:--:-- --:--:-- --:--:--  1339
* Connection #0 to host iptv.matthuisman.nz left intact
* Issue another request to this URL: 'https://tvnzioslive01-i.akamaihd.net/hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8'
*   Trying 23.1.240.115...
* TCP_NODELAY set
* Connected to tvnzioslive01-i.akamaihd.net (23.1.240.115) port 443 (#1)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* ALPN, server accepted to use http/1.1
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=a248.e.akamai.net,O="Akamai Technologies, Inc.",L=Cambridge,ST=Massachusetts,C=US
* 	start date: Mar 07 00:00:00 2017 GMT
* 	expire date: May 06 23:59:59 2018 GMT
* 	common name: a248.e.akamai.net
* 	issuer: CN=Symantec Class 3 ECC 256 bit SSL CA - G2,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> GET /hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8 HTTP/1.1
> Host: tvnzioslive01-i.akamaihd.net
> User-Agent: curl/7.51.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Length: 4615
< Content-MD5: 9NkSDkUnMPRRl9f4kgttPA==
< Expires: Mon, 04 Dec 2017 02:57:06 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Mon, 04 Dec 2017 02:57:06 GMT
< Connection: keep-alive
< Set-Cookie: _alid_=hrtlgd++ZPIZYbk6+ZqpHg==; path=/hls/live/245926/tvnzhlsingest/duke/; domain=tvnzioslive01-i.akamaihd.net
< Access-Control-Allow-Headers: *
< Access-Control-Expose-Headers: *
< Access-Control-Allow-Methods: GET, HEAD, OPTIONS
< Access-Control-Allow-Origin: *
< Content-Type: application/x-mpegURL
< 
{ [1747 bytes data]
 37  4615   37  1747    0     0   1097      0  0:00:04  0:00:01  0:00:03  1097* Curl_http_done: called premature == 0
100  4615  100  4615    0     0   2899      0  0:00:01  0:00:01 --:--:-- 2800k
* Connection #1 to host tvnzioslive01-i.akamaihd.net left intact

Sample content of https://tvnzioslive01-i.akamaihd.net/hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:236
#EXT-X-KEY:METHOD=AES-128,URI="https://d1g7ajy4kfl5sb.cloudfront.net/duke/K147770180R_5.key",IV=0x000000000000000000000000000000EC
#EXT-X-PROGRAM-DATE-TIME:2017-12-04T02:56:30.000Z
#EXTINF:6.000,
20171204T153256/masterR_5/00000/masterR_5_00236.ts
#EXT-X-KEY:METHOD=AES-128,URI="https://d1g7ajy4kfl5sb.cloudfront.net/duke/K147770180R_5.key",IV=0x000000000000000000000000000000ED
#EXTINF:6.000,
20171204T153256/masterR_5/00000/masterR_5_00237.ts
#EXT-X-KEY:METHOD=AES-128,URI="https://d1g7ajy4kfl5sb.cloudfront.net/duke/K147770180R_5.key",IV=0x000000000000000000000000000000EE
#EXT-X-PROGRAM-DATE-TIME:2017-12-04T02:56:42.000Z
#EXTINF:6.000,
20171204T153256/masterR_5/00000/masterR_5_00238.ts
#EXT-X-KEY:METHOD=AES-128,URI="https://d1g7ajy4kfl5sb.cloudfront.net/duke/K147770180R_5.key",IV=0x000000000000000000000000000000EF
#EXTINF:6.000,
20171204T153256/masterR_5/00000/masterR_5_00239.ts
#EXT-X-KEY:METHOD=AES-128,URI="https://d1g7ajy4kfl5sb.cloudfront.net/duke/K147770180R_5.key",IV=0x000000000000000000000000000000F0
#EXT-X-PROGRAM-DATE-TIME:2017-12-04T02:56:54.000Z
#EXTINF:6.000,
20171204T153256/masterR_5/00000/masterR_5_00240.ts
#EXT-X-KEY:METHOD=AES-128,URI="https://d1g7ajy4kfl5sb.cloudfront.net/duke/K147770240R_5.key",IV=0x000000000000000000000000000000F1
#EXT-X-PROGRAM-DATE-TIME:2017-12-04T02:57:00.000Z
#EXTINF:6.000,
20171204T153256/masterR_5/00000/masterR_5_00241.ts
#EXT-X-KEY:METHOD=AES-128,URI="https://d1g7ajy4kfl5sb.cloudfront.net/duke/K147770240R_5.key",IV=0x000000000000000000000000000000F2
#EXTINF:6.000,
20171204T153256/masterR_5/00000/masterR_5_00242.ts
#EXT-X-KEY:METHOD=AES-128,URI="https://d1g7ajy4kfl5sb.cloudfront.net/duke/K147770240R_5.key",IV=0x000000000000000000000000000000F3
#EXT-X-PROGRAM-DATE-TIME:2017-12-04T02:57:12.000Z
#EXTINF:6.000,
20171204T153256/masterR_5/00000/masterR_5_00243.ts
#EXT-X-KEY:METHOD=AES-128,URI="https://d1g7ajy4kfl5sb.cloudfront.net/duke/K147770240R_5.key",IV=0x000000000000000000000000000000F4
#EXTINF:6.000,
20171204T153256/masterR_5/00000/masterR_5_00244.ts
#EXT-X-KEY:METHOD=AES-128,URI="https://d1g7ajy4kfl5sb.cloudfront.net/duke/K147770240R_5.key",IV=0x000000000000000000000000000000F5
#EXT-X-PROGRAM-DATE-TIME:2017-12-04T02:57:24.000Z

comment:2 Changed 6 years ago by Steven Ellis <support@…>

comment:3 Changed 6 years ago by jpoet

Owner: changed from cpinkham to jpoet
Status: newassigned

comment:4 Changed 6 years ago by Steven Ellis <support@…>

Example Backend Logs showing the re-direction issue

Dec  3 18:08:44 mythtv mythbackend: mythbackend[17649]: I StreamHandler mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/tv.55.m3u8 -> https://tvnzioslive01-i.akamaihd.net/hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8
Dec  3 18:08:44 mythtv mythbackend: mythbackend[17649]: I HLSStream mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/20171121T111356/masterR_5/00058/masterR_5_02928.ts -> https://cdn.rawgit.com/matthuisman/iptv/e17d4e5fb7a3b8b81a2a05bdaf00e44d19ddf252/nz/20171121T111356/masterR_5/00058/masterR_5_02928.ts
Dec  3 18:08:45 mythtv mythbackend: mythbackend[17649]: I HLSPlaylist mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/tv.55.m3u8 -> https://tvnzioslive01-i.akamaihd.net/hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8
Dec  3 18:08:45 mythtv mythbackend: mythbackend[17649]: I HLSStream mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/20171121T111356/masterR_5/00058/masterR_5_02928.ts -> https://cdn.rawgit.com/matthuisman/iptv/e17d4e5fb7a3b8b81a2a05bdaf00e44d19ddf252/nz/20171121T111356/masterR_5/00058/masterR_5_02928.ts

comment:5 Changed 6 years ago by Steven Ellis <support@…>

Currently running a recent Ubuntu build

  • mythtv_2%3a29.0+fixes.20171128.39b2062-0ubuntu0mythbuntu4_all.deb

comment:6 Changed 6 years ago by Steven Ellis <support@…>

For testing outside of NZ can you try the following endpoint for Al Jazeera

Should redirect to something similar to

comment:7 Changed 6 years ago by Steven Ellis <support@…>

Couple of mythdownloadmanager tickets relating to re-directs

  • #12380 Patch for mythdownloadmanager to support relative redirects
  • #12381 Patch for mythdownloadmanager for improved POST redirect compatibility
  • #12384 Feature patch - mythdownloadmanager redirect with original headers

Ticket #12380 appears to be the initial implementation of relative re-direct but is this code path used for the IPTV streams?

comment:8 Changed 6 years ago by Steven Ellis <support@…>

For mythsingledownload we have this patch for re-directs but doesn't deal with relative

  • #12255 Mythsingledownload doesn't support HTTP redirects

So the relative map actually happens in httplivestreambuffer.cpp but it looks like the redirected location of the m3u8 isn't passed back to re-base the relative URLs for the .ts files.

comment:9 Changed 6 years ago by Steven Ellis <support@…>

Some additional debug output of mythtv-backend by modifying /etc/mythtv/additional.args

ADDITIONAL_ARGS="-v playback,record"

Clarified the sequence as follows

Part 1 - fetch m3u8 file - which allows for re-direct rule.

Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent iptvtuningdata.h:235 (IsHLSPlaylist) IsHLSPlaylist url ends with either .m3u8 or .m3u http:/
/iptv.matthuisman.nz/nz/tv.9.m3u8
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/tv.9.m3u8 -> https://t
vnzioslive04-i.akamaihd.net/hls/live/267188/1924997895001/channel1/masterR_55.m3u8
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent channelutil.cpp:2001 (GetIPTVTuningData) Loaded [data]http://iptv.matthuisman.nz/nz/tv.9.m3u8[fectype][fec0][fec1] for 9401
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/hlsstreamhandler.cpp:125 (run) HLSSH(http://iptv.matthuisman.nz/nz/tv.9.m3u8(0)): run() -- begin
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent recorders/hlsstreamhandler.cpp:43 (Get) HLSSH: Creating new stream handler http://iptv.matthuisman.nz/nz/tv.9.m3u8(0) for [data]http://iptv.matthuisman.nz/nz/tv.9.m3u8[fectype][fec0][fec1]

Part 2 - Create the Live TV Buffer

Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4670 (CreateLiveTVRingBuffer) TVRec[48]: CreateLiveTVRingBuffer(401)
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:196 (Throttle) HLSReader: Throttle(false)
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:48 (Open) HLSReader: Opening 'http://iptv.matthuisman.nz/nz/tv.9.m3u8'

Download the playlist

Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I StreamHandler mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/tv.9.m3u8 -> https://tvnzioslive04-i.akamaihd.net/hls/live/267188/1924997895001/channel1/masterR_55.m3u8

Attempt to stream but without the re-direct

Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent recorders/streamhandler.cpp:81 (AddListener) SH(http://iptv.matthuisman.nz/nz/tv.9.m3u8(0)):
 AddListener(0x7f0b44049fd0) -- begin
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent recorders/streamhandler.cpp:93 (AddListener) SH(http://iptv.matthuisman.nz/nz/tv.9.m3u8(0)):
 AddListener(0x7f0b44049fd0) -- locked
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent recorders/streamhandler.cpp:124 (AddListener) SH(http://iptv.matthuisman.nz/nz/tv.9.m3u8(0))
: AddListener(0x7f0b44049fd0) -- end
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent mpeg/mpegstreamdata.cpp:90 (SetDesiredProgram) MPEGStream[48](0x7f0b44049fd0): SetDesiredPro
gram(0)
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:1992 (SetupDTVSignalMonitor) TVRec[48]: Successfully set up MPEG table monitoring.
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(SignalMonitorRunning,) -> RunMainLoop,SignalMonitorRunning, @ tv_rec.cpp:3858
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4462 (ClearFlags) TVRec[48]: ClearFlags(WaitingForSignal,) -> RunMainLoop,SignalMonitorRunning, @ tv_rec.cpp:3859
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(WaitingForSignal,) -> RunMainLoop,WaitingForSignal,SignalMonitorRunning, @ tv_rec.cpp:3864
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(DummyRecorderRunning,) -> RunMainLoop,WaitingForSignal,SignalMonitorRunning,DummyRecorderRunning, @ tv_rec.cpp:3913
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:3914 (TuningFrequency) DummyDTVRecorder -- started
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(RingBufferReady,) -> RunMainLoop,WaitingForSignal,SignalMonitorRunning,DummyRecorderRunning,RingBufferReady, @ tv_rec.cpp:3915
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4462 (ClearFlags) TVRec[48]: ClearFlags(NeedToStartRecorder,) -> RunMainLoop,WaitingForSignal,SignalMonitorRunning,DummyRecorderRunning,RingBufferReady, @ tv_rec.cpp:3925
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(NeedToStartRecorder,) -> RunMainLoop,WaitingForSignal,NeedToStartRecorder,SignalMonitorRunning,DummyRecorderRunning,RingBufferReady, @ tv_rec.cpp:3927
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4049 (TuningSignalCheck) TVRec[48]: TuningSignalCheck: Still waiting.  Will timeout @ 20:30:00.000
Dec  5 20:08:12 mythtv mythbackend: mythbackend[23065]: I ProcessRequest tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(CancelNextRecording,) -> RunMainLoop,CancelNextRecording,WaitingForSignal,NeedToStartRecorder,SignalMonitorRunning,DummyRecorderRunning,RingBufferReady, @ tv_rec.cpp:2682
Dec  5 20:08:13 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:401 (ParseM3U8) HLSReader: Adding new stream 'http://iptv.matthuisman.nz/nz/tv.9.m3u8'
Dec  5 20:08:13 mythtv mythbackend: mythbackend[23065]: I StreamHandler HLS/m3u.cpp:321 (ParseKey) playback of AES-128 encrypted HTTP Live media detected.
Dec  5 20:08:13 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:132 (Open) http://iptv.matthuisman.nz/nz/tv.9.m3u8: Selected stream with 0 bitrate
Dec  5 20:08:13 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:140 (Open) http://iptv.matthuisman.nz/nz/tv.9.m3u8: Open -- end
Dec  5 20:08:13 mythtv mythbackend: mythbackend[23065]: I thread_unknown recorders/HLS/HLSPlaylistWorker.cpp:37 (run) http://iptv.matthuisman.nz/nz/tv.9.m3u8 playlist: run -- begin
Dec  5 20:08:13 mythtv mythbackend: mythbackend[23065]: I thread_unknown recorders/HLS/HLSStreamWorker.cpp:38 (run) http://iptv.matthuisman.nz/nz/tv.9.m3u8 worker: run -- begin
Dec  5 20:08:13 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:196 (Throttle) http://iptv.matthuisman.nz/nz/tv.9.m3u8: Throttle(true)
Dec  5 20:08:13 mythtv mythbackend: mythbackend[23065]: I HLSStream mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/20171202T050147/masterR_55/00017/masterR_55_01244.ts -> https://cdn.rawgit.com/matthuisman/iptv/7aa3714923f88a8fc141e520a5d8a6f58976ac45/nz/20171202T050147/masterR_55/00017/masterR_55_01244.ts
Dec  5 20:08:14 mythtv mythbackend: mythbackend[23065]: E HLSStream recorders/HLS/HLSReader.cpp:936 (DownloadSegmentData) http://iptv.matthuisman.nz/nz/tv.9.m3u8: 52244 failed: Error downloading https://cdn.rawgit.com/matthuisman/iptv/7aa3714923f88a8fc141e520a5d8a6f58976ac45/nz/20171202T050147/masterR_55/00017/masterR_55_01244.ts - server replied: Not Found

Critical file appears to be

Useful docs

My take is that

  • HLSReader::Open(const QString & m3u, int bitrate_index)

Should be called with the re-directed M3U URL rather than the original URL from the channel entry.

Stepping back to the call

  • void HLSStreamHandler::run(void)

This calls

QString url = m_tuning.GetURL(0).toString();


m_hls->Throttle(false);

HLSReader: Throttle


   while (_running_desired)
    {
        if (!m_hls->IsOpen(url))
        {
            if (!m_hls->Open(url, m_tuning.GetBitrate(0)))

So unless the URL has been re-directed before we call Open we're using the wrong Base URL for the streamed content requests further on.

comment:10 Changed 4 years ago by steven-ellis

Looks like the majority of this issue has been resolved under #13488

  • #13488
    IPTV fix for deadlock, handle redirects

comment:11 Changed 4 years ago by steven-ellis

Looks like we've gfot a secondary issue

  • #12486
    IPTV Recorder HLS AES-128 Problem

we work well with clear streams, but have issues with encrypted streams containing

#EXT-X-KEY:METHOD=AES-128

comment:12 Changed 4 years ago by Stuart Auchterlonie

Steven,

Is the remaining issue in this ticket now covered by #13608 ??

Regards Stuart

comment:13 Changed 4 years ago by steven-ellis

Please close this ticket as resolved by #13488. New issue is being tracked under

  • #13608
    IPTV Streaming issues with AES-128 encryption

comment:14 Changed 4 years ago by Stuart Auchterlonie

Milestone: needs_triage31.0
Resolution: Fixed
Status: assignedclosed

Thanks Steve

Note: See TracTickets for help on using tickets.