Opened 7 years ago
Closed 5 years ago
#13187 closed Bug Report - General (Fixed)
HLSPlaylist doesn't follow 302 redirects correctly
| Reported by: | Owned by: | jpoet | |
|---|---|---|---|
| Priority: | minor | Milestone: | 31.0 |
| Component: | MythTV - HTTP Streaming | Version: | v29-fixes |
| Severity: | medium | Keywords: | |
| Cc: | Ticket locked: | no |
Description
When using HLS to IPTV stream NZ channels the base URL for the playlist re-directs to the correct location via a 302.
This re-direct can't be permanent as the streaming URL changes over time.
The redirected playlist provides the correct TS streams but HLSStream tries to use the original base playlist domain rather than the domain provided via re-direction.
Change History (14)
comment:1 Changed 7 years ago by
comment:2 Changed 7 years ago by
MythTV is appending TS Component of Playlist to the wrong base URL
Initial base URL
ReDirected? Base URL
Appended TS location
- 20171204T153256/masterR_5/00000/masterR_5_00236.ts
Correct URL should be
comment:3 Changed 7 years ago by
| Owner: | changed from cpinkham to jpoet |
|---|---|
| Status: | new → assigned |
comment:4 Changed 7 years ago by
Example Backend Logs showing the re-direction issue
Dec 3 18:08:44 mythtv mythbackend: mythbackend[17649]: I StreamHandler mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/tv.55.m3u8 -> https://tvnzioslive01-i.akamaihd.net/hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8 Dec 3 18:08:44 mythtv mythbackend: mythbackend[17649]: I HLSStream mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/20171121T111356/masterR_5/00058/masterR_5_02928.ts -> https://cdn.rawgit.com/matthuisman/iptv/e17d4e5fb7a3b8b81a2a05bdaf00e44d19ddf252/nz/20171121T111356/masterR_5/00058/masterR_5_02928.ts Dec 3 18:08:45 mythtv mythbackend: mythbackend[17649]: I HLSPlaylist mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/tv.55.m3u8 -> https://tvnzioslive01-i.akamaihd.net/hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8 Dec 3 18:08:45 mythtv mythbackend: mythbackend[17649]: I HLSStream mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/20171121T111356/masterR_5/00058/masterR_5_02928.ts -> https://cdn.rawgit.com/matthuisman/iptv/e17d4e5fb7a3b8b81a2a05bdaf00e44d19ddf252/nz/20171121T111356/masterR_5/00058/masterR_5_02928.ts
comment:5 Changed 7 years ago by
Currently running a recent Ubuntu build
- mythtv_2%3a29.0+fixes.20171128.39b2062-0ubuntu0mythbuntu4_all.deb
comment:6 Changed 7 years ago by
For testing outside of NZ can you try the following endpoint for Al Jazeera
Should redirect to something similar to
comment:7 Changed 7 years ago by
Couple of mythdownloadmanager tickets relating to re-directs
- #12380 Patch for mythdownloadmanager to support relative redirects
- #12381 Patch for mythdownloadmanager for improved POST redirect compatibility
- #12384 Feature patch - mythdownloadmanager redirect with original headers
Ticket #12380 appears to be the initial implementation of relative re-direct but is this code path used for the IPTV streams?
comment:8 Changed 7 years ago by
For mythsingledownload we have this patch for re-directs but doesn't deal with relative
- #12255 Mythsingledownload doesn't support HTTP redirects
So the relative map actually happens in httplivestreambuffer.cpp but it looks like the redirected location of the m3u8 isn't passed back to re-base the relative URLs for the .ts files.
comment:9 Changed 7 years ago by
Some additional debug output of mythtv-backend by modifying /etc/mythtv/additional.args
ADDITIONAL_ARGS="-v playback,record"
Clarified the sequence as follows
Part 1 - fetch m3u8 file - which allows for re-direct rule.
Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent iptvtuningdata.h:235 (IsHLSPlaylist) IsHLSPlaylist url ends with either .m3u8 or .m3u http:/ /iptv.matthuisman.nz/nz/tv.9.m3u8 Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/tv.9.m3u8 -> https://t vnzioslive04-i.akamaihd.net/hls/live/267188/1924997895001/channel1/masterR_55.m3u8 Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent channelutil.cpp:2001 (GetIPTVTuningData) Loaded [data]http://iptv.matthuisman.nz/nz/tv.9.m3u8[fectype][fec0][fec1] for 9401 Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/hlsstreamhandler.cpp:125 (run) HLSSH(http://iptv.matthuisman.nz/nz/tv.9.m3u8(0)): run() -- begin Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent recorders/hlsstreamhandler.cpp:43 (Get) HLSSH: Creating new stream handler http://iptv.matthuisman.nz/nz/tv.9.m3u8(0) for [data]http://iptv.matthuisman.nz/nz/tv.9.m3u8[fectype][fec0][fec1]
Part 2 - Create the Live TV Buffer
Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4670 (CreateLiveTVRingBuffer) TVRec[48]: CreateLiveTVRingBuffer(401) Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:196 (Throttle) HLSReader: Throttle(false) Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:48 (Open) HLSReader: Opening 'http://iptv.matthuisman.nz/nz/tv.9.m3u8'
Download the playlist
Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I StreamHandler mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/tv.9.m3u8 -> https://tvnzioslive04-i.akamaihd.net/hls/live/267188/1924997895001/channel1/masterR_55.m3u8
Attempt to stream but without the re-direct
Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent recorders/streamhandler.cpp:81 (AddListener) SH(http://iptv.matthuisman.nz/nz/tv.9.m3u8(0)): AddListener(0x7f0b44049fd0) -- begin Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent recorders/streamhandler.cpp:93 (AddListener) SH(http://iptv.matthuisman.nz/nz/tv.9.m3u8(0)): AddListener(0x7f0b44049fd0) -- locked Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent recorders/streamhandler.cpp:124 (AddListener) SH(http://iptv.matthuisman.nz/nz/tv.9.m3u8(0)) : AddListener(0x7f0b44049fd0) -- end Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent mpeg/mpegstreamdata.cpp:90 (SetDesiredProgram) MPEGStream[48](0x7f0b44049fd0): SetDesiredPro gram(0) Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:1992 (SetupDTVSignalMonitor) TVRec[48]: Successfully set up MPEG table monitoring. Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(SignalMonitorRunning,) -> RunMainLoop,SignalMonitorRunning, @ tv_rec.cpp:3858 Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4462 (ClearFlags) TVRec[48]: ClearFlags(WaitingForSignal,) -> RunMainLoop,SignalMonitorRunning, @ tv_rec.cpp:3859 Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(WaitingForSignal,) -> RunMainLoop,WaitingForSignal,SignalMonitorRunning, @ tv_rec.cpp:3864 Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(DummyRecorderRunning,) -> RunMainLoop,WaitingForSignal,SignalMonitorRunning,DummyRecorderRunning, @ tv_rec.cpp:3913 Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:3914 (TuningFrequency) DummyDTVRecorder -- started Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(RingBufferReady,) -> RunMainLoop,WaitingForSignal,SignalMonitorRunning,DummyRecorderRunning,RingBufferReady, @ tv_rec.cpp:3915 Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4462 (ClearFlags) TVRec[48]: ClearFlags(NeedToStartRecorder,) -> RunMainLoop,WaitingForSignal,SignalMonitorRunning,DummyRecorderRunning,RingBufferReady, @ tv_rec.cpp:3925 Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(NeedToStartRecorder,) -> RunMainLoop,WaitingForSignal,NeedToStartRecorder,SignalMonitorRunning,DummyRecorderRunning,RingBufferReady, @ tv_rec.cpp:3927 Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I TVRecEvent tv_rec.cpp:4049 (TuningSignalCheck) TVRec[48]: TuningSignalCheck: Still waiting. Will timeout @ 20:30:00.000 Dec 5 20:08:12 mythtv mythbackend: mythbackend[23065]: I ProcessRequest tv_rec.cpp:4453 (SetFlags) TVRec[48]: SetFlags(CancelNextRecording,) -> RunMainLoop,CancelNextRecording,WaitingForSignal,NeedToStartRecorder,SignalMonitorRunning,DummyRecorderRunning,RingBufferReady, @ tv_rec.cpp:2682 Dec 5 20:08:13 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:401 (ParseM3U8) HLSReader: Adding new stream 'http://iptv.matthuisman.nz/nz/tv.9.m3u8' Dec 5 20:08:13 mythtv mythbackend: mythbackend[23065]: I StreamHandler HLS/m3u.cpp:321 (ParseKey) playback of AES-128 encrypted HTTP Live media detected. Dec 5 20:08:13 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:132 (Open) http://iptv.matthuisman.nz/nz/tv.9.m3u8: Selected stream with 0 bitrate Dec 5 20:08:13 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:140 (Open) http://iptv.matthuisman.nz/nz/tv.9.m3u8: Open -- end Dec 5 20:08:13 mythtv mythbackend: mythbackend[23065]: I thread_unknown recorders/HLS/HLSPlaylistWorker.cpp:37 (run) http://iptv.matthuisman.nz/nz/tv.9.m3u8 playlist: run -- begin Dec 5 20:08:13 mythtv mythbackend: mythbackend[23065]: I thread_unknown recorders/HLS/HLSStreamWorker.cpp:38 (run) http://iptv.matthuisman.nz/nz/tv.9.m3u8 worker: run -- begin Dec 5 20:08:13 mythtv mythbackend: mythbackend[23065]: I StreamHandler recorders/HLS/HLSReader.cpp:196 (Throttle) http://iptv.matthuisman.nz/nz/tv.9.m3u8: Throttle(true) Dec 5 20:08:13 mythtv mythbackend: mythbackend[23065]: I HLSStream mythsingledownload.cpp:75 (DownloadURL) http://iptv.matthuisman.nz/nz/20171202T050147/masterR_55/00017/masterR_55_01244.ts -> https://cdn.rawgit.com/matthuisman/iptv/7aa3714923f88a8fc141e520a5d8a6f58976ac45/nz/20171202T050147/masterR_55/00017/masterR_55_01244.ts Dec 5 20:08:14 mythtv mythbackend: mythbackend[23065]: E HLSStream recorders/HLS/HLSReader.cpp:936 (DownloadSegmentData) http://iptv.matthuisman.nz/nz/tv.9.m3u8: 52244 failed: Error downloading https://cdn.rawgit.com/matthuisman/iptv/7aa3714923f88a8fc141e520a5d8a6f58976ac45/nz/20171202T050147/masterR_55/00017/masterR_55_01244.ts - server replied: Not Found
Critical file appears to be
Useful docs
My take is that
- HLSReader::Open(const QString & m3u, int bitrate_index)
Should be called with the re-directed M3U URL rather than the original URL from the channel entry.
Stepping back to the call
- void HLSStreamHandler::run(void)
This calls
QString url = m_tuning.GetURL(0).toString();
m_hls->Throttle(false);
HLSReader: Throttle
while (_running_desired)
{
if (!m_hls->IsOpen(url))
{
if (!m_hls->Open(url, m_tuning.GetBitrate(0)))
So unless the URL has been re-directed before we call Open we're using the wrong Base URL for the streamed content requests further on.
comment:10 Changed 5 years ago by
comment:11 Changed 5 years ago by
Looks like we've gfot a secondary issue
- #12486
- IPTV Recorder HLS AES-128 Problem
we work well with clear streams, but have issues with encrypted streams containing
#EXT-X-KEY:METHOD=AES-128
comment:12 Changed 5 years ago by
comment:13 Changed 5 years ago by
comment:14 Changed 5 years ago by
| Milestone: | needs_triage → 31.0 |
|---|---|
| Resolution: | → Fixed |
| Status: | assigned → closed |
Thanks Steve
Note: See
TracTickets for help on using
tickets.
Sample M3u8 file for importing channels into MythTV
Tracing a request for the tv.55.m3u8 channel via curl
curl -L -O -v http://iptv.matthuisman.nz/nz/tv.55.m3u8 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 108.61.96.150... * TCP_NODELAY set * Connected to iptv.matthuisman.nz (108.61.96.150) port 80 (#0) > GET /nz/tv.55.m3u8 HTTP/1.1 > Host: iptv.matthuisman.nz > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 302 Moved Temporarily < Server: nginx < Date: Mon, 04 Dec 2017 02:57:05 GMT < Content-Type: text/html < Content-Length: 154 < Connection: keep-alive < Location: https://tvnzioslive01-i.akamaihd.net/hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8 < * Ignoring the response-body { [154 bytes data] * Curl_http_done: called premature == 0 100 154 100 154 0 0 1333 0 --:--:-- --:--:-- --:--:-- 1339 * Connection #0 to host iptv.matthuisman.nz left intact * Issue another request to this URL: 'https://tvnzioslive01-i.akamaihd.net/hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8' * Trying 23.1.240.115... * TCP_NODELAY set * Connected to tvnzioslive01-i.akamaihd.net (23.1.240.115) port 443 (#1) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * ALPN, server accepted to use http/1.1 * SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=a248.e.akamai.net,O="Akamai Technologies, Inc.",L=Cambridge,ST=Massachusetts,C=US * start date: Mar 07 00:00:00 2017 GMT * expire date: May 06 23:59:59 2018 GMT * common name: a248.e.akamai.net * issuer: CN=Symantec Class 3 ECC 256 bit SSL CA - G2,OU=Symantec Trust Network,O=Symantec Corporation,C=US > GET /hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8 HTTP/1.1 > Host: tvnzioslive01-i.akamaihd.net > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 200 OK < Content-Length: 4615 < Content-MD5: 9NkSDkUnMPRRl9f4kgttPA== < Expires: Mon, 04 Dec 2017 02:57:06 GMT < Cache-Control: max-age=0, no-cache, no-store < Pragma: no-cache < Date: Mon, 04 Dec 2017 02:57:06 GMT < Connection: keep-alive < Set-Cookie: _alid_=hrtlgd++ZPIZYbk6+ZqpHg==; path=/hls/live/245926/tvnzhlsingest/duke/; domain=tvnzioslive01-i.akamaihd.net < Access-Control-Allow-Headers: * < Access-Control-Expose-Headers: * < Access-Control-Allow-Methods: GET, HEAD, OPTIONS < Access-Control-Allow-Origin: * < Content-Type: application/x-mpegURL < { [1747 bytes data] 37 4615 37 1747 0 0 1097 0 0:00:04 0:00:01 0:00:03 1097* Curl_http_done: called premature == 0 100 4615 100 4615 0 0 2899 0 0:00:01 0:00:01 --:--:-- 2800k * Connection #1 to host tvnzioslive01-i.akamaihd.net left intactSample content of https://tvnzioslive01-i.akamaihd.net/hls/live/245926/tvnzhlsingest/duke/masterR_5.m3u8