Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#11358 closed Bug Report - General (fixed)

HDPVR - Green line on 1080i play back

Reported by: bernhart2002@… Owned by: Jim Stichnoth
Priority: minor Milestone: 0.26.1
Component: MythTV - Video Playback Version: 0.26-fixes
Severity: medium Keywords:
Cc: Ticket locked: no

Description

There is a green on the bottom of the screen that looks to be about 1 pixel high when playing back 1080i content captured with a HDPVR. Playing the file back with ffplay, vlc, and mplayer does not have this green line so I believe that this is a mythtv play back bug. Google seaching the internet shows other people have the same problem with no fix outside of adjusting the overscan. I am using VPDAU

here is a cut from the mythfrontend log

2013-01-17 18:24:23.500591 I  TV: Entering main playback loop.
2013-01-17 18:24:23.519765 I  PreviewQueue: Requesting preview for '13350_20130117050000.mpg_0x0_-1s'
2013-01-17 18:24:23.519801 I  PreviewQueue: Requested preview for '13350_20130117050000.mpg_0x0_-1s'
2013-01-17 18:24:23.535740 I  Player(0): Waiting for video buffers...
2013-01-17 18:24:23.558620 I  VidOutVDPAU: Added 2 new buffers. New buffer size 16 (4 decode and 12 process)
2013-01-17 18:24:23.598915 I  VidOutVDPAU: Created VDPAU decoder (4 ref frames)
2013-01-17 18:24:23.629029 I  OSD: Created window bb_OSD_INTERACTIVE
2013-01-17 18:24:23.629066 I  Player(0): Enabled deinterlacing
2013-01-17 18:24:23.667293 I  VDPAU: Added 2 output surfaces (total 4, max 4)
2013-01-17 18:24:23.713066 I  Player(0): Waiting for video buffers...
2013-01-17 18:24:24.085014 I  TV: DoPlayerSeek (29.9966 seconds)
2013-01-17 18:24:24.085042 I  UpdateOSDSeekMessage(Skip Ahead, 2)
2013-01-17 18:24:24.098135 I  AFD: DoFastForward(912 (22), do discard frames)
2013-01-17 18:24:24.098167 I  Dec: DoFastForward(912 (22), do discard frames)
2013-01-17 18:24:24.098184 I  Dec: FindPosition(912, search not adjusted) --> 
			[7:838(37351652),8:966(41864968)]
2013-01-17 18:24:24.098221 I  AFD: SeekReset(966, 0, do flush, do discard)
2013-01-17 18:24:24.098258 I  AFD: SeekReset() flushing
2013-01-17 18:24:24.098282 I  VidOutVDPAU: DiscardFrames(1)
2013-01-17 18:24:24.098322 I  VideoBuffers::DiscardFrames(1): AAAAAUUUUUUUUDDD
2013-01-17 18:24:24.098346 I  VideoBuffers::DiscardFrames(1): AAAAAAAAAAAAADDD -- done
2013-01-17 18:24:24.098361 I  VidOutVDPAU: DiscardFrames() 3: AAAAAAAAAAAAADDD -- done()
2013-01-17 18:24:24.100609 I  Player(0): SetFrameInterval ps:1 scan:3
2013-01-17 18:24:24.100882 I  Player(0): ClearAfterSeek(0)
2013-01-17 18:24:24.100909 I  Player(0): Waiting for video buffers...
2013-01-17 18:24:24.111172 N  VDP: Ignoring profile item 88 (Need a decoder and renderer)
2013-01-17 18:24:24.111331 I  VDP: Accepting: cmp(>= 0 720) dec(vdpau) cpus(2) skiploop(enabled) rend(vdpau) osd(vdpau) osdfade(enabled) deint(vdpauadvanced,vdpaubasic) filt()
2013-01-17 18:24:24.111355 I  VDP: Accepting: cmp(> 0 0) dec(vdpau) cpus(1) skiploop(enabled) rend(vdpau) osd(vdpau) osdfade(enabled) deint(vdpauadvanceddoublerate,vdpauadvanced) filt()
2013-01-17 18:24:24.111368 I  VDP: LoadBestPreferences(2048x2048, 0)
2013-01-17 18:24:24.111388 I  VDP: LoadBestPreferences(2048x2048, 60)
2013-01-17 18:24:24.111401 I  VDP: LoadBestPreferences(1920x1088, 60)
2013-01-17 18:24:24.111433 I  VDP: LoadBestPreferences(1920x1088, 29.97)
2013-01-17 18:24:24.111451 I  VidOutVDPAU: InputChanged(1920,1088,1.76471) 'H.264 VDPAU'->'H.264 VDPAU'
2013-01-17 18:24:24.111881 I  VidOutVDPAU: DiscardFrames(1)
2013-01-17 18:24:24.111911 I  VideoBuffers::DiscardFrames(1): AAAAAAAAAAAAAAAA
2013-01-17 18:24:24.111934 I  VideoBuffers::DiscardFrames(1): AAAAAAAAAAAAAAAA -- done
2013-01-17 18:24:24.111948 I  VidOutVDPAU: DiscardFrames() 3: AAAAAAAAAAAAAAAA -- done()
2013-01-17 18:24:24.203628 N  Player(0): Waited 103ms for video buffers AAAAAAAAAAAAAAAA
2013-01-17 18:24:24.789964 W  VDPAU: Display pre-empted.
2013-01-17 18:24:24.789991 E  VDPAU: Error at mythrender_vdpau.cpp:947 (#2, The display was pre-empted, or a fatal error occurred.)
2013-01-17 18:24:24.790007 N  VDPAU: Attempting to re-create VDPAU resources.
2013-01-17 18:24:24.815885 I  VDPAU: Set colorkey to 0x20202
2013-01-17 18:24:24.818443 I  VDPAU: Re-created output surfaces.
2013-01-17 18:24:24.831740 I  VDPAU: Re-created decoders.
2013-01-17 18:24:24.836153 I  VDPAU: Attempting to reset 8 video surfaces owned by this thread 338428992
2013-01-17 18:24:24.836176 I  VDPAU: Re-created 8 video surfaces.
2013-01-17 18:24:24.836184 I  VDPAU: 0 of 8 video surfaces still need to be reset
2013-01-17 18:24:24.837335 W  MythPainter: 38 images not yet de-allocated.
2013-01-17 18:24:24.837365 I  VDPAU Painter: Clearing VDPAU painter cache.
2013-01-17 18:24:24.861240 I  Clearing OpenGL painter cache.
2013-01-17 18:24:24.861263 I  Snapping height to avoid scaling: height: 1080, top: 0
2013-01-17 18:24:24.861269 I  Snapping width to avoid scaling: width: 1920, left: 0
2013-01-17 18:24:24.861284 I  Display Rect  left: 0, top: 0, width: 1920, height: 1080, aspect: 1.77778
2013-01-17 18:24:24.861293 I  Video Rect    left: 0, top: 0, width: 1920, height: 1080, aspect: 1.76471
2013-01-17 18:24:24.861299 I  Snapping height to avoid scaling: height: 1080, top: 0
2013-01-17 18:24:24.861304 I  Snapping width to avoid scaling: width: 1920, left: 0
2013-01-17 18:24:24.861311 I  Display Rect  left: 0, top: 0, width: 1920, height: 1080, aspect: 1.77778
2013-01-17 18:24:24.861320 I  Video Rect    left: 0, top: 0, width: 1920, height: 1080, aspect: 1.76471
2013-01-17 18:24:24.861325 I  VDP: SetVideoRenderer(vdpau)
2013-01-17 18:24:24.861334 I  VDP: SetVideoRender(vdpau) == GetVideoRenderer()
2013-01-17 18:24:24.862000 I  Doubling refresh rate for interlaced display.
2013-01-17 18:24:24.862073 I  VideoOutput: Pixel dimensions: Screen 1920x1080, window 1920x1080
2013-01-17 18:24:24.862084 I  VideoOutput: Actual display dimensions: 160x90 mm  Aspect: 1.77778
2013-01-17 18:24:24.862442 I  VideoOutput: Estimated window dimensions: 160x90 mm  Aspect: 1.77778
2013-01-17 18:24:24.902731 I  VDPAU: Created 2 output surfaces.
2013-01-17 18:24:24.902753 I  VDPAU: Set colorkey to 0x20202
2013-01-17 18:24:24.902766 I  VDPAU: Created VDPAU render device 1920x1080
2013-01-17 18:24:24.902779 I  VidOutVDPAU: Created VDPAU osd (1920x1080)
2013-01-17 18:24:25.039214 I  ColourSpace: PictureAttributes: Brightness, Contrast, Colour, Hue, Studio Levels, 
2013-01-17 18:24:25.039274 I  Snapping height to avoid scaling: height: 1080, top: 0
2013-01-17 18:24:25.039280 I  Snapping width to avoid scaling: width: 1920, left: 0
2013-01-17 18:24:25.039297 I  Display Rect  left: 0, top: 0, width: 1920, height: 1080, aspect: 1.77778
2013-01-17 18:24:25.039306 I  Video Rect    left: 0, top: 0, width: 1920, height: 1080, aspect: 1.76471
2013-01-17 18:24:25.039311 I  VidOutVDPAU: Created VDPAU context (GPU decode)
2013-01-17 18:24:25.039325 I  VDP: GetFilteredDeint() : vdpau -> 'vdpauadvanced'
2013-01-17 18:24:25.039929 I  VidOutVDPAU: Enabled deinterlacing.
2013-01-17 18:24:25.040821 I  Doubling refresh rate for interlaced display.
2013-01-17 18:24:25.041036 N  Player(0): Forcing decode extra audio option on (Video method requires it).
2013-01-17 18:24:25.041045 I  Player(0): ClearAfterSeek(1)
2013-01-17 18:24:25.041049 I  VidOutVDPAU: ClearAfterSeek()
2013-01-17 18:24:25.041053 I  VidOutVDPAU: DiscardFrames(0)
2013-01-17 18:24:25.041071 I  VideoBuffers::DiscardFrames(0): AAAAA AAAAAAAA
2013-01-17 18:24:25.041084 I  VideoBuffers::DiscardFrames(0): AAAAA AAAAAAAA -- done
2013-01-17 18:24:25.041097 I  VidOutVDPAU: DiscardFrames() 3: AAAAA AAAAAAAA -- done()
2013-01-17 18:24:25.041113 I  Player(0): LoadFilters(''..) -> 0x0
2013-01-17 18:24:25.041135 I  Player(0): detectInterlace(Detect Scan, Interlaced Scan, 29.97, 1088) ->Interlaced Scan
2013-01-17 18:24:25.045431 N  Player(0): Waited 945ms for video buffers AAAAA AAALAAAA
2013-01-17 18:24:25.056733 I  VidOutVDPAU: Added 2 new buffers. New buffer size 16 (4 decode and 12 process)
2013-01-17 18:24:25.074945 I  VidOutVDPAU: Created VDPAU decoder (4 ref frames)
2013-01-17 18:24:25.108801 I  VDPAU: Added 2 output surfaces (total 4, max 4)
2013-01-17 18:24:28.955438 I  Player(0): FPS:   22.67 Mean: 44109 Std.Dev: 114852 CPUs: 100% 100% 
2013-01-17 18:24:32.926038 I  Player(0): FPS:   29.98 Mean: 33360 Std.Dev:   199 CPUs: 34% 100% 
2013-01-17 18:24:36.896831 I  Player(0): FPS:   29.97 Mean: 33362 Std.Dev:   137 CPUs: 34% 100% 
2013-01-17 18:24:38.664858 I  Player(0): 400 interlaced frames seen.
2013-01-17 18:24:40.867400 I  Player(0): FPS:   29.98 Mean: 33360 Std.Dev:   172 CPUs: 37% 100% 
2013-01-17 18:24:44.838076 I  Player(0): FPS:   29.97 Mean: 33361 Std.Dev:   119 CPUs: 33% 100% 
2013-01-17 18:24:48.808753 I  Player(0): FPS:   29.98 Mean: 33361 Std.Dev:   201 CPUs: 33% 100% 
2013-01-17 18:24:52.011575 I  Player(0): 800 interlaced frames seen.
2013-01-17 18:24:52.426749 I  Player(0): Play speed: rate: 29.97 speed: 0 skip: 0 => new interval 33366
2013-01-17 18:24:52.460260 I  VidOutVDPAU: UpdatePauseFrame() AUUDDUUUuUUDLUUU
2013-01-17 18:24:53.764360 N  Preview: RemotePreviewRun() -- no reply..

}}}

Attachments (1)

hdpvr-1088i.patch (1.5 KB) - added by Jim Stichnoth 11 years ago.
Test patch applies to master and fixes/0.26

Download all attachments as: .zip

Change History (26)

comment:1 Changed 11 years ago by Jim Stichnoth

Owner: set to Jim Stichnoth
Status: newaccepted

Can you generate a small sample that demonstrates the problem? If you can trim a recording to 2MB (e.g. using 'dd if=my_recording.mpg of=sample.mpg bs=1024k count=2') and it still demonstrates the problem, it can be directly attached to this ticket.

This is reminiscent of #8409, #8937, #10001.

comment:2 Changed 11 years ago by bernhart2002@…

Hey, after playing around with this a bit I noticed that the greenline does not start until after the 2 second mark. There is a also a quick flash to black then the mostly green line starts. the 2mb file is too short for this as it is ~ 1 second long.

I have been able to make a sample file that can re create this problem however it is 8.3mb. If you email me directly I can send it to you. bernhart2002 at gmail

comment:3 Changed 11 years ago by bernhart2002@…

regarding the other tickets. The height of the green line seems to be only 1 pixel high vs the 8 pixel high in the other tickets. Also playing 1080p content does not produce this green line... however that was not captured from the hdpvr. But 720p from the hdpvr does not have a green line

comment:4 Changed 11 years ago by kevin@…

I'm getting the same problem, and I have a 10 MB clip that demonstrates the problem. The first 2 seconds are okay, then it flashes black, then the line on the bottom. FWIW, I'm using VDPAU High Quality playback profile.

Here's the clip:

http://download.statuspro.tv/test.mpg

Hope this helps! -- Kevin

Changed 11 years ago by Jim Stichnoth

Attachment: hdpvr-1088i.patch added

Test patch applies to master and fixes/0.26

comment:5 Changed 11 years ago by Jim Stichnoth

I uploaded a patch that applies to both master and fixes/0.26. If you are affected by this problem and in a position to test patches, I would appreciate feedback.

comment:6 Changed 11 years ago by kevin@…

I just tested this, and the garbage at the bottom is now gone. Since the fix involves scaling, does it reduce the video quality?

There's still a glitch at the 2-second mark, but I'll have to file that as a separate bug (if it isn't already filed).

Thanks!

Kevin

comment:7 Changed 11 years ago by Jim Stichnoth

Unless you have a 1088-row display, there was already scaling going on. The only quality difference I notice is that some legitimate lines are now being truncated. You can tell if you enable Manual Zoom mode while paused and shift the image up and down. Changing the 1080 constant in the patch to 1087 still leaves a hint of a green line, and changing it to 1086 removes the green line. I may use 1084 in the final commit to be safe.

The glitch at the 2-second mark appears to be related to loading the OSD, judging from log messages.

Thanks for testing this.

comment:8 Changed 11 years ago by Jim Stichnoth <jstichnoth@…>

In 9d9de7b75d5354c9130526311a0c74d7172ebe6d/mythtv:

Rescale 1088-row videos to avoid showing trailing garbage.

Refs #11358. 1080i videos may present themselves as 1088 rows
(divisible by 16). The HD-PVR in particular may contain garbage in
those last rows. Specifically, the last two rows may form an annoying
green line. We add a scaling pass to shift the bottom 4 rows (4 to be
on the safe side) off the bottom of the display area.

comment:9 Changed 11 years ago by Jim Stichnoth <jstichnoth@…>

In 3587b8240a5b136df1b454b6b4e2d44294e7ed0a/mythtv:

Rescale 1088-row videos to avoid showing trailing garbage.

Refs #11358. 1080i videos may present themselves as 1088 rows
(divisible by 16). The HD-PVR in particular may contain garbage in
those last rows. Specifically, the last two rows may form an annoying
green line. We add a scaling pass to shift the bottom 4 rows (4 to be
on the safe side) off the bottom of the display area.
(cherry picked from commit 9d9de7b75d5354c9130526311a0c74d7172ebe6d)

comment:10 Changed 11 years ago by Jim Stichnoth

Milestone: 0.26.1
Resolution: Fixed
Status: acceptedclosed

comment:11 Changed 11 years ago by kevin@…

I tried this on some TV's and monitors that aren't 1920x1080, for example my 1920x1200 computer monitor. The garbage lines are still there on those monitors.

comment:12 Changed 11 years ago by Jim Stichnoth

Resolution: Fixed
Status: closednew

I also see this if I run with e.g. "-geometry 640x480".

comment:13 Changed 11 years ago by Jim Stichnoth

Status: newaccepted

comment:14 Changed 11 years ago by Jim Stichnoth <jstichnoth@…>

In c0bc92822fe63221ad8fc5cf52a6e661b37b6f21/mythtv:

Improved cropping of the bottom 8 lines of 1088-line video.

Refs #11358. Reverts 9d9de7b75d5354c9130526311a0c74d7172ebe6d and
cleans up the bottom 8 lines by applying a crop filter to the source.
This is tested for the XVideo output method.

The VDPAU output method does not use the standard filter mechanism,
but it works fine on 1088-line video until the h.264 parser thinks the
resolution has changed from 1080 to 1088 lines. To compensate, we
don't acknowledge a resolution change when this happens.

comment:15 Changed 11 years ago by kevin@…

Not only does this fix the problem of the garbage lines at the bottom of the screen, it also fixes the problem with the glitch at the 2-second mark.

I suspect the resolution was changing at the 2-second mark, causing the glitch, since that's when the garbage would appear, not before. This is just speculation on my part, of course.

Thanks for the fix!

-- Kevin

comment:16 Changed 11 years ago by kevin@…

Had I read your note a little more carefully, I would have seen that you said pretty much the same thing as me. :)

Thanks!

-- Kevin

comment:17 Changed 11 years ago by Jim Stichnoth <jstichnoth@…>

In 64f0e4ec24eb52bbbd4318365b3252f9b462ade1/mythtv:

Improved cropping of the bottom 8 lines of 1088-line video.

Refs #11358. Reverts 9d9de7b75d5354c9130526311a0c74d7172ebe6d and
cleans up the bottom 8 lines by applying a crop filter to the source.
This is tested for the XVideo output method.

The VDPAU output method does not use the standard filter mechanism,
but it works fine on 1088-line video until the h.264 parser thinks the
resolution has changed from 1080 to 1088 lines. To compensate, we
don't acknowledge a resolution change when this happens.
(cherry picked from commit c0bc92822fe63221ad8fc5cf52a6e661b37b6f21)

Note: the hard-coding of 1088->1080 is not the best way to handle
this. It will be cleaned up in Master.

comment:18 Changed 11 years ago by Jim Stichnoth <jstichnoth@…>

In cc7d7749e38eb3c42edf3403aedf05c82e2e2f3b/mythtv:

Detect the cropped picture size in the h.264 parser.

This removes one instance of hardcoded 1088->1080 translation.
Refs #11358.

comment:19 Changed 11 years ago by Jim Stichnoth <jstichnoth@…>

Resolution: fixed
Status: acceptedclosed

In 0a4e3e423d6604252aeb31f849085a00f9f4b2e9/mythtv:

Fixes #11358. Use dimension and cropping information from the video.

This helps avoid hard-coding of 1088->1080 truncation and manual
rounding up to a multiple of 16. There are still many instances in
the code base, but this is progress.

Cropping in the horizontal dimension is not yet implemented.

There is still some funny business with slightly changing aspect
ratios that needs further investigation.

comment:20 Changed 11 years ago by mark.kendall@…

Jim

This is a mess.

  • h264 streams do not reliably provide cropping information.
  • other codecs can and do have 1088 line video. There are plenty of mpeg2 samples in the wild. Relying solely on h264 stream parsing isn't going to work and afaik, mpeg2, for example, does not give you any cropping hints.
  • the video output code has accounted for the 1088/1080 issue for years. The bottom 8 lines are not displayed so applying a crop filter is a waste of time (not least because the issue is only reported with VDPAU, which you haven't dealt with).

You may have stumbled upon a fix somewhere that is picking up on a 1080/1088 transition, but fundamentally this looks like a VDPAU interlacing issue:-

  • it doesn't happen with progressive content
  • it's one green line - suggesting the deinterlacer is referencing memory outside of the defined 'frame' area.
  • the blank screen/flash is a classic symptom of display pre-emption in the VDPAU library - which is supported by the original log.
  • the pre-emption could be triggered by the 1080/1088 change (and hence may have been inadvertently fixed somewhere along the line)

comment:21 Changed 11 years ago by Jim Stichnoth

Mark, I appreciate your comments.

Are you aware that the sample in comment:4 caused the garbage lines for both VDPAU and Xv? It may very well be related to deinterlacing, but it's not specific to VDPAU.

The momentary blank screen happened when the H264Parser determined that the resolution had changed from 1920x1080 to 1920x1088, thereby reopening the codec (due to an alleged ffmpeg multithreaded decoding bug), reinitializing the OSD, and whatever cascaded from there - possibly including display preemption. I believe 1920x1080 came from the stream or the codec, whereas 1920x1088 came from the h.264 packet without applying the cropping values.

Do you have more information about the unreliability of h.264 cropping information?

Although mpeg2 probably lacks cropping hints, the sequence header seems to be reporting 1920x1080 which is used as the display dimensions, while the buffer dimensions are explicitly rounded up to multiples of 16 (1920x1088), making everything work out. I'm sure there are broken videos in the wild, and I would be interested in getting my hands on some samples, but my primary concern is whether current or past MythTV recorders had this problem.

As for the crop filter, I realize that VDPAU does not invoke it, but without the filter, Xv still displays the garbage if the display aspect ratio is less than 16:9, such as a 4:3 or 16:10 display where letterboxing is added.

If explicit 1088->1080 translations were required, I would expect to see evidence of this in the ffmpeg code, but I failed to find it. Any idea if there is some special handling that I am missing?

comment:22 Changed 11 years ago by mark.kendall@…

Jim

This is the first 10mb of one of my h264 streams. It is reported as 1088 and appears to have no cropping information in the sps:-

https://dl.dropbox.com/u/68287086/anixe.ts

For VC1, the ticket you already mentioned:-

http://code.mythtv.org/trac/ticket/8409

references VC1 1088 streams, though I still haven't seen an actual VC1 clip that demonstrated the problem and there is no VC1 stream parser anyway - so if cropping detail is available, there would have to be a whole new parser to handle it.

I've started digging through my test clips for mpeg2 examples but haven't found anything yet.

But fundamentally, I don't think you can avoid the 1080/8 issue. The correct detail may be available with some streams but if you can't rely on it, then it is next to useless.

So personally I would take the pre-existing code, try and reduce/remove any duplicated handling of 1080/8 adjustments and ensure that the ffmpeg and h264 parser output are consistent (hence avoiding unnecessary video re-initialisation).

ffmpeg won't be too concerned with the problem as it is a parsing/decoding library and doesn't handle display

I do think, however, that it is largely a red herring in this case - or at most one factor...

If you turn off deinterlacing, you won't see the noise/garbage at the bottom. So I would look for regressions in that area (pretty hard to role back to code from 2-3 years ago, but I'm pretty certain this was all working as intended previously).

comment:23 Changed 11 years ago by Jim Stichnoth <jstichnoth@…>

In 7d2ad07e3553bb224f969517fe06c13addab961c/mythtv:

Detect the cropped picture size in the h.264 parser.

This removes one instance of hardcoded 1088->1080 translation.
Refs #11358.

comment:24 Changed 11 years ago by Jim Stichnoth <jstichnoth@…>

In d59a3fd9e092338b3d3759b0b322e5bfd1cddb6e/mythtv:

Fixes #11358. Use dimension and cropping information from the video.

This helps avoid hard-coding of 1088->1080 truncation and manual
rounding up to a multiple of 16. There are still many instances in
the code base, but this is progress.

Cropping in the horizontal dimension is not yet implemented.

There is still some funny business with slightly changing aspect
ratios that needs further investigation.

comment:25 Changed 11 years ago by Jim Stichnoth

Mark,

I concur that the anixe.ts sample doesn't contain cropping and yet displays without garbage at the bottom. I assume that is either because it is encoded cleanly, or that it is progressive and not hitting potential deinterlacing bugs. The HD-PVR sample still displays the garbage line when using deinterlacer=None, but is clean when setting scan type to Progressive. That definitely needs some investigation.

There is some more holistic cropping going on in VideoOutWindow::ApplySnapToVideoRect?(), but this only helps when the window aspect ratio is equal to the intended video aspect ratio.

Assuming explicit 1088->1080 translations are necessary, I agree it should be factored out into a single location.

Note: See TracTickets for help on using tickets.