Voctomix

Full-Stack OpenSource HD-Capable Live-Video-Mixer

 

MaZderMind

& the C3VOC

C3VOC Usual Setup

?

?

Professional Hardware

"StudioTech 110 - Blackmagic ATEM 4K switchers: Part 2 of 3 ATEM Production Studio 4K" by "StudioTech TV" - Creative Commons-Lizenz mit Quellenangabe

Professional Software?

"StudioTech 111 - Blackmagic ATEM 4K switchers: Part 3 of 3 ATEM 1 M/E Production Studio 4K" by "StudioTech TV" - Creative Commons-Lizenz mit Quellenangabe

  • OpenSource
  • Software
  • Commodity Hardware

Replicable

GStreamer

to the rescue!!!

GStreamer

"a pipeline-based multimedia framework"

gst-launch-1.0 \
    videotestsrc ! \
    video/x-raw,width=1024,height=576 ! \
    ximagesink
  • Generate Test-Video
  • Fixate some Parameters (width & height)
  • Let some be auto-negotiated (framerate, format)
  • Pipe into X-Server Sink
  • Sink does the scheduling

GStreamer

plugin introspection

gst-inspect-1.0 videotestsrc

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { I420, YV12, YUY2, UYVY, AYUV, RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR, Y41B, Y42B, YVYU, Y444, v210, v216, NV12, NV21, NV16, NV61, NV24, GRAY8, GRAY16_BE, GRAY16_LE, v308, RGB16, BGR16, RGB15, BGR15, UYVP, A420, RGB8P, YUV9, YVU9, IYU1, ARGB64, AYUV64, r210, I420_10LE, I420_10BE, I422_10LE, I422_10BE, Y444_10LE, Y444_10BE, GBR, GBR_10LE, GBR_10BE, NV12_64Z32, A420_10LE, A420_10BE, A422_10LE, A422_10BE, A444_10LE, A444_10BE }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-bayer
                 format: { bggr, rggb, grbg, gbrg }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]


Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "videotestsrc0"
  pattern             : Type of test pattern to generate
                        flags: readable, writable
                        Enum "GstVideoTestSrcPattern" Default: 0, "smpte"
                           (0): smpte            - SMPTE 100% color bars
                           (1): snow             - Random (television snow)
                           (2): black            - 100% Black
                           (3): white            - 100% White
…

GStreamer

complex pipeline

gst-launch-1.0 --eos-on-shutdown \
    videotestsrc ! \
        video/x-raw,width=1024,height=576,framerate=25/1,format=UYVY ! \
        videoconvert ! \
        x264enc bitrate=4000 ! \
        queue ! \
        mux. \
    \
    audiotestsrc ! \
        audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
        lamemp3enc quality=6 ! \
        queue ! \
        mux. \
    \
    mp4mux faststart=false name=mux ! \
        filesink location=lala.mp4

GStreamer

compositor

gst-launch-1.0 --eos-on-shutdown \
    videotestsrc pattern=smpte ! \
        video/x-raw,width=1024,height=576,framerate=25/1,format=UYVY ! \
        queue ! \
        mix. \
    \
    videotestsrc pattern=ball ! \
        video/x-raw,width=1024,height=576,framerate=25/1,format=UYVY ! \
        queue ! \
        mix. \
    \
    compositor name=mix \
            sink_1::xpos=50 sink_1::ypos=50 \
            sink_1::width=400 sink_1::height=225 \
            sink_1::zindex=1 sink_1::alpha=0.75 ! \
        videoconvert ! \
        ximagesink

Reliability of Sources

 

  • Sources are unreliable. Period.
    • Cables can be unplugged
    • Power can be shut down
    • Requirements can Change
  • A GStreamer-Pipeline is a static construct
  • Dynamic changes are possible but complicated

 

Blocking Barriers

  • inter*-Elements
  • Works like a clutch
  • F*cks up timestamps
  • Screws up A/V Sync

Composition

Fullscreen

Picture in Picture

Side by Side Preview

Side by Side Preview

IOs & Flows

IOs & Flows

17000… VSource** (Stream-Blanker) ---\
18000  ASource** (Stream-Blanker) ----\
                                       \
16000 VSource (Background)              \
                      \                  \
                       --> VideoMix       \
                      /             \      -> StreamBlanker** -> StreamOutputPort** 15000
                     /               \    /
                    /                 ------> OutputPort 11000
                   /                 /    \-> Encoder* -> PreviewPort* 12000
                  /                 /
                 /----- -> AudioMix
                /
10000… AVSource --> MirrorPort 13000…
                \-> Encoder* -> PreviewPort* 14000…

9999 Control-Server
9998 GstNetTimeProvider Network-Clock

IOs & Flows

  • Core only mixes Video
  • Sources and Sinks are external Scripts
    • Camera-Source
    • Slide-Source
    • Recording-Sink
    • Streaming-Sink
    • Lots of Options!
  • GUI is External
    on same System or Notebook

IOs & Flows

  • Raw I420 + Raw S16LE in Matroska
    (for Timing and Metadata)
  • via TCP ~791 MBit/s
    loopback or dedicated NIC (or 10GE)
    For Streaming or Recording
  • Preview-Outs: JPEG + Raw S16LE in Matroska
    ~30-50 MBit/s
    For GUI

Control Protocol

Control Protocol

< set_video_a cam1
> video_status cam1 cam2

< set_composite_mode side_by_side_equal
> composite_mode side_by_side_equal

< get_video
> video_status cam1 cam2


< set_stream_blank pause
> stream_status blank pause


… meanwhile in another control-server connection

> video_status cam1 cam2
> composite_mode side_by_side_equal
> stream_status blank pause

Control Protocol

 

  • rlwrap telnet
  • nc
  • http / html?
  • scriptable!

 

The GUI

…could be better

State and Future Plans

Used by

Future Plans

  • Stability & Reliability
  • Cropping slide sources
  • GUI Improvements
  • f.lux Compensator
  • Performance
    • Put stuff onto the GPU
  • Documentation
  • Improveme 1st Impression

Code

https://github.com/voc/voctomix

 

https://github.com/voc/voctomix/tree/master/voctocore

https://github.com/voc/voctomix/tree/master/voctogui

https://github.com/voc/voctomix/tree/master/example-scripts

Run

 

Recent Version of GStreamer (1.6 and later)

Debian Stretch or Sid or Ubuntu Wily

 

or run a Docker

or build your own (uninstalled) GStreamer

 

https://packages.debian.org/stretch/voctomix

http://packages.ubuntu.com/yakkety/voctomix

https://hub.docker.com/r/c3voc/voctomix/

 

Talk

Questions?

froscon12-voctomix

By mazdermind

froscon12-voctomix

My talk about Voctomix for FrOSCon12

  • 691