FFglitch:

the multimedia
bitstream editor

FOSDEM 2025

Ramiro Polla

FFmpeg + glitch art
=
FFglitch

Common glitch art examples in the wild:

  • SD card corruption
  • Scratched DVDs
  • DVB-T reception issues
  • Streaming connectivity issues
  • Mostly data corruption
    in some way or another...
  • ... that create visual artifacts

JPEG

SD card (file system) corruption

JPEG 2000

ePirat, 2018

https://glitchgallery.org/psychedelic-rainbow/

Sample file

WMV3

Broken experimental encoder

FFglitch early history:

  • Glitch Artists Collective Facebook group

 

 

 

  • It should be possible...
  • So I set out to create a bitstream editor

"Is it possible to edit the motion vectors in a video file?"

Proof of concept:

  • Reuse FFmpeg codebase
  • Pass 1:
    • decode file normally
    • export motion vectors to JSON file
  • ... modify JSON file ...
  • Pass 2:
    • decode file again
    • while decoding, replicate bitstream
    • while replicating, modify motion vectors

GetBitContext:

  • Structure used to read bits
    from a bitstream
  • Functions such as:
    • get_bits1()
    • get_bits(n)
    • get_vlc2() [variable length codes]

MPEG-4 P frame

with motion vectors

MPEG-4 inter macroblock

Bytes:
52       C1       D2      [...]
Bitstream:
01000100 10000100 11000001
01000100 10000100 11000001
get_bits1():
0     not_coded
01000100 10000100 11000001
get_vlc2():
1     cbpc
01000100 10000100 11000001
get_vlc2():
00010 cbpy
01000100 10000100 11000001
get_vlc2():
get_bits1():
get_bits(5):
01
0        MV delta x (3)
00010
\}
01000100 10000100 11000001
get_vlc2():
get_bits1():
get_bits(5):
01
1        MV delta y (-1)
00000
\}
01000100 10000100 11000001

PutBitContext:

  • Structure used to write bits
    to a bitstream
  • Functions such as:
    • put_bits()

Bitstream replication:

Input:
01000100 10000100 11000001
01000100 10000100 11000001
get_bits1():
0     put_bits(0)
0
Output:
01000100 10000100 11000001
get_vlc2():
1     put_bits(1)
01
01000100 10000100 11000001
get_vlc2():
00010 put_bits(00010)
0100010
01000100 10000100 11000001
get_vlc2():
get_bits1():
get_bits(5):
01
0        put_bits(010​00010)
00010
\}
01000100 1000010
01000100 10000100 11000001
get_vlc2():
get_bits1():
get_bits(5):
01
1        put_bits(011​00000)
00000
\}
01000100 10000100 11000001
01000100 10000100 1100000
0        put_bits(010​00010)
00010    put_bits(1)

Bitstream transplication:

Input:
01000100 10000100 11000001
01000100 10000100 11000001
get_bits1():
0     put_bits(0)
0
Output:
01000100 10000100 11000001
get_vlc2():
1     put_bits(1)
01
01000100 10000100 11000001
get_vlc2():
00010 put_bits(00010)
0100010
0100010
01000100 10000100 11000001
get_vlc2():
get_bits1():
get_bits(5):
01
0
00010
\|
01000101
01000100 10000100 11000001
get_vlc2():
get_bits1():
get_bits(5):
01
1        put_bits(011​00000)
00000
\}
01000100 10000100 11000001
01000101 01100000
0        put_bits(1)
export function glitch_frame(frame)
{
  // get the forward motion vectors
  const fwd_mvs = frame.mv?.forward;
  if ( !fwd_mvs )
      return;

  // clear horizontal element of all motion vectors
  for ( let i = 0; i < fwd_mvs.length; i++ )
  {
    // loop through all rows
    const row = fwd_mvs[i];
    for ( let j = 0; j < row.length; j++ )
    {
      // loop through all macroblocks
      const mv = row[j];
      mv[0] = 0; // this sets the horizontal motion vector to zero
    }
  }
}

mv_sink_and_rise.js

Glitched file

MPEG-2 glitch

PNG

change filter type from "up" to "avg"

JPEG

from pixels to bitstream

http://jpeg.ffglitch.org/tsunami.html

Supported codecs:

  • JPEG
    (DCT coefficients, Quantization Table)
  • MPEG-2
    (​DCT coefficients, Motion Vectors, qscale)
  • MPEG-4​
    (Motion Vectors, Global Motion Compensation)
  • PNG
    (​Headers, Image Data (IDAT))

Supported formats:

  • rawvideo
  • AVI
  • MOV

Extra features:

  • Scripting support through QuickJS and Python3
  • HID support with SDL2
  • MIDI controllers support with RtMidi
  • Network support using ZeroMQ

FFglitch is 3 programs:

  • ffedit (new tool)
    • the main tool for FFglitch
    • it's a multimedia bitstream editor
  • fflive (it's the 'ffplay' binary)
    • integrated with bitstream editor
    • create live glitch in real-time
  • ffgac (it's the 'ffmpeg' binary)
    • but with some extra features for glitch artists

Upstreaming to FFmpeg?

  • Most of it is way outside the scope of FFmpeg
  • Capture support (v4l2)
  • Bug fixes
  • Tests
  • Optimizations

Orange Pi 5 Ultra

What's next for FFglitch?

  • More video codecs (H.264, AV1, ...)
  • More image codecs (GIF, JPEG 2000)
  • Audio codecs! (MP3, AAC)
  • Integration with live mixing/streaming:
    • Syphon
    • Spout
  • Steganography (?)
  • Codec security (?) (kind of like h26forge)
  • Your contributions!!!!!!

Gallery

Kaspar Ravel and Thomas Collet

Thomas Collet

Sébastien Brias

fflive demo

Thank you!

 

 

Let's go for a 🍺 later today and talk about glitch art!

FFglitch FOSDEM 2025

By Ramiro Polla

FFglitch FOSDEM 2025

  • 74

More from Ramiro Polla