This is another amazing tool that complements gstreamer. It lets you mix audio and video feeds quickly and easily and most importantly dynamically without having to write programs.
Here are some notes from the wiki, the discussion group there,  but mainly by looking at the sample snowmix ini files in the Snowmix/ini directory:

1. This doesn’t have a git repository, so there is no easy way to see what has changed from version to version or see , who is doing the editing. It’s probably easiest just to suck it into your own github repository just for your own sanity. That is pretty easy to do. Create it on github.com and then git clone the empty one down. Then extract the tarball into it. Commit that and you have something with version control.

2. The setup is with bootstrap (no relation to the Twitter invention). It is an interactive script so not well suited to installing with other modules. An easy addition is to silence all the read commands with a option flag if you really need it to be quiet. strapboot is the equivalent of make clean and blows it all away. Personally I’ve always liked buildall.sh and cleanall.sh but that’s a nit.

3. The thing cooperates with gstreamer via shared memory pipes so you have to start snowmix first and then start gstreamer for input and then another gstreamer for output. Basically, you can set it up to read from a file input or from a shared memory pipe. It’s output is another shared memory pipe.

4. This program likes to lives in it’s own directory. It is quite node-like in that this way. Even the binary runs inside the current working directory. Each program needs the shell variable SM set to that directory.

5. There are a host of configurations that you feed it at the beginning via the single command line ini file. However, you can also dynamically talk to it and configure it via a port (by default 127.0.01 port 9999) which it listens to for plain text commands. For instance nc 127.0.0.1 9999 <<<“feed list” gives you a list of feeds in the system. Pretty cool.

Syntax of Snowmix commands

It’s a little hard to sort through the documentation, but the wiki looks like the main source and Peter (the developer) is active on the SourceForge discussions. But here’s a quick guide for programmers:

1. There are no variables in the language, so every object just has an object id (typically starting from 1).

The arguments that are passed are completely positional. This means that you need a big of decoder ring as some things take 8 arguments (x, y, step x, step y) and it is like reading a shell script argument list

The interactive and programming language are the same. It is all run time interpreted which is nice and flexible.

Data Types

1. Feeds. These are the basic video feed element. They are hard wired by a feed socket command so you can change feeds and their sockets much. By default there are 12 feeds supported but you can recompile and change this. They can be moved with feed shift and scaled with feed scale which is set it to n/d the size. This convenient when you want it to fit in say a smaller windows like 120/96. There is a stack command that gives the z ordering of the feeds. Note that you cannot rotate or move or change the feed socket, so this is the basic primitive.

2. Text. These are primitives referred to by id numbers

text string 1 Hello Hello World

You can delete a string by leaving the final argument blank so text string 1 by itself is delete. Text strings also have some substitution variables.
How you set this is a little complicated because text unlike feeds isn’t dynamic, so you basically have to run it in a command macro that runs on every frame. There is a magic command overlay finish ShowText that says after the overlays are done run me and this is the thing that does text placement. So we need to understand the execution environment but you can change text with commands and also by changing them through the control console. We will show this later, but this works so that you can use shell script to control snowmix

#!/bin/bash
while true; do
# Change a string
nc 127.0.0.1 9999 <<<“text string 1 Changed to Good Bye!!"
# Create a place id 1 for text string 2 using font 3 and put it at at x=200, y=300
nc 127.0.0.1 9999 <<<“text place 1 2 3   200 300"
sleep 10
nc 127.0.0.1 9999 <<<“text string 1 Say Hello Again"
od

1. Images. These are also first class object. The main thing is that you should use .png images as these have an alpha channel so that you can have logos without white backgrounds.

Higher level objects

1. Virtual Feeds. For some reason this isn’t in the current wiki documentation. Perhaps because shape feeds are more powerful (see below) are more powerful. They are better than feeds but have less flexbility than shapes. However the discussion has an ini file that explains how it works. As a small trap, virtual feeds only work if you define some text, so make sure to do that. After you define a feed,

# virtual feed add
virtual feed add 1 this is virtual feed number 1
# virtual feed source feed
virtual feed source feed 1 1
# the place thing is massive with location x,y and the size w,h and then the source location so you can do a cut out, then the rotation you apply and how you scale it and then the alpha blending (1.0 is solid, 0 is nothing there).
# virtual feed place rect
# this says for virtual feed 1, put it at the upper left (0,0) and it is 1280x800. The source to get it from is at 0,0 with 0 rotation and scale it by ? on each dimension and set the alpha at 1
virtual feed place rect 1 0 0 1280 800 0 0 0.0 0.3333 0.3333 1.0

There are some other commands as well for virtual feeds including:

# This gradually fades virtual feed 1 to 0.04
virtual feed move alpha 1 -0.04 25
# this move the feed to x, y
virtual feed move cord 250 100

1. Shapes and Cairo. Built at the highest levels are arbitrary shapes that the 2-d graphics library Cairo can alter. Here is where you get lots of control of images. There isn’t much documentation here, so you basically have to try to map the Snowmix command to the Cairo primitive. The most important thing is that with shape feed you can bind a feed directly to a shape for really complex manipulates. The same is true for other primitives via shape text and shape image. You can create arbitrary shapes as well by giving it paths so

shape add 1 Arc Ewith a stroke at the end
shape arc_cw 1 0.0 0.5 0.1 3PI/12
shape close path 1

Then once you have a shape you can fill it, so this says put a stroke or line around it

shape stroke 1

You can then move shapes around the screen with shape place commands

Command macros and TCL

There are two programming languages in the snowmix. A homegrown macro language and a TCL interpreter which runs the way that Javascript runs with HTML.

1. The basic command language has a command create and ending in command end and is invoked with just “ at the beginning of a line. It is a basic macro language and can call other macros. There is some limited conditionals based on the feed state (suspended or not), but there are no variables, return codes etc. The main concept is that it will run each frame and when you are done you can call next 10 and it will skip the next 10 frames and at the end you call loop to start over.

The biggest issue with the language is that it is highly stateful. The opposite of say node.js. Everytime you run a macro, it will execute differently. It uses a next which means after you get there, the next execution starts after the next and in effect skips the previous block. Also when you get to the end of command, it won’t start executing again unless you have a loop.

command create several_things_at_once
# This line will only run once
text place 1 0 0
# This says for the next 24 times you call this, run here
next 24
text place 2 0 0
next 24
text place 3 0 0
# if you don’t have the loop, then after 1+24+24 calls it will do nothing
loop
command end

1. These macros get called in two ways. First is that from the command console or from the port, then it will run async. But the main execution loop is started by two commands, overlay finish <command /> so that means it will that command on ever

2. The same macro language can also be used dynamically. If you plug into the control port, you can execute a macro just by typing it’s name in.

3. The macro language can also call out to TCL by a prefaced tcl eval so it can do some math.

4. There seems to be a stack of commands as well found in one of the ini files where lines of macros are taken off and on.

command pop Show
command push Show text place 3 23 43

The system runs at a high frame rate, so it is all about keeping up with that. That means the command macros are run

command create ShowText
# put text object 1, 2 through 7 on the screen
text overlay 1 2..7
command end
overlay finish ShowText

In order to change things, you can not just have a new command running, but instead, you have to do an in place edit of command macros (yikes, that sounds dangerous), so this will have to do this on the command interface. This seems to be the basic way to get into the execution loop, so if you have run in the ini file, overlay finish Show

# This deletes line 1 of the Show macro so it is in the main execution loop
command deleteline Show 1
# This adds a line 1 which shows all the text
command addaline Show 1 text overlay all

Using TCL

Before 0.4.4, you used TCL in one of two ways:

• From within a command language, you can evaluate any TCL command with eval tcl

You can write a tcl routine with command create <command />.tcl and have it return a string with a \n which gets pushed into the interpreters. This allows you to run static commands, but not things that are procedural so you get around this by not using next which only works in macros, but by using at +<time> <command /> which is the same as the next if you think about it but without the looping.

With 0.4.4, this got extended so that instead of this long return string, you can just issue snowmix commands with snowmix parse <command /> directly in TCL making it more straightforward. And you can access snowmix variables with set a [snowmix info text string 1] which would give you all the attributes of a text string. You can then change things and push it out again withsnowmix  parse` and substitute tcl variables.

1. maxplaces. This sets the limits for various objects. Obviously, smaller means higher performance:

# Change default settings for maxplaces BEFORE any of the shape, text etc. commands. This sets the maximum id number you can have in the system and probably the amount of memory

maxplaces shapes 170
maxplaces shapeplaces 128
maxplaces strings 180
maxplaces texts 180

3. If you want you can use the tcl way instead of the macro way of setting things and is probably a little bit more readable.

Load TCL libraries

include ../slib/system.slib
include ../slib/basic-shapes.slib
message SYSTEM LOADED
include ../slib/scenes.slib
message SCENES LOADED
include ../slib/feeds.slib
message FEEDS LOADED
include ../slib/images.slib
message IMAGES LOADED
include ../slib/clocks2.slib
include ../slib/texts.slib
message Snowmix Libraries Loaded
tcl eval FeedCreate “Feed #1” 1 1280 720 /tmp/feed1-control-pipe
tcl eval FeedCreate “Feed #2” 2 1280 720 /tmp/feed2-control-pipe
tcl eval FeedCreate “Feed #3” 3 704 576 /tmp/feed3-control-pipe
tcl eval FeedPAR 3 12 11
tcl eval FeedDeadImage 1 100 ../frames/dead-1280×720.bgra
“`

Debugging

There are few ways to do debugging that isn’t clear from the documentation, but thanks to a note from Peter, here is how it works:

1. If you are out of the overlay loop then you can use the command message which pushes to the standard output or messagef which includes the frame number. This doesn’t run while it is in the main loop as this would flood the system

If you are in main loop, then you can still generate messages by escaping to out with tcl with tcl eval puts “the string name”

And to debug the entire system, the easiest thing to do is to run the command outside of the overlay pre or post loop and keep trying it.

Snowmix will core dump if you Ctrl-C too much and the shared memory is not correctly deleted, you can see how much is available as the shared memory with df  /run/shm and you can get rid of shared memory with rm /run/shm/shm*

CVideoOutput out of chm buffers. This is because the output side can’t keep up with what Snowmix is emitting as described below, so you it’s a good idea to use the included scripts until you figure out your own. In my case I tried to modify the output2screen and didn’t do it well 😉

 When Snowmix runs out of shared memory for outputting frames, it will print the message CVideoOutput out of cshm buffers. and it will suspend outputting more frames until at least half of the configured shm buffers are available again.
It is very important that the script that reads mixed frames from Snowmix runs flawlessly and have sufficient CPU and memory bandwidth availble.When Snowmix runs out of shared memory for outputting frames, it will print the message CVideoOutput out of cshm buffers. and it will suspend outputting more frames until at least half of the configured shm buffers are available again.

It is very important that the script that reads mixed frames from Snowmix runs flawlessly and have sufficient CPU and memory bandwidth availble.

Overlays

This is perhaps one of the most confusing parts of the product and isn’t documented much but basically, you have to overlay things:

• Overlays happen in order. The normal thing is stack 0 which is the lowest layer and then it is block.

• For real feeds, the order is left to right with the right most at highest priority `stack 0 1 2 3 4 5 6 7 8 9’ means 9 is the highest.

• The overlay commands are processed in the overlay loop

Scenes

Quite a lot of snowmix is about changing the screen layout. It is pretty laborious as you have to use macros. There is a new scenes system that makes it easier to switch and is much higher level that command macros but there seem to be very few examples of this.