Next: Blue Banana Up: Video Effects Previous: Auto Scale Contents Index
The following description is based on the /cinelerra-5.1/doc/README.blendalg that accompanies the code by the author of both the code and the document. It is the authoriative documentation; the most detailed/original source and has additional commentary and programming information. If you encounter problems or confusion when using these plugins, always refer directly to that document here.
The idea behind these plugins6 is that the user can program sufficiently short and simple blending algorithms, called Functions, by following detailed instructions with no real programming skills required. Blend Algebra and Blend Program come with a library of Functions (System Library) that mimic the 30 overlay effects found in the Patchbay or Overlay plugins. The user can modify these Functions as needed or create new ones. To look at these example functions, there is an Edit... button in the respective plugin dialog. You will need a text editor to do so, which by default is emacs if it is installed. If not, you can set your favorite editor via the corresponding environment variable in a terminal (export CIN_EDITOR='name editor') and then start CINELERRA-GG from the same terminal.
If you can not wait to get started, checkout these examples here. You will also find test projects to download to learn more about the plugins and to see the various ways they can be used to create your own functions.
Both plugins, Blend Algebra and Blend Program, are similar. They are described together along with their differences. The main difference between the two plugins is that Blend Algebra is like a function; it combines its arguments (tracks) and yields the result which is placed into the single track configured for output. However, Blend Program does not produce a function result, but instead directly modifies its arguments. If more than one track is to be modified, Blend Program is the only choice. If a single track is to be processed, Blend Program is the best choice and might run slightly faster. Any Blend Algebra function can be rewritten to become a Blend Program function. But if a function processes several tracks and modifies only one, it will run faster in Blend Algebra which has more controls in its plugin dialog.
How the plugin works is defined by the user. The word function is used to denote this for either of the two plugins throughout this description. These functions are for their plugins the same as the plugin shared objects in the main CINELERRA-GG binary. But these functions are simple enough that they can be easily written by users and manipulated in binary or in source form. The plugins make it possible to have a user library and a system library of functions. Thus, an experienced user can have a number of such functions to reuse in many projects.
The plugin is attached to a track as usual. Generally, it needs several tracks as arguments and it has to be attached to the additional tracks as a shared effect.
All the parameters described below are keyframable, including user functions themselves. The top area of the dialog deals with programmatic aspects. The middle area of the dialog is color specific. The bottom area of the dialog has controls for the argument track ordering, similar to that of the Overlay plugin.
export CIN_EDITOR=kate export CIN_EDITOR=gedit ... $ ~/cinelerra5/cinelerra-5.1/bin/./cin |
export CIN_EDITOR='konsole -e vi' $ ~/cinelerra5/cinelerra-5.1/bin/./cin |
There may be illegal floating point operations in the user's code, such as division by zero (producing infinity), or square root from minus unity (producing NaN, not-a-number). Even if only legal finite floating point numbers are computed, the results may go out of range. This dialog section controls what to do in such cases.
RGB: [0.0 .. 1.0] for R, G, B YUV: [0.0 .. 1.0] for Y, [-0.5 .. +0.5] for U, V HSV: [0.0 .. 360.0] for H, [0.0 .. 1.0] for S, V Transparency: [0.0 .. 1.0] for Alpha |
All the parameters are keyframable, including user functions themselves. That is, a plugin can have several functions attached at the same time and switch between them while rendering. The color and alpha channels of the substitution (key) color are interpolated between keyframes; the other parameters are switched.
The user library directory is that defined by the environment variable $CIN_USERLIB, or, if CIN_USERLIB is not defined, $HOME/.bcast5lib as a fallback default. The functions for Blend Algebra reside there in the subdirectory dlfcn/ba and for Blend Program in dlfcn/bp (dlfcn stands for dynamic loaded function). Having user functions in such a library in a single place makes it convenient to reuse the same functions in different projects. The Copy to userlib button in the Attach... file selection dialog is handy to save such functions in the user library for future use.
The reason to define user library $HOME/.bcast5lib by default (and not $HOME/.bcast5) is the following. There is a common workaround: if a newer version of CINELERRA-GG does not start, to delete $HOME/.bcast5 completely. Doing so, all user functions would be deleted if they were saved there.
The system library is defined by the environment variable $CIN_DAT, the CINELERRA-GG installation path, and under the same subdirectories dlfcn/ba or dlfcn/bp. System library is meant for functions distributed with CINELERRA-GG.
When a complete project is to be exported, or fed to a render farm, it is often necessary to place all the resources under the same directory where the project .xml file resides. If some functions are used which reside elsewhere (for example, in a user library), they could not be found after exporting the project to another computer. The Copy to project button in the Attach... file selection dialog is used to copy such functions to the project directory in a convenient manner.
Blend Algebra/Blend Program functions are written in C, and with usage of a set of special cpp macros hiding from the user all the Cinelerra internals. This approach makes it easy to write functions of moderate complexity with almost no background in C programming. All macros defined for user functions are written in capital letters. A typical Blend Algebra function consists of the following logical blocks.
<optional static variable definitions> BLEND_ALGEBRA_INIT <macros and/or C statements for the INIT (preparation) phase, will be executed once before processing of each frame> BLEND_ALGEBRA_PROC <macros and C statements for the PROC (processing) phase, will be executed repeatedly for each pixel in the frame> BLEND_ALGEBRA_END |
The macro BLEND_ALGEBRA_STOP terminates execution of the INIT or PROC phase immediately and returns to the calling plugin.
The structure of a Blend Program function is the same, only the word ALGEBRA in all macros must be changed to PROGRAM, here and below. The following macros can be used in the INIT phase:
COLORSPACE_RGB COLORSPACE_YUV COLORSPACE_HSV |
Declare the working color space inside the user function. If there is no such declaration, the native color space of the project (RGB or YUV) will be used.
REQUIRE_TRACKS(<n of tracks>)
Declares the minimum required number of tracks the function uses. If the effect is attached to more tracks, it is OK. If there are less tracks, it is an error and the function is not executed. The absence of such declaration means that the function can process any number of tracks (one or more).
PARALLEL_SAFE
This declaration means that it is safe to execute the function in parallel. Without this statement the function will be executed sequentially, independent of the state of the parallelization checkbox in the plugin GUI.
The following special variables (macros) can be used for queries:
PARALLEL_REQUEST
1 == parallelization requested in the GUI
0 == parallelization switched off
TOTAL_TRACKS
The actual number of tracks passed to the function.
HAS_ALPHA
1 == the project color space has alpha channel (transparency)
WIDTH, HEIGHT
The dimensions of processed frames, in numbers of pixels.
The following variables (macros) can be used in the PROC phase.
TOTAL_TRACKS, HAS_ALPHA, WIDTH, HEIGHT can be used as well.
R(i), G(i), B(i) Y(i), U(i), V(i) H(i), S(i), V(i) A(i) |
Here the letters R, G, B, H, S, V, etc. are for readability. If the user has declared COLORSPACE_RGB, and then written something like H(i), it does not mean that he should get the H-component of color autoconverted to the HSV color space. Actually R, Y, H are mutually identical to the 1-st color component, so are G, U, S to the 2-nd, and B, V to the 3-rd one.
KEY_R, KEY_G, KEY_B KEY_Y, KEY_U, KEY_V KEY_H, KEY_S, KEY_V KEY_A |
Corresponding color components of the key (substitution) color, in the working color space of the function, and the opacity slider.
R_OUT, G_OUT, B_OUT Y_OUT, U_OUT, V_OUT H_OUT, S_OUT, V_OUT A_OUT |
Color components for the result, for Blend Algebra only. Blend Programs return no result, and do not have such definitions.
PIX_X, PIX_Y |
Integer X and Y coordinates of the current pixel in its frame, in the ranges of [0 .. WIDTH] and [0 .. HEIGHT], respectively.
CLIP_RGB(i) CLIP_YUV(i) CLIP_HSV(i) CLIP_RGBA(i) CLIP_YUVA(i) CLIP_HSVA(i) CLIP_A(i) |
These macros clip color components of the i-th track to the bounds appropriate in the respective color space.
CLIP_RGB_ALL CLIP_YUV_ALL CLIP_HSV_ALL |
Clip all color components (including alpha) for all tracks.
CLIP_RGB_OUT CLIP_YUV_OUT CLIP_HSV_OUT CLIP_RGBA_OUT CLIP_YUVA_OUT CLIP_HSVA_OUT CLIP_A_OUT |
Like clipping track color components, but for the result of a Blend Algebra function. Not relevant for Blend Programs. Clipping is done according to the inherent bounds of the respective color space, as follows:
RGB: [0.0 .. 1.0] for R, G, B YUV: [0.0 .. 1.0] for Y, [-0.5 .. +0.5] for U, V HSV: [0.0 .. 360.0] for H, [0.0 .. 1.0] for S, V Transparency: [0.0 .. 1.0] for Alpha |
H color channel of HSV is brought into range by repeated rotation around 360 degrees, all the others by simple clipping.
ABS(x), SQR(x), MAX(x,y), MIN(x,y) absolute value, square, max and min values for floating point arguments. Can be used in any phase of a function.
TO_RAD(x), TO_DEG(x) conversion from degrees to radians and vice versa.
CLIP(x,y,z) CLAMP(x,y,z) |
Both macros clip the 'x' argument to bounds between y and z. CLIP returns value leaving x unchanged. CLAMP assigns that value to x.
All the macros are defined in the text header $CIN_DAT/dlfcn/BlendAlgebraStart for Blend Algebra and the analogous header BlendProgramStart for Blend Program in the same dlfcn subdirectory of the Cinelerra installation directory. These headers are prepended to the user functions during compilation. The user with a basic C knowledge can easily understand what happens in these headers.
Of course, you can use any valid C statements inside user functions. As they are linked with -lm, the standard C math library, and of course with glibc, almost any mathematical or C library function can be called. C-style comments can also be used, and are welcome.
Compilation of Blend Algebra and Blend Program functions is carried out by a Perl script BlendAlgebraCompile.pl and BlendProgramCompile.pl. Both scripts are distributed in the dlfcn subdirectory of $CIN_DAT, the Cinelerra installation directory. Like the famous ContextManual.pl script of Context Help, the system script at the first call is copied into $CIN_CONFIG ($HOME/.bcast5) where it can be edited by the user and adapted according to your needs.
While a regular compilation, the script works in that directory where the function source file resides. First, the BlendAlgebraStart (BlendProgramStart) header is prepended to the source to produce an intermediate file with the .c suffix. This C source is compiled by the system C compiler (gcc by default) and linked with -lm to produce the shared object ready for dynamic loading (attaching to the plugin).
Usually the script is executed by the respective CINELERRA-GG plugin when a user function is to be compiled. But the script can also be run by the user explicitly. Execution can be controlled by options starting with the '-' character, the first argument not starting with '-' being the function name to compile. The following options are recognized:
-compile don't check modification time, compile unconditional
-cfile don't remove intermediate .c fil
-opt add optimizing options to compiler command line
-debug add debugging options to compiler command line
-warn add warning options to compiler command line
-edit open function source in text editor
-verbose verbose execution
-noapi don't copy itself to $HOME/.bcast5
-h, -help, -? print short help and current configuration
In the beginning of the script some variables can be redefined according to the user's needs: the C compiler executable, compiler options for optimization or debugging, text editor executable.
The following environment variables are taken into account in various places in Blend Algebra / Blend Program.
$CIN_CC: C compiler executable (default: gcc)
$CC: C compiler executable if $CIN_CC is not defined (default: gcc)
$CIN_EDITOR: text editor (default: emacs)
$CIN_DAT (set by Cinelerra binary): Cinelerra installation directory. The Blend Algebra / Blend Program system library with scripts, headers, and functions distributed with Cinelerra resides in the subdirectory $CIN_DAT/dlfcn.
$CIN_CC$CIN_CONFIG (set by Cinelerra binary): user's config directory, usually
$CIN_CC$HOME/.bcast5
$CIN_USERLIB: user's library (default: $HOME/.bcast5lib). User's functions reside in the subdirectory $CIN_USERLIB/dlfcn.
In the distribution 30 Blend Algebra functions are provided for all the 30 CINELERRA-GG overlay modes according to the formula defined in CinelerraGG manual. In principle, they are not needed as the built in overlayer engine is OpenGL accelerated and runs faster. Nevertheless, these functions can be handy as starting examples for users who may like to modify the formula in some way. Furthermore, their behavior differs from that of the standard overlayer in several aspects:
There are several example functions from totally different areas of application: a user programmable plugin is a really universal toy and can be handy even in unexpected cases. Even a programmer can sometimes make use of this plugin when creating new effects. You one can more quickly test different combinations without spending time to restart CINELERRA-GG, recompile it, reload project, and reattach plugins, etc.
Also, there is the ydiff.ba function, actually representing the complete ydiff application running inside a plugin. See the examples ![[*]](crossref.png)
here. In addition, there is a Transition example,
and 2 more function examples showing use of the Blend Programs: the chromakey.bp and background.bp functions. They too serve as excellent starting points for further user's experimentations. The usage of the provided functions is either self evident, or briefly
described in the comments of their source files.
make install copies the functions into Cinelerra's bin directory in source (*.ba, *.bp) and binary (*.so) forms by cp -a, to preserve modification times of the files. Otherwise, if modification time of the function source accidentally becomes newer than that of its binary, the plugin will repeatedly try to recompile the function. If they are installed system-wide, so that the user has no permission to modify them, such recompilations will fail.
User's functions can have bugs.
Most probable bugs in a user function can be illegal arithmetic such as division by zero or logarithm of a negative number (FPE, floating exception), and out of bounds indexation of arrays (usually leading to SEGV).
Most modern processors don't generate FPE. Instead, they generate NaN as the result of an illegal arithmetics. After calling user functions the plugins test the results not to be NaN and replace them with the configured substitution color if necessary.
SEGV is more problematic. If the user has written in his function, let's say, REQUIRE _TRACKS(2), and then accessed R(1000000), most probably Cinelerra binary will crash. Although theoretically it could be possible to trap SEGV via sigaction()/setjmp() /longjmp(), in Cinelerra it is problematic. In a multithreaded application, which Cinelerra is, only one signal handler for all its threads in the same time is allowed, and Cinelerra has already its SEGV handler which we are not allowed to overwrite. Thus, the user is responsible for his indexation bugs, where most attention has to be paid, as usually, to the off-by-one errors, like referencing elements like R(TOTAL_TRACKS) (the last legal element being R(TOTAL_TRACKS-1)), or R(i) where i has been decremented to -1.
The user is allowed to call any C function from the -lm and glibc libraries. But some functions are not thread safe (which means, PARALLEL_SAFE macro should not be used), and some others have undesired side effects. For example, should the user write inside his function something like exit(R(0));, the whole Cinelerra would immediately halt. The user is left to imagine what would happen after the following expression, for example:
R_OUT = R (system ("killall -9 X"));
Blend Algebra / Blend Program functions are not accelerated. Hardware acceleration would mean, the function algorithm is to be programmed as an OpenGL shader.
However, functions can be parallelized (using the load balancing engine of Cinelerra), and can be compiled with optimization (switched on by default). The default optimizing option (defined in BlendAlgebraCompile.pl and BlendProgramCompile.pl) is -O2. One can redefine it to -O3 or 0fast, for example. However, it has to be noted that using 0fast or -ffast-math options can lead to ignoring some IEEE rules for floating point math, namely, some intrinsic tests on infinities or NaN can yield unpredicted results.
Although user's function can be compiled with debugging option such as -g or -ggdb, debugging functions can be tricky. The programmer can set breakpoint into the function only after its code has been loaded in memory by the plugin, and set another breakpoint into plugin only after that plugin itself has been attached to a track. Moreover, a function can get detached from memory under some conditions, and then its breakpoints will be lost.
Debugging printout inside the PROC phase, although possible, would seem too verbose: the PROC phase would be called for full-HD footage 2073600 times per frame! Here perhaps one could make use of PIX_X, PIX_Y coordinates at some selected place like in an example:
if (PIX_X == 320 && PIX_Y == 200)
Thus far, to edit user functions an external editor has to be used. Although it would be possible to implement editing directly in Cinelerra via some multiline scrollable text field, this could hardly present features comparable with a dedicated text editor.
The current implementation should be rather portable. One prerequisite is a working C compiler. That compiler which was used to build Cinelerra itself, will work by definition. But other compilers perhaps are compatible as well. I tested at least functions which were compiled with clang, attached to Cinelerra compiled with gcc, they worked.
If the user has built Cinelerra from sources, he definitely has a working C compiler. But if one has installed the binary distribution, then the compiler clearly is a prerequisite.
It is not known at this time if Blend Algebra / Blend Program work under Cinelerra in an AppImage form. Other architectures are still to be tested. If some other architecture is Unix-like, and is ELF based, it should be portable. But in the case of Windows it is not portable.
As in any realtime plugin, the process_buffer() method in Blend Algebra / Blend Program gets a set of frames from the tracks the plugin is attached to. Then the following events take place. As in any other plugin, it is checked if the configuration (the parameters) got changed. There is one parameter which requires special treatment, the user function name.
In order to prevent from resource consuming recompilation of the functions on each new frame, the plugin maintains in memory cache of the successfully compiled and attached functions. If at some keyframe the function name gets changed, the plugin searches if this function is already known and cached. Among other important function related objects, such as entry points, there is a timestamp, when this function was last checked to be up to date.
If the current function name is empty, it means a function is not used. Nothing else has to be done, all tracks are fetched and passed further in the processing pipeline unchanged. If the function is not empty and seen for the first time, or its timestamp is older than the global timestamp, it is checked as follows.
While recompilation and dynamic linking various things may go wrong.
Updating global timestamp forces all cached functions to be checked for recompilation at their first access. The global timestamp itself is updated by the following events: changing function name, pressing Edit... or Refresh button, exiting the Attach... dialog with the OK button.
If currently active function is ensured to be up to date and correctly attached, the plugin fetches video frames from all the affected tracks, with the important parameters like frame width and height. Then the INIT phase of the function is executed (once for each frame). Here, several important parameters are requested, which are defined by the function. They are:
After this preparation phase, the processing itself takes place. In case of sequential flow, the following is done, for each frame pixel individually.
For each input track, the corresponding pixel is splitted to color components according to the actual color space of the project. All color components are converted to float (type of the C language) in the ranges [0.0..1.0] for R, G, B, A, Y or [- 0.5.. + 0.5] for U, V. If the project color space has no alpha channel, the alpha component is set to A=1.0.
If the function requires other color space than the project uses, the necessary conversions are performed. The key color components (selected in the plugin GUI) are also converted to the function color space in the same manner.
For Blend Algebra, the values for output are preinitialized from that track which the output is to go to. All the other tracks are cleared if the corresponding checkbox in the plugin GUI says to do so. For Blend Program, this step is not needed.
The user function is called with the parameters: actual number of tracks, 4 arrays for the 4 color components (dimensions are equal to number of tracks), 4 key color components, current pixel coordinates (x, y) (upper left frame corner is (0,0), lower right is (width-1,height-1), width and height. For Blend Algebra, placeholders for the result additionally.
The user function returns. First of all, the color components of the relevant pixels are clipped to range if the corresponding checkbox in the plugin GUI is on. Relevant for Blend Program are pixels of all the tracks (all tracks can be modified), for Blend Algebra the result only.
After optional clipping, the color components are checked for not being NaNs. If so, the pixel is replaced with substitution color, then backconversion to the project color space is performed.
If the project has no alpha channel, but the function returned something not equal to 1.0, alpha channel is simulated as if an opaque black background were used.
If the project color space is not FLOAT, unconditional clipping followed by 8-bit transformation takes place.
For Blend Algebra, the result is placed to the right output track. For Blend Program, this step is unnecessary, all tracks are modified in place.
Then the loop is repeated for the next pixel in a row, next row in a frame, next frame in the whole sequence...
If the function is to be run parallelized, the necessary number of threads is created (as defined in Settings → Preferences → Performance; Project SMP cpus). Although it might seem that running parallel on 1 cpu be the same as running sequential, strictly speaking it is not the case. To run parallel on 1 cpu, an extra processing thread is still created, while sequential execution takes place inside the main plugin thread. This could induce some subtle differences if the function uses static variables inside or something else which is thread unsafe.
![\includegraphics[scale=0.5]{Blend_Program.png}](img28.gif)
