GPUImage

Processing library for high-performance image computing and GPGPU computing (GLSL).

Plateforme :	Status:
Windows
OSX
Linux

Features

• Image filtering (color, blur, effects, filtering...)
• Image compositing (fusion mode, mask, chroma key...)
• Ping Pong buffering
• Data packing into RGBA texture

Tested platforms

• Windows :
  • Windows 10 x64, 16GO, intel i7 3.6Ghz, GPU NVidia GTX 970
  • Windows 10 x64, 16GO, intel i5 3.4Ghz, GPU NVidia GTX 660 #31
• Linux 16.04, 16GO, intel i7 3.60Ghz, GPU NVidia GTX 1080 #32
• OSX 10.13.5, 16GO, intel i7 2.5Ghz, GPU AMD Radeon R9 M370X #33

Informations and major updates

• The v.0.0.1 the library is build on the top of processing which means we are using processing class such as PGraphics, Pimage, PApplet...
• The v.1.0 is based on the processing PJOGL profile. There is no change in the GL context and it's still build on GL2ES2 (a version between GL2, GL3 and GLES2) GL2ES2 JOGAMP
• Because the library relies on the PJOGL profile (GL2ES2) all shaders are based on GLSL 1.50 (150) GLSL versions table. We don't set the version directive on the shader on order to keep compability with OSX plateform. See issue #29 for more informations
• #11 The effector system is design as a base component : GPUImageBaseEffects which is extends by 3 differents class : Filter, Compositor and ProceduralTexture. Each of them is based on the ping pong buffer system set in the GPUImageBaseEffects. Each of them can take cares of differents actions :
  • Filter : All filtering and VFX operations on a single image (chromawarp, blur, sobel...)
  • Compositor : Composition between two images (mask, blending...)
  • ProceduralTexture : Generation of procedural texture (noise, FBM, voronoi...)

Architecture et Design Pattern

To do

Global/Recette

• Add try catch and handle errors
• Add custom throw error (view adidas source)
• Find good licenced image for library usage/example

Core

• Library info
• simple GPU info
• custom GPU info (memory size...)

PingPong Buffer

PingPong buffer is an utils class allowing you to create a ping pong buffer in order to make read-write texture pipeline by creating 2 buffers. For more information about the differents implementation test see issue #5

• Processing ping pong buffer using PGraphics Each buffer can be swapped so the second is always a previous version of the first one. The main idea of this class is to keep the paradigm of an offscreen buffer made in processing using PGraphics through the PinPongBuffer object. User does not have to learn an new offscreen drawing implementation with various context and drawing methods. Drawing can be handle using

object.dst.beginDraw();
object.dst.endDraw();

• Trying extend PGraphics in order to create a simple PingPongBuffer → ref :

• Extending PGraphics (p5 forum)
• Extending Pgraphics (2) (p5 forum)
• Praxis implementation WIP Processing ping pong buffer extending PGraphicsOpenGLClass The class extends the PGraphicsOpenGL class in order to have the same drawing methods. At each swap the main buffer is copied into the prebious buffer using arrayCopy. User can draw into the PingPongGraphics using all the PGraphics methods

object.beginDraw();
object.endDraw();

This methods is a test methods only. We need to make a benchmark in order to define the best solution. NB : Tested on a filter methods inside a forloop, this method is too slow on the arrayCopy() (30 fps at 60 iterations) vs the previous one (60 fps at 60 iterations)

• independant ping pong buffer (custom JOGL implementation) for Floating Point Texture

• add smooth to PPB

• Add a custom PGraphicsOGL implementation using nearest filtering, warping... see issues #28 and #26

Others

• add filtering capabilities :

  buffer.hint(DISABLE_TEXTURE_MIPMAPS);
  ((PGraphicsOpenGL)buffer).textureSampling(3);

Filtering/Compositor/Procedural Texture

GPUImageBaseEffects

Filter extends GPUImageBaseEffects

Filtering

• Bilateral

• • Debugger le filtre GLSL (Inversion d'UV)

• Simple Denoise

• • set custom param binding to shader

• Median 3×3 (TBD : iteration avec PingPong Buffer)
• Median 5×5 (TBD : iteration avec PingPong Buffer)
• Edge detection + Sobel + Canny
• High-Passing
• Hue segmentation
• Dilation
• Erosion
• Signed Distance Field To be corrected as true SDF

Blur

• Gaussian

• • set custom param binding to shader

• Fast Blur 5x5, 7x7, 9x9, 13x13
• Radial blur + optimized low, medium, high

Color :

• Contrast/Sat/Bright
• Desaturate
• Level
• Gamma

• • set custom param binding to shader

• LUT (1D)
• LUT (2D)
• Ramp (1D)
• Color Threshold
• Threshold
• Invert image

Effect/VFX :

• ChromaWarp + optimized low, medium, high
• Fast ChromaWarp 4, 10, 20
• Grain
• Grain RGB
• Dithering : Bayer (2x2, 3x3, 4x4, 8x8), Cluster Dot (4x4, 8x8, 5x3), Random (3x3)
• Pixelate
• ASCII Est-ce un filtre ou une composition ? → http://paulbourke.net/dataformats/asciiart/ (class dédiée)
• Bloom
• SlitScan effect
• Glitches
• Datamoshing

Other :

• Optical Flow
• FrameDifferencing
• Pixel sorting
• DoF from Depth (TBD)
• Normal Map converter

Compositor extends GPUImageBaseEffects

• Photoshop fusion mode (multiply, add...)
  • add
  • average
  • color
  • colorburn
  • colordodge
  • darken
  • difference
  • exclusion
  • glow
  • hardlight
  • hardmix
  • hue
  • lighten
  • linearburn
  • lineardodge
  • linearlight
  • luminosity
  • multiply
  • negation
  • overlay
  • phoenix
  • pinlight
  • reflect
  • saturation
  • screen
  • softlight
  • substract
  • vividlight
• Mask

• • set custom param binding to shader
• • clean shader

• Alpha Matte (sprite mode)
• Chroma key

Procedural extends GPUImageBaseEffects

• Noise & Random
  • Noise
  • Random
  • Simplex
  • Gradient
  • Cellular
  • Fractal
  • Cellular
• Isosurface
• Reaction-Diffusion

Packing RGBA

• Float to RGBA algorithm precision test
• Abstract class with various encoding/decoding methods (RGBA16, 24, 32) from float and double
• Main float packing class
• RGBA1616 double packing → Allowing two data (1 per 2 channel 1616)
• ModIntPacking allowing to pack int value into 8bits + 8bits index
• Bit shift packing to PImage (only usefull for CPU computation)
• Performances
  • Quid to use generic type in order to simplify class writing ? (Not quite efficience for now)
  • Quid to thread encoding operation for performance ?
  • Quid to bypass the PImage object in order to encode in a spediest way ?
    • Using BufferedImage seems to speed up the performance Reference
    • Using specific method without switch/case. The test was too low
  • Find a way to create a rectangle from area only. See sehrope implementation en stackoverflow
• Pack/unpack RGBA (16/24/32) shader model
• Pack/unpack RGBA1616 shader model
• Pack/unpack RGBAMod shader model

Geometric (VBO)

Wiki (TBD)

Exemples

• Composition
  • Blending
  • Mask
  • Alpha sprite
  • Chromakey
• Core
  • Infos
• Filters
  • Sequential filtering
  • Filtering
    • Denoise (denoise, bilateral, median3x3, median5x5)
    • Edge detection + Sobel + Canny
    • High-Passing
    • Hue segmentation
    • Dilation/Erosion
  • Blur
    • Radial blur
    • Gaussian blur
  • Color grading
    • Threshold (gray + color)
    • LUT1D
    • Ramp 1D
    • Level
    • Gamma correction
    • Desaturation
    • Contrast Saturation Brightness
    • Invert
  • Post fx
    • White grain
    • RGB grain
    • Chroma warping
    • Pixelate
    • Ditherings
    • Ditherings RGB
    • Glitches
• Other
  • Accentuation
  • ProcessHueSegmentation
  • Kinect Depth IntToRGBA
  • 2D GPU physics engine
  • 3D GPU physics engine (broken)
• Procedural
• RGBA Encoding
• Utils
  • Generate 1D LUT
  • Ping Pong Buffer