mtImageCodec

The mtImageCodec is an open-source library of accelerated codecs with unified interface. It is designed as a framework for extension modules which delivers codec plugins.

This mtImageCodec release includes the following key features:

• Unified API for decoding and encoding images
• Batch processing, with variable shape and heterogeneous formats images
• Codec prioritization with automatic fallback
• Builtin parsers for image format detection: jpeg, jpeg2000, tiff, bmp, png, pnm, webp
• Python bindings
• Zero-copy interfaces to PyTorch
• End-end accelerated sample applications for common image transcoding

Currently there are following native codec extensions:

• mtjpeg_ext

  • Hardware jpeg decoder
  • Hardware jpeg encoder

• mtjpeg2k_ext

  • GPU jpeg 2000 decoder (High Throughput Jpeg2000)
  • GPU jpeg 2000 encoder (High Throughput Jpeg2000)

• mtbmp_ext (as an example extension module)

  • CPU bmp reader
  • CPU bmp writer

• mtpnm_ext (as an example extension module)

  • CPU pnm (ppm, pbm, pgm) writer

Additionally as a fallback there are following 3rd party codec extensions:

• libturbo-jpeg_ext

  • CPU jpeg decoder

• libtiff_ext

  • CPU tiff decoder

• opencv_ext

  • CPU jpeg decoder
  • CPU jpeg2k decoder
  • CPU png decoder
  • CPU bmp decoder
  • CPU pnm decoder
  • CPU tiff decoder
  • CPU webp decoder

Pre-requisites

This section describes the recommended dependencies to use mtImageCodec.

• Linux distro:
  • x86_64
    • Ubuntu 20.04, 22.04
  • arm64-sbsa
    • Ubuntu 20.04, 22.04
• MUSA driver
• MUSA Toolkit > = 4.3.3
• mtJPEG >= 1.0.2
• mtJPEG2000 >= 0.0.2
• Python >= 3.10

Manual installation of musa toolkit library

MUSA Toolkit

Manual installation of torch musa library

Torch MUSA

Manual installation of opencv library

mkdir -p ~/opencv_build && cd ~/opencv_build
git clone --branch 4.9.0 --depth 1 https://github.com/opencv/opencv.git
git clone --branch 4.9.0 --depth 1 https://github.com/opencv/opencv_contrib.git

cd ~/opencv_build/opencv
mkdir build && cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
      -D CMAKE_INSTALL_PREFIX=/usr/local \
      -D OPENCV_EXTRA_MODULES_PATH=~/opencv_build/opencv_contrib/modules \
      -D WITH_TBB=ON \
      -D WITH_EIGEN=ON \
      -D WITH_OPENMP=ON \
      -D BUILD_opencv_python3=ON \
      -D BUILD_EXAMPLES=OFF \
      -D BUILD_TESTS=OFF \
      -D OPENCV_ENABLE_NONFREE=ON \
      -D BUILD_NEW_PYTHON_SUPPORT=ON \
      ..

make -j$(nproc)
sudo make install
sudo ldconfig

Build and install from Sources

Additional pre-requisites

• Linux
  • GCC >= 9.4
  • cmake >= 3.18
  • patchelf >= 0.17.2
  • libzstd-dev
• Dependencies for extensions. If you would not like to build particular extension you can skip it.
  • mtJPEG >= 1.0.3
  • mtJPEG2000 >= 0.0.2
  • libjpeg-turbo >= 2.0.0
  • libtiff >= 4.5.0
  • opencv >= 4.9.0
• Python packages:
  • clang==14.0.1
  • libclang
  • pytest
  • opencv-python
  • opencv-contrib-python
  • wheel
  • setuptools
  • sphinx_rtd_theme
  • breathe
  • future
  • flake8
  • sphinx==4.5.0
  • torch
  • torch_musa

Build

Linux

git submodule update --init --recursive --depth 1
mkdir build
pushd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make
popd

Testing

Add directory with libmtimgcodec.so to LD_LIBRARY_PATH

Run sample transcoder app tests

MTIMGCODEC_EXTENSIONS_PATH=$(pwd)/build/extensions/mtjpeg ./build/example/nvimtrans_dec/nvimtrans_dec -i $(pwd)/resources/jpeg/padlock-406986_640_420.jpg -o $(pwd)/output/out.rgb

MTIMGCODEC_EXTENSIONS_PATH=$(pwd)/build/extensions/mtjpeg ./build/example/nvimtrans_enc/nvimtrans_enc -i $(pwd)/resources/jpeg/padlock-406986_640_420.jpg -o $(pwd)/output/out.jpg

Run Python API tests

First install python wheel. You would also need to have installed all Python tests dependencies.

Run tests

export LD_LIBRARY_PATH=/usr/local/musa/lib:$LD_LIBRARY_PATH
export PYTHONPATH="./build/python:$PYTHONPATH"
pytest -v ./test/python/test_code_stream.py