Author: Alex Orsholits
In the realm of edge AI computing, inferencing performance is constrained by power consumption, heat generation, and total device footprint. Addressing these challenges requires a delicate balance between optimizing for efficiency and maintaining high-performance computation, which is where dedicated hardware shines. With the emergence of co-processing solutions, highly integrated platforms for autonomous driving tasks that make effective use of limited computational resources are slowly but surely becoming reality.
In response these considerations, Tsukada laboratory has begun to explore the deployment of Hailo-8 AI coprocessors in conjunction with ARM-based RK3588 single board computers (SBCs) for AI vision offloading. The overall platform leverages hardware-accelerated video processing to enable simultaneous and real-time inference of multiple vision models at less than 15W total system load.
With this guide, we aim to provide information for maximizing overall performance of this platform.
1. Environment Installation
To simplify the initial OS installation process for all RK3588 platforms, rather than using official manufacturer images, it is recommended to utilize the following community-managed image of Ubuntu 22.04 as it comes pre-installed with all video and graphics hardware acceleration packages which are not usually included in the default images.
Unified Ubuntu 22.04 for RK3588 SBCs
Please check the Github page for flashing to SD card and installing to onboard eMMC.
Hailo environment preparation
Prior to any Hailo software installation, the system requires at minimum the Dynamic Kernel Module Support
package to be installed. Without this package, the Hailo-8 PCIe driver will brick the entire Ubuntu system, requiring a complete re-install.
sudo apt install dkms
To play it safe, it is possible to run a full package installation as follows:
sudo apt update && sudo apt install adduser apport apt-transport-https apt-utils apt atop base-files base-passwd bash-completion bash bc bcache-tools binfmt-support bison bluez bsdutils btrfs-progs build-essential byobu ca-certificates chromium-browser cloud-guest-utils cloud-initramfs-copymods cloud-initramfs-dyn-netconf cmake console-setup-linux console-setup coreutils cpio cron curl dash dbus-x11 dbus debconf-i18n debconf debianutils debootstrap device-tree-compiler diffutils dirmngr distro-info-data distro-info dkms dmidecode dmsetup dosfstools dpkg dvb-tools e2fsprogs eject ethtool exfat-fuse fake-hwclock fakeroot fdisk ffmpeg findutils flex fonts-ubuntu-console fwupd g++-9 gcc-12-base gcc-9 gcc gdisk gir1.2-glib-2.0 gir1.2-gtk-3.0 git-lfs git gnupg gpgv grep gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-libav gstreamer1.0-plugins-bad gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-pulseaudio gstreamer1.0-qt5 gstreamer1.0-rockchip1 gstreamer1.0-tools gstreamer1.0-x gzip hostname htop hwinfo i2c-tools ifupdown init-system-helpers init initramfs-tools iotop iproute2 iputils-ping ir-keytable isc-dhcp-client isc-dhcp-common iso-codes kbd keyboard-configuration kmod less libacl1 libapparmor1 libapt-pkg6.0 libargon2-1 libatm1 libattr1 libaudit-common libaudit1 libblkid1 libbpf0 libbsd0 libbz2-1.0 libc-bin libc6-dev libc6 libcairo2-dev libcanberra-pulse libcap-ng0 libcap2-bin libcap2 libcom-err2 libcrypt1 libcryptsetup12 libdb5.3 libdbus-1-3 libdebconfclient0 libdevmapper1.02.1 libdns-export1110 libdvbv5-0 libdvbv5-dev libdvbv5-doc libegl-mesa0 libegl1-mesa-dev libelf-dev libelf1 libestr0 libexpat1 libext2fs2 libfastjson4 libffi8 libfribidi0 libgbm-dev libgcc-s1 libgcrypt20 libgirepository-1.0-1 libgirepository1.0-dev libgl1-mesa-dev libgles2-mesa-dev libglib2.0-0 libglib2.0-data libglx-mesa0 libgmp10 libgnutls30 libgpg-error0 libgssapi-krb5-2 libgstreamer-plugins-bad1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev libhogweed6 libicu70 libidn2-0 libip4tc2 libisc-export1105 libjson-c5 libk5crypto3 libkeyutils1 libkmod2 libkrb5-3 libkrb5support0 liblocale-gettext-perl liblz4-1 liblzma5 libmd0 libmnl0 libmount1 libmpdec3 libncurses6 libncursesw6 libnetplan0 libnettle8 libnewt0.52 libnsl2 libnss-systemd libp11-kit0 libpam-cap libpam-modules-bin libpam-modules libpam-runtime libpam-systemd libpam0g libpcre2-8-0 libpcre3 libpopt0 libprocps8 libpython3-stdlib libpython3.10-minimal libpython3.10-stdlib libreadline8 librga-dev librga2 librist-dev librist4 librockchip-mpp-dev librockchip-mpp1 librockchip-vpu0 libseccomp2 libselinux1 libsemanage-common libsemanage2 libsepol2 libslang2 libsmartcols1 libsqlite3-0 libss2 libssl-dev libssl3 libstdc++6 libsystemd0 libtasn1-6 libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl libtinfo6 libtirpc-common libtirpc3 libudev1 libunistring2 libuuid1 libv4l-0 libv4l-dev libv4l-rkmpp libv4l2rds0 libv4lconvert0 libwidevinecdm libxml2 libxtables12 libxxhash0 libyaml-0-2 libzmq3-dev libzstd1 linux-rockchip-5.10 lm-sensors locales login logrotate logsave lsb-base lsb-release lshw lsof lvm2 lxd-agent-loader make mali-g610-firmware malirun man-db manpages mawk mdadm media-types mesa-common-dev mesa-utils mesa-vulkan-drivers mmc-utils motd-news-config mount mpv mtd-utils mtools multipath-tools nano ncurses-base ncurses-bin net-tools netbase netcat-openbsd netplan.io networkd-dispatcher ntfs-3g open-iscsi open-vm-tools openssh-client openssh-server openssl p7zip-full parted passwd patch pavucontrol pciutils perl-base pigz pollinate procps pulseaudio python-apt-common python-gi-dev python-is-python3 python2 python3-apt python3-dbus python3-dev python3-gi-cairo python3-gi python3-minimal python3-netifaces python3-pip python3-pkg-resources python3-setuptools python3-virtualenv python3-yaml python3.10-minimal python3.10 python3 qemu-efi qemu-system-arm qemu-user-static qt6-qpa-plugins qt6-wayland qtwayland5 qv4l2 readline-common rfkill rist-tools rockchip-firmware rockchip-mpp-demos rockchip-multimedia-config rsync rsyslog screen sed sensible-utils shared-mime-info snapd software-properties-common sosreport sudo systemd-sysv systemd-timesyncd systemd sysvinit-utils tar tmux tzdata u-boot-rock-5b u-boot-tools ubuntu-advantage-tools ubuntu-desktop ubuntu-drivers-common ubuntu-keyring ubuntu-minimal ucf udev unattended-upgrades update-notifier-common usb-modeswitch-data usb-modeswitch usbutils usrmerge util-linux uuid-runtime v4l-utils vim-common vim-tiny vim virtualenv wget whiptail wireless-regdb wireless-tools wpasupplicant x11-utils xcb xdg-user-dirs xfsprogs xkb-data xterm xxd xz-utils zip zlib1g
This will ensure that all dependencies for hailo software compilation will be installed.
2. Installation of HailoRT PCIe driver and control software, Tappas
While Hailo provides a Docker container with all required software and dependencies, it does not allow us to leverage the benefits of hardware acceleration for video data handling. We must therefore build everything from source. The workflow is as follows:
- Install HailoRT PCIe driver package (compiling the package from source does not install some needed files for unknown reasons)
- Compile and install from source HailoRT control software
- Compile and install from source corresponding HailoRT PCIe driver
- Compile and install from source corresponding Tappas software
- Compile and install Rockchip-specific 2D engine acceleration for GStreamer
Please note that Hailo software releases are version-specific, meaning that only a specific version of the PCIe driver will work with a specific version of the control software, and this extends to Tappas.
Install HailoRT PCIe driver package (December 2023 v4.16.0)
To get things started, we need to install the PCIe driver to ensure that the required files for the self-compiled driver will function. You must have access to the Hailo Developer Zone to download the latest driver: https://hailo.ai/developer-zone/software-downloads/
To install, simply run:
sudo dpkg -i hailort-pcie-driver_4.16.0_all.deb
You will be prompted whether you want to install the driver using DKMS. Say Y(es).
Do you wish to use DKMS? [Y/n]:
After installation, reboot:
sudo reboot now
Compile and install from source HailoRT control software
We will start with the HailoRT control software which is currently one version ahead of the pre-compiled release.
HailoRT control software (GitHub)
Start by downloading the source code using git.
git clone https://github.com/hailo-ai/hailort.git && cd hailort
Depending on when (or if) you follow this guide, the branches will most probably be updated to the latest source release. To ensure compatibility, you should select a specific version branch for compilation. To achieve this, you can checkout a specific branch to work with utilizing the following command (while inside of the project folder):
(for this guide we will use v4.16.1)
git checkout master-vx.xx.x
Now that everything is setup, the control software can be compiled with the following command:
cmake -H. -Bbuild -DCMAKE_BUILD_TYPE=Release && sudo cmake --build build --config release --target install
If all dependencies have been properly installed, this command should configure, build, and finally install the software. Verify it is working by running the following command:
hailortcli -v
Compile and install from source corresponding HailoRT PCIe driver
Now that the control software is installed from source, we need to upgrade our PCIe driver to the same version as the control software.
HailoRT PCIe driver (GitHub)
Start by downloading the source code using git.
git clone https://github.com/hailo-ai/hailort-drivers.git && cd hailort-drivers
Just like the HailoRT control software, there are version-specific branches. Therefore we must checkout the same branch to ensure that we have corresponding version numbers:
(for this guide we will use v4.16.1)
git checkout master-vx.xx.x
To compile the driver, we need to run make
from within the pcie folder. Do the following:
cd linux/pcie
make all
sudo make install_dkms
sudo modprobe hailo_pci
Next, we need to return to the root folder of the GitHub project (cd ../..
)
And download the co-processor binary firmware blob corresponding to the driver version. Run the following script:
./download_firmware.sh
And move the downloaded blob and udev rules:
sudo mv hailo8_fw.hailo8_fw.4.16.1.bin /lib/firmware/hailo/hailo8_fw.bin
sudo cp ./linux/pcie/51-hailo-udev.rules /etc/udev/rules.d/
and finally reboot.
After rebooting, both the hailoRT control software and the corresponding PCIe driver should be installed. It should therefore be possible to communicate with the co-processor now. Verify this fact by running the following command:
hailortcli fw-control identify
This should return the serial number of the installed co-processor, among other things.
Compile and install from source corresponding Tappas software
Tappas provides various GStreamer-based pipelines for AI inferencing, as well as a guide for implementing our own pipelines that connect to python or C++, or generate processed video frames.
Tappas (GitHub)
Start by downloading the source code using git.
git clone https://github.com/hailo-ai/tappas.git && cd tappas
Unlike the HailoRT package and driver, Tappas utilizes a different version numbering schema. The compatible HailoRT version is only summarily noted in the GitHub readme. Again, it is important to select a specific branch for compatibility purposes!
(for this guide we will use v3.27.1)
git checkout vx.xx.x
Before starting, we need some extra files from the HailoRT github project to ensure that Tappas can be compiled on ARM64 platforms. After cloning the Tappas repository and setting the specific branch, we need to clone in the HailoRT files as follows:
mkdir hailort && git clone https://github.com/hailo-ai/hailort.git hailort/sources
Tappas requires several dependencies to be installed prior to compilation, namely OpenCV 4.5.2, GStreamer, and PyGobject.
OpenCV 4.5.2
First we need to download and unzip and configure the specific OpenCV version Tappas requires:
wget https://github.com/opencv/opencv/archive/4.5.2.zip
unzip 4.5.2.zip && cd opencv-4.5.2
mkdir build && cd build
cmake -DOPENCV_GENERATE_PKGCONFIG=ON -DBUILD_LIST=core,imgproc,imgcodecs,calib3d,features2d,flann -DCMAKE_BUILD_TYPE=RELEASE -DWITH_PROTOBUF=OFF -DWITH_QUIRC=OFF -DWITH_WEBP=OFF -DWITH_OPENJPEG=OFF -DWITH_GSTREAMER=OFF -DWITH_GTK=OFF -DOPENCV_DNN_OPENCL=OFF -DBUILD_opencv_python2=OFF -DINSTALL_C_EXAMPLES=ON -DINSTALL_PYTHON_EXAMPLES=ON -DCMAKE_INSTALL_PREFIX=/usr/local ..
Compile and install:
make -j8
sudo make install
sudo ldconfig
GStreamer
If you haven’t already installed all the packages from the environment preparation step, then do so here:
sudo apt install -y libcairo2-dev libgirepository1.0-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio gcc-9 g++-9 python-gi-dev
PyGobject
If you haven’t already installed all the packages from the environment preparation step, then do so here:
sudo apt install -y python3-gi python3-gi-cairo gir1.2-gtk-3.0
Tappas
Be sure to return to the root directory of the Tappas git and run the install script which will (hopefully) automatically handle compilation. There is now a specific build target for the Rockchip RK3588 platform, you can run the install script with this flag, if you want. The only difference is that it will only download the rockchip-specific AI inference pipeline examples that were tested on the platform, which is a far cry from the total amount of available examples.
./install.sh --skip-hailort --target-platform rockchip
Tappas should now be installed!
Compile and install Rockchip-specific 2D engine acceleration for GStreamer
Most of the Rockchip SoCs contain a 2D engine that enables hardware-accelerated cropping and scaling of video and image frames. To enable real-time multistream AI inferencing on the RK3588 platform, we must be able to use these functions to maximize performance, as many of the vision AI models require specific input resolutions, such as YoloV5m @ 640×640. This being said:
At present, some platforms such as RK3588, RGA function is abnormal, so it is not recommended to use it.
Indeed, when trying to enable the 2D engine as a general feature for GStreamer, there is a nearly 99% chance of specific pipelines crashing, causing the co-processor to hard lock up. To ensure that we can make stable pipelines for all situations, we will compile our own GStreamer plugin that enables us to use non-hardware-accelerated videoconvert
in conjunction with hardware-accelerated rgaconvert.
gstreamer-rgaconvert plugin source
Start by downloading the source code using git.
git clone https://github.com/higithubhi/gstreamer-rgaconvert.git && cd gstreamer-rgaconvert
Make sure meson
is installed:
sudo apt install meson
Setup the build folder and compile:
mkdir build && meson setup build
cd build && ninja
Copy the compiled plugin files to the GStreamer folder:
sudo cp plugins/libgstrgaconvert.so /usr/lib/$(uname -m)-linux-gnu/gstreamer-1.0/
sudo cp -r plugins/libgstrgaconvert.so.p /usr/lib/$(uname -m)-linux-gnu/gstreamer-1.0/libgstrgaconvert.so.p
Finally, verify that the plugin is detected by GStreamer by running the following command:
gst-inspect-1.0 rgaconvert
You should get the following output:
Everything is now installed for AI inferencing!
Running the example GStreamer pipelines
Tappas provides example pipelines for detection, license plate recognition, multistream detection, and tiling under the following path:
<tappas folder>/apps/h8/gstreamer/rockchip/
We will only provide a quick summary using the detection
example which is a yolo-based single network pipeline (yolov3, yolov4, yolov5, nanodet).
contained within this detection folder is a detection.sh
script which effectively generates a GStreamer bash string based on some user-configurable settings. You can see which settings can be changed by running ./detection.sh --help
An extremely useful parameter is the --print-gst-launch
variable which allows us to print the entire pipeline string without running it. Let’s do that and then work on modifying it to include hardware acceleration!
gst-launch-1.0 filesrc location=/home/hailo/Projects/tappas/apps/h8/gstreamer/rockchip/detection/resources/detection.mp4 name=src_0 ! decodebin ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! videoscale qos=false n-threads=2 ! video/x-raw, pixel-aspect-ratio=1/1 ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! videoconvert n-threads=2 qos=false ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailonet hef-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/rockchip/detection/resources/yolov5m_wo_spp_60p.hef batch-size=1 nms-score-threshold=0.3 nms-iou-threshold=0.45 output-format-type=HAILO_FORMAT_TYPE_FLOAT32 ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailofilter function-name=yolov5 so-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/libs/post_processes//libyolo_hailortpp_post.so config-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/rockchip/detection/resources/configs/yolov5.json qos=false ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailooverlay qos=false ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! videoconvert n-threads=2 qos=false ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! fpsdisplaysink video-sink=xvimagesink text-overlay=false name=hailo_display sync=false
Let’s look at what this pipeline is doing (with queues omitted):
An input video file is being fed into the pipeline via filesrc
and decoded with decodebin
. To meet the requirements of the yolov5 model, the video source is resized and cropped via videoscale
to match the 640×640 model resolution. This is then sent off to the Hailo inferencing pipeline after receiving a colorspace conversion via the videoconvert
plugin from NV12 to F32. The Hailo system consists of three GStreamer plugins in the case of this detection pipeline: hailonet
, hailofilter
, and hailooverlay
.
Note, for a list of all Hailo GStreamer plugins, please refer to the documentation in the Tappas GitHub repository here.
The use of hailonet
is self-explanatory. It is the plugin for loading in the AI models. hailofilter
on the other hand represents the point of contact for other software to access AI inference data. It can either load in a compiled C++ file or a python script with the goal of providing an interface for handling object detection information. Finally, hailooverlay
works in conjunction with hailofilter
to overlay on the video the detections via bounding boxes. All of this is then output in a 640×640 xvimagesink
after yet another colorspace conversion back to NV12.
This pipeline already takes advantage of the hardware decoding features of the RK3588 and works completely fine for single video sources up to 3840×2160@25fps. That being said, we can do better.
///////////////////////////////////////////////////////////////////
// Note: the act of printing the fps causes a processing slow-down
///////////////////////////////////////////////////////////////////
//640x640 input NORGACONVERT
GstPipeline:pipeline0/GstFPSDisplaySink:hailo_display: last-message = rendered: 718, dropped: 0, current: 71.65, average: 70.90 //fps
//3840x2160 input NORGACONVERT
GstPipeline:pipeline0/GstFPSDisplaySink:hailo_display: last-message = rendered: 517, dropped: 0, current: 26.48, average: 24.84
/////////////////////////////////////////////////////////////////////////////////////////////////////
///////////////////////////////// Now with 2D hardware acceleration /////////////////////////////////
/////////////////////////////////////////////////////////////////////////////////////////////////////
//640x640 input WITHRGACONVERT
GstPipeline:pipeline0/GstFPSDisplaySink:hailo_display: last-message = rendered: 651, dropped: 0, current: 218.23, average: 216.01
//3840x2160 input WITHRGACONVERT
GstPipeline:pipeline0/GstFPSDisplaySink:hailo_display: last-message = rendered: 521, dropped: 0, current: 98.66, average: 103.00
Hardware-accelerated filesrc pipeline
Firstly, let’s explicitly ensure that we are decoding all input files with the hardware decoder. To achieve this, we need to replace decodebin
with two plugins, parsebin
and mppvideodec
This changes the start of the pipeline as follows:
gst-launch-1.0 filesrc location=<file> name=src_0 ! parsebin ! mppvideodec ! [...]
The next step is to replace videoscale
and videoconvert
with sub-pipelines utilizing rgaconvert
instead. Let’s first tackle the video scaling down to 640×360, the addition of black bars to make a 640×640 video, and finally the conversion of the colorspace to the proper format. To ensure that rgaconvert
does not fail or crash, we need to be more verbose when writing the pipeline. We therefore need to explicitly include our rescaling goals. For example, to scale down a video to 640×360, the rgaconvert
pipeline would be as follows:
[...] ! rgaconvert qos=false ! video/x-raw,format=RGB,width=640,height=360 ! [...]
To add the black bars and make a 640×640 frame that is compatible with rgaconvert
, for this specific detection pipeline, the use of videobox
presented a viable solution. It crops the video based on explicitly defined capabilities and hands it off to a second rgaconvert
which implicitly converts the colourspace to F32 for the subsequent Hailo sub-pipeline.
[...] ! videobox autocrop=true ! video/x-raw,format=RGB,width=640,height=640 ! rgaconvert ! hailonet [...]
A limitation of rgaconvert seems to be that it can be used only a few times in a single pipeline before it becomes unstable and prone to crashing. As such, the second videoconvert in the pipeline is difficult to replace with a third rgaconvert. What we get is the following pipeline:
gst-launch-1.0 -v filesrc location=/home/hailo/Projects/tappas/apps/h8/gstreamer/rockchip/detection/resources/detection.mp4 name=src_0 ! parsebin ! mppvideodec ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! rgaconvert qos=false ! video/x-raw,format=RGB,width=640,height=360 ! videobox autocrop=true ! video/x-raw,format=RGB,width=640,height=640 ! rgaconvert ! hailonet hef-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/rockchip/detection/resources/yolov5m_wo_spp_60p.hef batch-size=1 nms-score-threshold=0.3 nms-iou-threshold=0.45 output-format-type=HAILO_FORMAT_TYPE_FLOAT32 ! hailofilter function-name=yolov5 so-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/libs/post_processes//libyolo_hailortpp_post.so config-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/rockchip/detection/resources/configs/yolov5.json qos=false ! hailooverlay qos=false ! queue leaky=no max-size-buffers=60 max-size-bytes=0 max-size-time=0 ! videoconvert n-threads=8 qos=false ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! fpsdisplaysink video-sink=xvimagesink text-overlay=false name=hailo_display sync=false | grep -e hailo_display -e hailodevicestats
Hardware-accelerated filesrc pipeline with full-resolution output
We can also take advantage of the hailo muxing plugins to run an inference pass at the required model input video scale, and apply said inference to the original full resolution video. Here is a quick example of a face detection pipeline being accelerated with rgaconvert.
Original:
gst-launch-1.0 -v filesrc location=~/Projects/tappas/apps/h8/gstreamer/resources/mp4/detection7.mp4 name=src_0 ! parsebin ! mppvideodec ! videoconvert n-threads=8 ! tee name=t hailomuxer name=mux t. ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! mux. t. ! videoscale ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailonet hef-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/resources/hef/lightface_slim.hef ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailofilter function-name=lightface so-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/libs/post_processes/libface_detection_post.so config-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/general/face_detection/resources/configs/lightface.json qos=false ! mux. mux. ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailooverlay ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! videoconvert ! fpsdisplaysink video-sink=xvimagesink name=hailo_display sync=false text-overlay=false
Accelerated version:
gst-launch-1.0 -v filesrc location=~/Projects/tappas/apps/h8/gstreamer/resources/mp4/detection7.mp4 name=src_0 ! parsebin ! mppvideodec format=NV12 ! tee name=t hailomuxer name=mux t. ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! mux. t. ! rgaconvert ! video/x-raw,format=RGB,width=320,height=240 ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailonet hef-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/resources/hef/lightface_slim.hef ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailofilter function-name=lightface so-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/libs/post_processes/libface_detection_post.so config-path=/home/hailo/Projects/tappas/apps/h8/gstreamer/general/face_detection/resources/configs/lightface.json qos=false ! mux. mux. ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailooverlay ! queue leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! rgaconvert ! video/x-raw,format=NV12,width=1920,height=1080 ! fpsdisplaysink video-sink=xvimagesink name=hailo_display sync=false text-overlay=false | grep hailo_display
Here, the raster graphic accelerator (RGA) increases the frames per second from an average of 19.54fps to 162.24fps for an input 1080p h264 file.
For those looking to deploy similar hardware, we hope that our guide proves useful. We are excited to see what applications are developed in the near future!