Image/Video Processors & Design

Image/Video Processors & Design Overview

An image is a visual stimulus for interacting with this world. A video is a compilation of images in sequential order that creates movement. For humans, this is done biologically by rods and cones in our eyes to collect image data. The brain compiles this data and turns it into information. The same thing needs to be done for digital images. Digital images are formatted in a specific way that allows computers to understand what is being shown. These digital images are processed by modifying and classifying pixels or voxels. Digital signal processing, DSP, is the process of using mathematical operators to modify digital signals. Images being functionally matrices or matrices of matrices, mathematical operators like vector multiplication are commonly used to process this data. HLS is beneficial in developing image and video technology because HLS allows for the quick prototyping of hardware modules. This speed is critical in the new and developing image and video processing field. When trying and using different algorithms, these operations can be refined in significant margins by users of HLS.

Basics of Image Processing

Image analysis, enhancement, segmentation, pattern recognition, machine vision, remote sensing, and image and video coding are all forms of digital signal processing. Even though each of these processes requires a different series of mathematical operations, they all are based on matrix manipulation.

When an image is formed, it is expected to think of something you would see taken by a modern camera. Images can be much more or much less than that. A time-of-flight sensor can form an image. Think of an ultrasonic range finder or a lidar. Each of these sensors returns the amount of time and, in theory, the distance an object is from the sensor. If the object is static and the sensor is moved on a flat plane. A topographical image can be formed using a two-dimensional matrix, with each cell containing the sensor’s distance value to the object. This fundamental idea is used in autonomous vehicles to navigate terrain and has been utilized for many years. In recent times, this technology has been improving rapidly, and using arrays of these sensors placed in strategic locations allows for better navigation than ever before.

Depending on the codex, this matrix would contain some color data when considering an image that would come from a complete camera. This data includes physical color but also “shade” and intensity. This shade is where the majority of manipulation occurs. The shade will be referred to as grayscale, and this grayscale transformation is the first step in most algorithms for image manipulation.

The above figure contains the image processing pyramid. This image was created for previous researchers to classify the different levels of image processing. The first level of image processing is preprocessing. This level primarily works specifically with pixels and leverages itself to the next level, which includes regions. When going up the pyramid, the width gets smaller. This is because as more processing is done on an image, the amount of data is reduced, but the information of the object is increased. Suppose we were to take an image of a dog sitting alone on a solid backdrop. The first level would determine that it contains pixels. Moving to the next region, we can determine that there are two distinctly different regions in the image. The next level would contain features. This would include points and corners, the size of pixel clusters, etc. The last section is recognition. This section includes identifying features like ears and tail, but also can determine that the image is a dog and possibly the breed. This scheme is fundamental to the understanding of digital image processing.

Hardware-based signal processing is incredibly fast, and HLS can help. The design of FPGAs is the ideal choice when designing vision systems.

Image/Video Hardware Needs

Every time an image is manipulated, it requires computation. For faster processing, this computation is performed on hardware. An arithmetic logic unit (ALU) is a standard device used for computation. However, as ALUs can only perform one calculation at a time, it is important to use an array of ALUs and parallel processing to achieve better performance. The flexibility of High-Level Synthesis (HLS) and Field-Programmable Gate Arrays (FPGAs) allows users to try different types of hardware modules to determine the best fit for a particular task. Specialized hardware accelerators are increasingly being used, which allow for high-speed and power-efficient computations. Two commonly known hardware accelerators are tensor cores and CUDA cores. These modules are specifically designed to handle the data types used for machine learning (ML) and vision for specific algorithms. For example, tensor cores handle the data type "tensors," which are commonly used in the TensorFlow toolbox. This idea is fundamental for the next generation of products, and HLS can help businesses unlock the possibilities and prototype their own hardware accelerators.

Common-use digital signal processing blocks are still highly efficient at computing. Matrix multiplication is the foundation of image processing. A simple pre-add, multiple, and accumulate pipeline can be used effectively. Another important consideration is memory. RAM and quick access to RAM are critical for video and image computations. HLS can let businesses try different configurations and combinations of these modules quickly to try and find the best option for their application.

In combination with HLS, modern FPGAs can be used to explore and design the hardware needs and considerations for this vastly growing industry.

Real-World Applications

There are many industry and personal applications for image and video signal processing. Navigation is one of the most widely published recent image and video processing topics. This includes real-time vehicle navigation and path planning to send objects to specific locations. Other applications include identification. Being able to identify what objects are in images can be used for a variety of end-use purposes. Tracking who comes in and out of a secure location benefits security. Also, tracking objects in an assembly line allows us to track visual deformities of products and reject them early in the manufacturing process without human intervention. These recognition events don’t only have to be physical. Digital images generated by software can also use this technology. Applications are limitless, and using HLS for quick prototyping allows for possibilities that are only limited by the developers creating them.

Image/Video Processors & Design Conclusion

The field of image and video processing is expanding at a rapid pace, leading to the development of new products at a similar pace. The use of high-level synthesis for rapid prototyping has revolutionized the dynamic nature of this constantly evolving industry. You can follow this page to stay updated on the latest information about image and video processing and its connection with HLS.