Optimized lighting improves dynamic machine vision, deep learning, and IIoT applications.
Steve Kinney, Director of Engineering, Smart Vision Lights
Like many technology sectors, machine vision continues to advance at a rapid pace. So machine vision lighting technology must constantly evolve to meet increasing demands and to keep up with advancing imaging technologies. Dynamic machine vision systems, the Industrial Internet of Things, and deep learning are areas where lighting innovations have enabled previously unattainable factory applications while moving machine vision beyond the plant floor.
Dynamic Machine Vision and Intelligent Lighting
Dynamic machine vision systems are used in applications with moving or changing objects, fluctuating illumination, or changing viewpoints. Examples include active inspection applications, where robotic activity is inspected with video in real time. Embedded computing devices, cloud computing, 3D nonvisible multi-sensor fusion, artificial intelligence, and event-based imaging are some of the technologies enabling dynamic machine vision. Intelligent lighting is one of the most important technologies enabling dynamic machine vision, however, since lighting must be adaptable and intelligent in today’s applications.
Adaptable means that light properties can be changed dynamically. This includes changing intensity; moving the field of view or focus over a coverage area to highlight a specific object; changing angle or direction, as in the case of photometric stereo; carrying out wavelength or multispectral inspections; and using polarization or other filters that may be switched in and out.
Intelligent means that the system has advanced levels of integration, control, and live data. This technology relies on optimization and plug-and-play integration with cameras and vision systems so that they all work together seamlessly. Intelligent lighting systems also provide data to Industry 4.0 systems that host or control overall processes.
Adaptive lighting can change its field of view (FOV) dynamically in response to the needs of the system. Consider an FOV light with three rows of LEDs, with 10-, 30-, and 50-degree lenses, respectively. If channel 1 is on, the very narrow 10-degree lens is focused, resulting in the maximum projection over the narrowest area. If the 50-degree lens is enabled down the center channel, then the widest field of view is achieved, providing the biggest spread of light over the largest area. As one would expect, the 30-degree channel is somewhere in between. By mixing and matching these channels, an even, uniform coverage can be achieved, all the way from the 10-degree narrow to the 50-degree wide field of view. Unique combinations can be used to tune and optimize lighting for applications in which fixed lighting is insufficient.
Adaptive lighting systems that may be used for color or multispectral analysis include Smart Vision Lights’ RGBW ring light. The standard product includes red/green/blue and a white channel for reference and to increase the color gamut. With these channels, it is possible to create almost any visible color. Tuning enables the optimization of contrast, regardless of the color of the object or part being inspected. This delivers the ability to get exactly the right color needed for the type of inspection being conducted.
Intelligent lights can be modified. For example, to conduct multispectral analysis, a system might need visible color lighting as well as lights in specific bands outside the visible, such as ultraviolet (UV) or near infrared (NIR).
Some light technologies utilize only red, green, and blue, leaving many wondering about the benefit of a white channel. It’s true that red, green, and blue mixed together evenly will indeed produce white, but it’s often not a pure enough white for color or for reference. The white channel provides a white reference for color images outside of the other channels, resulting in a brighter, improved white for a color rendering index (CRI). It also increases the color gamut (the number of available colors).
Intelligent Lighting Standards
Any discussion about intelligent lighting must involve standards that affect how vision systems and components are joined together — that is, Industry 4.0, also called the Industrial Internet of Things (IIoT). The standards that govern these systems have so far been largely focused on cameras and the physical aspects of how data gets from one place to another (e.g., GigE Vision, USB Vision, Camera Link, CoaXPress, and Camera Link HS). But going forward, lighting and optics support will be included in the conversation.
For example, GenICam — the European Machine Vision Association (EMVA) standard —addresses plug-and-play functionality and the sockets for connecting devices in a meaningful, standardized way, so they can communicate with one another. GenICam is now adding support for lighting and optics, enabling communication with cameras and the manipulation of certain parameters so that the host system can control a camera’s lighting and optics seamlessly.
In 2019 the OPC Foundation’s OPC UA (open platform communication unified architecture) specification incorporated GenICam into its communications protocol. In effect, because GenICam is present in all physical standards, it can be used in OPC UA control schemes to facilitate the integration of products.
Dynamic Vision in the Plant
When talking about dynamic machine vision in the manufacturing plant, it’s important to recognize the components that are needed relative to the high bandwidth required. For applications where high bandwidth isn’t a necessity, it's perfectly acceptable to use GigE Vision or USB Vision cameras. But when dealing with high-speed processes with movement in real time, high-speed captures at a very high frame rate are required, necessitating high bandwidth.
Cameras like the MV4 from Photonfocus are capable of capturing many thousands of frames per second and are often coupled with frame grabbers. Frame grabbers can capture and reconstruct images using very little CPU time, freeing up the CPU to work on programming. Programmable lighting controllers further optimize the process by enabling the sequencing and control of lighting in conjunction with a camera and frame grabber to ensure proper image capture.
As plant processes increase in speed, it’s important that lighting can respond. It must turn on and off fast enough for high-speed pulses, which are very short, to capture motion. To support these objectives, Smart Vision Lights has developed NanoDrive, a driver capable of delivering full power to a light in 500 nanoseconds or less. This technology allows lights to be turned on and off within hundreds of nanoseconds, providing sharp, square wave pulses down to the one microsecond range.
Example Application: Automotive Manufacturing
In automotive manufacturing applications, high-speed painting of auto bodies is typically accomplished using robots. It’s becoming increasingly common to also incorporate machine vision technologies to “see” the paint, examine surfaces as they are being painted, and inspect auto bodies as they are exiting the paint booth. In this process, machine-to-machine communication and control are major emerging trends. The complex work of determining when and where spray is dispensed requires real-time feedback and vision-guided robotics.
The cameras are not only watching the spray application in real time but are also monitoring the paint application and providing data, so that software can make dynamic adjustments as the robot arm moves. Such an application requires high-speed capture and a very high frame rate to keep up with the movement of the robot. Similar complex processes can be seen in other multispectral inspection applications, such as kitting/assembly inspection, hard-to-define defect detection, and large-area inspections in textiles.
Example Application: Logistics
A common challenge in logistics applications is detecting and sorting packages of varying sizes, from flat packs to 1-meter boxes. Portal systems are typical in automated warehouses, where companies like Amazon, Walmart, and Target work with extensive shipping and distribution businesses such as FedEx, UPS, and the U.S. Postal Service. But variation in the size of packages on the conveyor— from 1 m3 boxes to small envelopes — can be problematic, because the portal is at a fixed distance and height from the belt. Lighting and focus optimized to catch the wide top of a 3- or 4-foot box very near the camera is different than the lighting and focus required to see a much smaller flat pack or envelope that’s flush on the conveyor.
To address depth of focus, cameras must make adjustments. The solution is a tunable FOV light with wireless control. Using a box height detector/profiler coupled with liquid lens autofocus, the system knows where the surface of a box is and its width as it’s coming down the line. That information is then used to send a control voltage to the liquid lens focus at the right height for the surfaces being presented and to tell the light to position the FOV over the top of the box.
The hardware looks standard, but it is dynamic and being controlled by the other components within the system. It leverages the benefits of advanced LED lighting technology to turn on very fast — in less than 500 nanoseconds — as these boxes move along at a high rate. It provides up to 200,000 lux to achieve the intensity needed to freeze the motion at that rate.
Example Application: Dynamic Color Inspection
Dynamic color inspection is important in applications where a product of a single color must be inspected for blemishes, punctures, gouges, or other defects as it’s being processed and packaged. The solution is using an RGBW light (or a multispectral light if NIR is being used) and a monochrome camera. Monochrome allows for the highest sensitivity and resolution due to the absence of color filters. Variable light allows for optimization of contrast for the different colored products or materials being presented.
For example, a food processor might be running green peppers in the morning and red peppers in the afternoon. Color tunability allows operators to change the product while using the same system and retaining the optimized light wavelength. With a monochrome camera and narrow parameters, one simple algorithm can handle all the pepper colors, giving the highest accuracy for detecting fine details.
Lighting for Deep Learning
AI, deep learning, and the training of smart cameras are other emerging trends in machine vision technology. A common misconception is that deep learning can eliminate or reduce the need for optimized lighting and high-quality images. However, several studies, including one from Arizona State University,1 indicate that this isn’t the case.
In general, although deep neural networks perform on par with (or better than) humans on good-quality images, their performance is much lower on images with quality distortions such as blur and noise. Poor lighting configurations will result in poor feature extraction and poorly trained models. This can result in confusion about the identity of an object or what is, or is not, a defect. After all, AI and deep learning are akin to human learning — and they can’t compensate for poor light.
Beyond the Factory Floor
Dynamic machine vision applications are emerging in new areas like augmented reality (AR) and virtual reality (VR), digital cinema, agriculture, and drone inspections. In France, for example, the agriculture technology company Bilberry uses machine vision to identify weeds in its field-spraying applications. With multiple inspection stations assembled along a spray boom, it can selectively apply pesticide, targeting only the weeds of concern. This enables the company to use less pesticide and reduce pesticide contamination.
Machine vision is also becoming important in entertainment fields. Relighting applications are major components of complex AR/VR platforms like Oculus and Facebook Reality Labs, as well as video conferencing tools and film production. In relighting applications, each light point (there can be hundreds) has a camera and a light associated with it. They light the surface of a scene from all points to gather data and then run massive calculations that enable the platform to place a subject in any kind of environment, even though the model was captured only once.
Lighting: The Heart of Dynamic Machine Vision
At the heart of dynamic machine vision is fast, accurate lighting. It is capable of handling high relative motion within frames requiring brighter light and shorter exposure. Adaptable lighting receives and changes light according to the dynamics of the application. Finally, the networkability of smart lighting makes it ideally suited for IIoT and the standards that govern it.
Steve Kinney is director of training, compliance, and technical solutions at Smart Vision Lights. He is an electrical engineer with many years of experience in machine vision imaging and industrial cameras and imagers. Prior to his current role, Steve held positions at Basler, JAI, and CCS. He is a current member of the A3 board of directors and is a past chairman of A3’s Camera Link Committee.