Imaging with hyperspectral sensors: the right design for your application Frederik Schönebeck Framos GmbH f.schoenebeck@framos.com June 29, 2017 Abstract In many vision applications the relevant information is encoded into the color of the scenery. In normal color cameras this information is extracted based on the three standard color channels, red, blue, and green (RGB), respectively. With this technique color reproduction is only approximative and often insufficient to solve the vision proplem reliably. Hyperspectral imaging overcomes this limitation by providing a much greater number of spectral bands, while keeping the spatial resolution still acceptable. With recent progress in sensor design, the narrow-band spectral filters can be implemented at sensor level. With these sensors the level of complexity of hyperspectral cameras can be dramatically decreased. In turn, this enables compact, reliable, and easy-touse hyperspectral cameras that can benefit virtually any application in which accurate color information is key to success. I. Introduction C olor is one of the key parameters in many vision applications and is often used as basis for classification, background / foreground discrimination, or object identification. To capture color, cameras are typically equipped with three broadband color channels, red, green, and blue (RGB), which are implemented in the form of a regular, mosaic-like filter pattern, the so-called Bayer-pattern. With only these three standard filters the resulting color information is merely approximate and often insufficient to characterize subtle color gradients reliably. Robust color discrimination, however, is often key to success (e.g., the robust discrimination between tissue, nerves, and blood vessels during noninvasive surgery) and thus the performance of conventional color sensors hampers visionbased solutions in many applications. By contrast, hyperspectral imaging, or imag1 ing spectroscopy, is a combination of digital imaging and narrow-band spectroscopy. This technique allows to capture for each pixel on the detector the light intensity for a great number of spectral bands (typically some tens to hundreds). As a consequence each pixel in the image contains a full color spectrum (as opposed to only three values for red, green, and blue), which in turn can be used to characterize the scenery with great color detail and accuracy. This enables object classification pipelines based on spectral properties via statistical matching or neural networks, thereby openening up entirely new approaches in the vision industry. With recent progress in sensor design and processing speed, a wide field of applications can now benefit from hyperspectral imaging. Applications range from industrial piece inspection, over specimen classification in medicine and biophysics, to airborne remote sensing and military target detection. This article will explain the two most common operating principles of hyperspectral cameras and
highlight the main applications in which they are typically used. II. Principles of hyperspectral cameras The spectral decomposition of light is traditionally achieved with a narrow slit in combination with a number of dispersive optical elements. While this approach enables a high spectral accuracy, the necessary optical setup inside the camera is elaborate and complex. This can lead to a large camera footprint, unreliable performance, and increased costs. In recent years advances in sensor design enabled the implementation of precisely tuned narrow-band spectral filters at pixel-level. In contrast to the three filters used in conventional color sensors with Bayer patterns, hyperspectral sensors have filter patterns that sample the full spectral range with a great number of evenly distributed narrow band filters. Depending on the application, this range can span from the ultra-violet to the near-infrared, and might be even significantly beyond the perception of the human eye. Based on the characteristics of the utilized filter patterns, the operation modes of hyperspectral cameras can be divided into two main categories, snapshot mosaic and pushbroom scanning, both imposing different requirements on the application setup. i. Pushbroom Scanning Relative motion between camera and imaged object is often a natural requirement in many vision applications. Typical examples include piece inspection on conveyor belts, remote sensing via aircraft or satellites, or autonomous agriculture via unmanned ground vehicles. These kinds of applications are best addressed with pusbroom scan hyperspectral cameras, in which contigous pixel rows of the image sensor are coated with spectrally adjacent narrowband filters. The relative motion between cam- 2 era and scenery causes the object to effectively drift over the image sensor. By synchronizing the sensor line read-out to the relative motion speed, the scenery is imaged line-by-line, or, due to the row-wise filter coating of the sensor, with one spectral band after the other. The full spectrum is obtained once the object has passed completely over the sensor. Hence, on the two-dimensional area sensor, the pixel rows are used for imaging one spatial dimension, while the columns capture the spectral dimension. The second spatial dimension is obtained from the relative motion of camera and scenery, the so-called pushbroom scan. This working princple is illustrated in Figure 1. The number of spectral bands in pushbroom scanning hyperspectral cameras is typically higher than one hundred. This yields very detailed spectral information, which in turn enables more reliable identification and classification results. The obtainable spatial resolution is very high, as it is given by the raw resolution of the sensor along one dimension (typically 2048 4096 pixels) and the scanning speed along the other dimension. On the other hand, the high spectral and spatial resolutions come at the expense of the scanning requirement and might lead to more complicated application setups. It should be emphasized, however, that the scanning is often intrinsic to the application and therefore does not necessarily have to be considered a disadvantage. ii. Snapshot Mosaic Snapshot mosaic hyperspectral cameras are very similar to standard color cameras. The filter coating is arranged as a mosaic of repetitive tiles, but, contrary to the 2 2 Bayer pattern, these tiles typically consist of 4 4 or 5 5 pixels. In each of these tiles the individual pixels are coated with narrowly-defined bandpass filters (compare Figure 3), and thus the number of spectral bands is significantly increased compared to the traditional red, green, and blue color channels. It is important to note that this gain in spectral information is accompanied with a decrease in spatial resolution, which
Figure 1: Sketch of the CMOSIS CMV2000 sensor with a narrow band filter pattern aligned with the sensor rows. Each spectral band covers an area given by the full width (2048 pixels) multiplied by 8 rows. The covered spectral range from 600nm to 1000nm is sampled with 100 spectral bands. The modified sensor is provided by imec and implemented by a number of camera manufacturers. Image credits: XIMEA https://www. ximea.com/support/attachments/4675/ximea_imec_hsi_technology-part-v1.1.pdf Figure 2: Sketch of the CMOSIS CMV2000 sensor with repetitive 4 4 pixel tiles. In each tile the full spectral range between 465nm and 630nm is sampled with 16 spectral bands. The modified sensor is provided by imec and implemented by a number of camera manufacturers. Image credits: XIMEA https://www.ximea.com/ support/attachments/4675/ximea_imec_hsi_technology-part-v1.1.pdf results from the large size of the individual tiles in the filter mosaic. Resulting raw resolutions are 3
typically on the order of 500 250 pixels, but can be increased with sophisticated interpolation algorithms. As the name suggests, the complete spatial and spectral information can be obtained in one snapshot, and thus snapshot mosaic hyperspectral cameras can also be used for conventional video acquisition or other applications, where scanning is not applicable. As a consequence, snapshot mosaic hyperspectral cameras are very versatile and can be easily integrated into virtually any application, in which conventional color cameras are traditionally used. These include quality inspection, food sorting, tissue analysis, endoscopy, or microscopy. The only downside of this ease-of-use is the limited number of spectral bands of approx. 20 compared to the 100+ of pushbroom scanning cameras, which, however, is often still sufficient to address imaging problems that could not be solved with normal color cameras. III. Conclusion Using semi-conductor thin-film processing it is now possible to implement narrow-band spectral filters at pixel-level. With this technology hyperspectral cameras can be realized as reliable, compact, and easy-to-use systems that can be integrated into many different applications. These may range from precision agriculture supported by unmanned vehicles, over robust discrimination between tissue, nerves, and blood vessels during noninvasive surgery, to significant improvements in food sorting or quality inspection. In particular when combined with powerful computing approaches like neural networks, which are capable of analyzing and extracting the desired information from the vast amounts of raw data, hyperspectral cameras will boost virtually all applications in which the color of the object plays a crucial role. iii. Further considerations Both sensor-designs explained above do not put any special constraints on the utilized lenses, other than a high transmission and low chromatic abberation over the spectral range of interest. As a consequence, cameras with hyperspectral image sensors are readily equipped with existing pro grade machine vision lenses. The output of hyperspectral cameras comes in the form of 3D data cubes, with two spatial and one spectral dimension, i.e., a full spectrum for each pixel. The concept of such a data cube is depicted in Figure 3, where x and y represent the well-known spatial dimensions of the image and the vertically arranged λ1..n represent the n spectral bands. It should be noted that a significant amount of image post processing is required to transform the raw information into usable data, which can then be used for object identification or classification. 4
Figure 3: 3D data cube, with x and y representing spatial dimensions and λ 1..n depicting n spectral bands. Image credits: XIMEA https://www.ximea.com/support/ attachments/4675/ximea_imec_hsi_ technology-part-v1.1.pdf 5