Trends

Lisa A Cassis², Aaron Urbas¹ and Robert A Lodder¹ 

Robert A Lodder Email: lodder@uky.edu

Received: 18 October 2004  Revised: 9 November 2004  Accepted: 10 November 2004  

In the past, optics has served mainly to render the world more easily visible to humans. Now, computers are increasingly employed to make sense of the visual world in ways that people cannot. With a new generation of optics, scientists and engineers are recasting visual scenes for interpretation exclusively by computers. To the human eye, these pictures appear distorted at best, or at worst look like visual noise, without discernable meaning. But to computers, such data are worth more than a thousand words. Optimizing complete vision-and-action systems for computers lies at the core of integrated computational imaging. Computers are well-established manipulators of digitized images, and image-processing programs do it routinely on desktop machines. However, what is new is the strategy of modifying image information as it is sensed to make it better suited for the computer mind [1, 2].

For example, rather than the customary concave and convex disks, optical engineers are fabricating strangely shaped, fundamentally different lenses adapted to the strong points of computers. These optics diverge from the traditional approach in which lenses form something humans recognize as an image. In nature, some beetles navigate by detecting certain colors or the polarization of light in air without forming an image from the data. Scientists have been slow to explore such alternatives, however, because they have modeled optical instruments such as cameras after our own image-rendering eyes.

The revolution in integrated computational imaging extends beyond just lenses, however. A new trend in hyperspectral imaging is to speed the visual data processing and reduce data storage requirements by downloading some of the computation to the sensing detector itself. In many cases the detector array can perform both feature extraction (of both physical and spectral features) and encoding of these features. The codes are transmitted by the array to the computer, integrating the computation and imaging (ICI) to reduce the huge data load and speed the processing. Similarly, molecular computing in a multiplex image bandpass spectrometer can accomplish hyperspectral imaging as spatial integrated computational imaging performs feature extraction [2].

A simple dueling analogy suggests the advantage of doing as much of the processing as possible in the sensor. Imagine two swordsmen in a fight. The first swordsmans hand and sword are controlled by his brain using image information transmitted from the retinas of his eyes. Impulses must travel from his eye to his brain, and then from his brain to his hand. The second swordsmans hand and sword are controlled directly by the retinas of his eyes using nerve impulses that travel only one path instead of two. The second swordsmans weapon is likely to always be a bit ahead of the firsts. Moreover, the second swordsmans brain is free to consider other strategies.

Both spatial and spectral features of samples can be encoded in ICI. When spectral images are simultaneously obtained and encoded at many different wavelengths, the process is termed hyperspectral integrated computational imaging (HICI). Molecular absorption filters can be used as mathematical factors in spectral encoding to create a factor-analytic optical calibration in a high-throughput spectrometer. In this system of molecular computing, the molecules in the filter effectively compute the calibration function by weighting the signals received at each wavelength over a broad wavelength range. Lenslet arrays and masks can also be employed to encode spatial features of a hyperspectral image. Spectrometer designs are possible that use molecular-computing to replace traditional principal component analysis in a computer with molecular filters (MFs) tailored to produce factor scores at the detector. Spectrometer designs that use lenslet arrays to extract and encode selected image features are also being produced.

Given a set of training spectra collected at all available wavelengths (see Fig. 1, left side), it is possible to rationally select MF materials to perform PCA (see Fig. 1, right side). PCA is designed to maximize the signals from the spectral regions with the most variability by most heavily weighting them (with the loadings line in the left graph in Fig. 1) in calibration. However, PC loadings heavily weight signals in the positive and negative direction, which cannot be done with MFs without offsetting signal gained at one wavelength with signal lost at another wavelength. Because only absolute values can be represented in MFs, two filters are needed for a PC, one for the positive loadings (MF1) and one for the negative loadings (MF2). The filter materials can be selected by examining the sample spectra. The transmission spectrum (%T) of the filter material should be as similar as possible to the absolute value of the loadings spectrum being targeted.

Fig. 1  Principal components (PCs) of spectral data are formed from loadings vectors (left). The highest loadings correspond to wavelengths where the variation of interest in the sample spectra is greatest. The variations can be captured optically by selecting molecular filter (MF) substances with transmission spectra similar to the loadings. If there are positive and negative loadings in the MFC bandpass, two MFs must be employed for that PC to avoid ambiguity

---

Bandpass filters should be selected to ignore regions of the spectrum where there is no difference between the training spectra, as extra photons in those regions simply saturate the detector or add noise without providing any additional signal. The MF filters do not have to be featureless in the areas away from their peaks in the pictures above as long as bandpass filters (or prisms or gratings) are used to wipe out the %T peaks in undesired areas. In the infrared region, radiation sources like the synchrotron are ideal for near-field microspectrometry with molecular computing because the bright, collimated beam has uniform intensity across the spectrum.

Current sensor system architectures detect signals from a stimulus, convert them to electrical signals, convert the electrical signals to digital form for processing by computers, and, finally, extract critical information from the processed signals for utilization. Integrated sensing and processing (ISP), an initiative launched in the Defense Advanced Research Projects Agency (DARPA), seeks to exchange this chain of processes, each optimized individually, with new methods for crafting sensor systems that treat the total structure as a single end-to-end process that can be optimized globally [3].

The military rationale for ICI parallels the scientific one. In the 21st century global information dominance is necessary to protect US air, space, and ground assets. Sensor systems like synthetic aperture radar (SAR) and IR video collect unprecedented amounts of data, greater than 10¹² pixels/day that require more than 10¹⁶ flops/day to process. At the same time, the downsizing personnel trend persists and the ratio of pixels to pupils is heading toward infinity. These trends combine to make training data collection, processing, downlink and distribution all problematic as the US military seeks ways to rapidly reduce data from physical fields to high-level information. At the same time, computing resources are limited in size, weight, power, and cost. Application-specific integrated circuits (ASICs) do not really help because they solve a fixed problem in a changing sensor/target environment. ASIC design time and cost tend to be prohibitive. More flexible detection schemes like the digital micromirror array (DMA) measure features, not pixels, under computer control. This holistic approach boosts signal-to-noise ratio (SNR) and concentrates information.

Algorithms for both design and operation of sensor systems are being constructed that permit back-end exploitation processes, such as target identification and tracking, to automatically organize and establish the operating modes of sensor elements to guarantee the most relevant data are always being gathered as circumstances and settings evolve. The ISP program approach is enabling an order-of-magnitude performance enhancement in detection sensitivity and target classification accuracy with no change in computational cost, across a broad assortment of DoD sensor systems and networks—from surveillance to radar, sonar, optical and other weapon guidance systems. ISP has produced statistical methods to apportion the sensing channels in a configurable chemical sensor and developed feedback tactics to supervise the elements of an adaptive optical sensing system. ISP has invented new mathematical frameworks for global optimization of design and operation of a number of different types of sensor systems. It is also implementing its software prototypes of the new methodology in test-bed hardware systems, such as missile guidance and automatic ground target recognition modules. ICI will bring these same benefits to chemical analysis.

ICI researchers are conferring extensive depth of field on microscopes and other optical instruments [4]. Optical engineers are developing novel optics to assist computers in sensing motion and the physical and chemical properties of distant objects. Engineers are designing similar lenses that can manage other segments of the electromagnetic spectrum, enlarging the broad transformation in progress in the way scientists look at sensing. Standard cameras, microscopes, and other optical instruments use collections of convex and concave lenses to focus light onto planar sections of film or electronic detectors. For example, an autofocus camera classically moves the positions of certain optical elements forward and backward until a sensor that scrutinizes contrast variations in the field of view perceives satisfactory detail. Eliminating autofocusing and reducing component count begins by considering any scene observed through a lens as a montage of small points of illumination. Paradoxically, abolishing autofocusing systems depends on a defocusing lens. Rather than using a movable convex lens to focus light, a saddle-shaped lens is held stationary. This fixed lens contributes an apparently blurred image to a computer, which then runs a program that rebuilds the image point by point. The product of this procedure, which is termed wavefront coding, is an image with large depth of field (i.e., an image in sharp focus in both the foreground and background) [5].

The extended depth of field, which is at least an order of magnitude larger than it is for regular lenses, does involve compromises. As the computer eliminates the general blurring initiated by the wavefront coding lens, the computer adds a bit of random error in the form of noise. The noise appears as a slight coarsening of shiny and smooth surfaces. Nevertheless, the enhancement of total focus more than compensates for the effect of that noise. Also, supplementary computer processing can filter that noise. New industrial and medical devices that feature the wavefront coding technology include components for microscopes and extended depth-of-field endoscopes. Wavefront-coding presents a means to reduce the number of aberration-correcting optical elements used in standard cameras and similar instruments because computers can also rectify some lens aberrations as images are de-blurred. Large space telescopes capable of spectrometry of distant planetary atmospheres [6] might be fabricated with relatively lenient construction tolerances by means of wavefront coding technology. The saddle-like lens and other wavefront-coding lenses produced up to now correspond to only a few of the myriad potential forms for computer-adapted optical elements.

Insect eyes also suggest sensing using arrays of miniature traditional lenses, known as lenslets [7, 8]. Every lenslet focuses a small, low-resolution image onto a section of an electronic detector array. A computer can determine a single large scene at approximately twice the resolution than would be achievable if one traditional lens had been employed by manipulation of all of the lenslets different viewpoints. A specific benefit of this method is that the thin lenslet array can focus light onto a detector less than a millimeter away. This extreme contraction of focal length has been used to establish a model camera as slim as a microscope slide. A number of other exceptionally thin cameras must employ tricks such as reflecting light off internal mirrors to achieve the necessary focal length inside a miniature container.

Other lenslet arrays are less pretentious, using merely apertures in place of lenses. For instance, a small polymer block packed with correctly angled holes allows photodetectors behind the block to collect light from a scene simultaneously from different viewpoints. The outcome is a tool that can reconstruct the movement of an enemy asset like an armored personnel carrier (APC) without acquiring or analyzing any images of the APC. A similar technique could be applied to cells under a microscope. Most contemporary motion-tracking mechanisms acquire images of a two-dimensional field and then analyze pixel patterns in pursuit of changes representing movement. This search is a protracted, computer-intensive process predisposed to errors. Using innovative aperture array devices, light from a selected target strikes detectors and forms a unique optical code from which a computer can quickly recreate movement with negligible computation.

Other optical elements are intended for concurrently recording spectra across the pixels of a full field of view. Such hyperspectral data may expose camouflaged missiles in a satellite image. Hyperspectral data can also reveal biological activities [9, 10], often with the aid of fluorescent labels that bind to special cellular structures. A spectra-capturing lens, or filter like a linear gradient filter, yields a pattern in which a multicolor spectrum connected with every point in a field of view is mapped onto a detector. The pattern is not an image at that juncture, only a confusion of colors and pixels. However, sorting the data in a computer transforms this apparent disharmony into an image of the field of view at any selected wavelength. Hyperspectral data have become one of the principal methods by which scientists analyze the physical and chemical properties of sample targets ranging from atoms to Martian landscapes. ICI cameras that perform at infrared wavelengths for military surveillance and biological studies are now in development, as well as ICI cameras that use ultraviolet frequencies for studying fluorescently tagged biological samples.

Spatial feature encoding can be applied in pharmaceutical and industrial process environments to reduce raw data bandwidth and increase high-level information transfer. Figure 2a (on the left) depicts a bottle (black circle) centered on a conveyor belt in a production process. During regular production, the bottle moves in the direction indicated by the arrow. The analytical task is to measure the ethanol content of the bottle without stopping the process or opening the bottle. A typical approach would be to perform single-point, stopped-flow, and wavelength-by-wavelength scanning spectrometry on the bottles as they pass a certain location, or to perform hyperspectral imaging on the bottles while they move. Single-point spectra have low data bandwidth, but they fail to capture the information on sample number, sample position, and sample velocity that can be captured by hyperspectral imaging. Hyperspectral images capture both spectral (chemical) information and information on sample container position and velocity. Unfortunately, hyperspectral images collected at frame rates of 30 frames per second or more consume large amounts of disk space and network bandwidth, much more than such a controlled situation would seem to justify. Analyzing the large volumes of data produced by hyperspectral imaging can be a problem.

Fig. 2  a An ICI camera containing only four lenslets (right) can collect both spectral and spatial information from bottles as they pass by the camera. The curved mask holds the lenslets in position. MFC can be employed on the lenslets to provide hyperspectral information. b The pulse frequency from the ICI camera is directly proportional to the bottle speed

---

An ICI camera containing only four lenslets (or even simple shaped apertures) can collect both spectral and spatial information from the bottles as they pass by the camera (see Fig. 2a). The curved mask holds the lenslets in position. MFC can be employed on the lenslets to provide hyperspectral information. Using an ICI camera, if a bottle rolls off the line or is otherwise missing, it will be detected as a missing pulse from the camera. The pulse frequency is directly proportional to the bottle speed (see Fig. 2b). The distance to the bottles or the extent of bottle filling can be determined in part by the height of the pulses. Both distance and fill can be determined simultaneously with multiple ICI cameras. If the bottles become very close to the ICI camera, pulse pile-up can occur, producing a step function on the output instead of individual pulses. Slow baseline variation from the detector can be eliminated by optical chopping or by use of the first derivative of the detector signal. Because of the ease with which differentiation can be accomplished with an operational amplifier, this approach is usually preferred. The net result is a simple camera system in which the raw data rate is relatively low, while the encoded information content is high.

Lasers typically emit light of only one wavelength at a time. The quantum cascade laser (QCL) deviates from this norm by producing a beam including all the wavelengths in a broad band of the spectrum [11]. Early efforts to create broadband laser light used unusual crystals or radical operating environments. In contrast, the QCL runs at room temperature. Innovative QCLs function in the infrared spectrum, and the multiwavelength emission makes the lasers more appropriate for molecular factor computing applications than ordinary single-wavelength lasers. A typical laser emits only a single wavelength because it is constructed with a light-emitting material that physically generates one wavelength of light when energized. Emitting a broad band of wavelengths necessitates a microchip with alternating layers of two semiconductors in an arrangement termed a quantum cascade. The cascade contains hundreds of delicate, thin semiconductor layers, each one modifying the energies of electrons passing through it. A high voltage forces an electric current to penetrate sheet after sheet in the stack in a quantum-cascade laser. The stringent physical confinement of many of those stacked sheets causes them to function as quantum wells, in which electrons can only possess certain quantities of energy.

A laser beam emanates from the stack of sheets because many of the electrons propelled into the wells by the high voltage bear more energy than the wells are able to accept. Those electrons discard their surplus energy as photons or as heat. At each end of the stacked sheets, partially reflective crystal surfaces cause a substantial fraction of the photons to rebound back and forth in what amounts to a cavity. The ricocheting photons increase the probability that other electrons in the quantum wells will also change their energy into photons as an alternative to heat. A rapid upsurge in light intensity allows sufficient photons to escape past the cavity reflector surfaces to establish a beam. Varying the width of the quantum wells from a small number of atomic layers to tens or more makes each well produce light at a different wavelength. In addition, each well has modes that generate minor quantities of light at wavelengths somewhat shorter and longer than the principal mode. The outcome is a beam with high intensity at every wavelength over a 2,000-nm-wide range, a perfect light source for MF spectrometry.

Stacking quantum wells in the QCL yields a device that is approximately 2-mm long and less than 5-m thick. Each well comprises a sheet of indium gallium arsenide between sheets of aluminum indium arsenide. While the first QCLs functioned in the mid-infrared, employing similar design rules with different materials produces broadband lasers that work in other portions of the electromagnetic spectrum. Shorter near-infrared wavelengths can be produced for molecular factor computing in aqueous biological samples.

In vibrational spectrometry of biological samples, high intensity excitation is the norm, whether broadband or narrow, and one must frequently accept few return signal photons because of the strong dipole absorption of water. A new photon detector, the superconducting transition edge detector (STED), is sufficiently sensitive to register the arrival of a single photon, and measure its energy (or frequency) with excellent precision. The STEDs ability to directly measure the location, arrival time, and frequency of individual photons may someday revolutionize biological and medical imaging [12]. Not only can this detector measure all of an individual photons important attributes, but it can do so throughout the infrared, optical, and ultraviolet portions of the spectrum.

The first tungsten transition edge sensors reported were squares about 18 m on each side, and were able to detect single photon arrivals above a threshold of 0.3 eV (ca. 4-m wavelength) with an energy resolution of 0.15 eV FWHM. This detector exhibited a rise-time of 0.5 s and a fall-time of approximately 60 s. The calibration data collected extended up to the UV cutoff of the fiber optic feed at 3.5 eV (ca. 350 nm). When the sheets are cooled down to a temperature of 0.080 K, the tungsten becomes superconducting. Tungstens transition between acting as a normal metal and a superconductor is remarkably abrupt, so very small adjustments in the metals temperature generate substantial variation in its electrical properties. The sharp resistive transition made it challenging to keep the tungsten within the narrow operating temperature range required. The control problem was resolved by employing negative feedback. The sensor was linked in a circuit that created a tiny electrical current that automatically maintained the metal at its critical transition temperature. To accomplish this control, the detector was cooled slightly lower than its transition temperature, and the electrical current raised its temperature to the critical value. When the energy from an individual photon arrived at the tungsten, it heated the electrons in the substance. This heating triggered a small increase in the electrical resistance of the thin film. In consequence, the increased resistance brought about a decrease in the electrical heating that precisely matched the amount of energy that the photon dropped into the film. This electrothermal feedback technique maintained the film at the proper temperature and also provided an accurate measurement of the photons energy and its arrival time.

The combination of only two components, the QCL and the STED, forms a powerful spectrometer capable of integrated sensing and processing. Absorption and reflection spectrometry are as simple as placing a sample in the light path between the QCL and the STED. MFs can be placed in the path along with the sample for preprocessing ISP in the spectral domain. In the spatial domain, orthogonal codes can be applied to multiple QCLs to perform active excitation imaging using a single STED.

Analytical and bioanalytical research is often data-rich but information-poor. Data threaten to cause a bottleneck in genetic research, drug discovery and development, and other areas. Research depends increasingly upon multidimensional images like hyperspectral images, three-dimensional multiwavelength confocal images to expose sites of gene expression, or timelapse videos to investigate cell behavior. ICI can help to reduce the analytical bottleneck in these studies, and should be considered as an approach.

References

1.	Weiss P (2003) New lenses create distorted images for digital enhancement. Sci News 163(13):200
2.	Cassis LA, Dai B, Urbas A, Lodder RA (2004) In vivo applications of a molecular computing-based high-throughput NIR spectrometer. Prog Biomed Opt Imaging (in press)
3.	DARPA (2002) http://www.darpa.mil/
4.	CDM Optics (2004) http://www.cdm-optics.com/site
5.	Bains S (2004) Wavefront coding finds increasing use. Laser Focus World 40(1). http://lfw.pennnet.com/
6.	Terrestrial Planet Finder Book, NASA (1999) http://planetquest.jpl.nasa.gov/TPF/tpf_book/index.html
7.	Lewis MF, Wilson RA (1994) The use of lenslet arrays in spatial light modulators. Pure Appl Opt 3:143–150
8.	Barge M, Hamam H, Defosse Y, Chevallier R, de Bougrenet de la Tocnaye JL (1996) Array illuminators based on diffractive optical elements. J Opt 27:151–170
9.	Liu Y, Windham WR, Lawrence KC, Park B (2003) Simple algorithms for the classification of visible/near-infrared and hyperspectral imaging spectra of chicken skins, feces, and fecal contaminated skins. Appl Spectrosc 57(12):1609–1612
10.	Gillies R, Freeman JE, Cancio LC, Brand D, Hopmeier M, Mansfield JR (2003) Systemic effects of shock and resuscitation monitored by visible hyperspectral imaging. Diab Technol Ther 5(5):847–855
11.	Gmachl C, Capasso F, et al. (2002) Ultra-broadband semiconductor laser. Nature 415:883–887
12.	Cabrera B, Clarke RM, Colling P, Miller AJ, Nam S, Romani RW (1998) Detection of single infrared, optical, and ultraviolet photons using superconducting transition edge sensors. Appl Phys Lett 73(6):735–737