Three-Dimensional Digital Microscopy Using the PHOIBOS Scanner

SCANNING Vol. 9, 227-235 (1987) 0 FACM, Inc. Received November 3, 1987 Three-Dimensional Digital Microscopy Using the PHOIBOS Scanner N. ASLUND, A. LILJEBORG, P.-0. FORSGREN, S. WAHLSTEN* Department of Physics, The Royal Institute of Technology, "Sarastro AB, Stockholm, Sweden Summary: New perspectives are opened in fluorescence microscopy by combining confocal scanning with digital recording and image processing. A series of consecutive optical sections obtained by confocal scanning are used to generate a data volume that constitutes a basis for digital three-dimensional microscopy. A description is given of an instrument system, PHOIBOS, which works according to these principles, and some representative results achieved with the system are presented. A great advantage as compared with traditional methods (e.g., using a microtome) is that the specimen is left undamaged. No artifacts are introduced as when slicing mechanjcally. The method can also be used to study surface topologies either in fluorescent or reflected light. Introduction Different schemes have been proposed to perform threedimensional (3-D) reconstructions of a specimen observed in a microscope. An early, noninvasive method is based on taking a series of photograpks of the specimen at a number of different focal settings ( Agard and Sedat 1983). The photographs are digitized to form a stack of digital images to be processed by a computer. Each image represents a superposition of contributions from in-focus and out-offocus regions and a fundamental problem is to remove the out-of-focus information. This is done by means of computer calculations. Three-dimensional microscopy can be done without such calculations by employing confocal scanning to generate optical sections of the specimen. Boyde et af. (1983) have developed schemes for studying the depth structure of a Address for reprints: Per-Ola Forsgren Physics IV S-100 44 Stockholm, Sweden specimen by using a series of optical sections. The sections are recorded photographically using a confocal tandemscanning light microscope (Petran et af. 1968). Davidovits and Egger (197 l), performing confocal scanning with a laser beam, obtained optical sections of a specimen on the screen of a cathode ray oscilloscope. Cox and Sheppard (1983) combined this with digital recording to obtain digital images in which each point was scanned in the axial direction and the maximum intensity value was recorded. By also recording the depth coordinate at which this maximum value occurs, they could determine the topography of reflecting surfaces and generate stereo pairs of such surfaces. Wijnaendts van Resandt etaf. (1985) also combined confocal laser scanning with digital recording and obtained optical sections in two different directions, both parallel to the focal plane of the microscope and in a direction perpendicular to this plane. Optical serial sectioning and digital image processing have been combined to generate a 3-D raster covering a volume of the specimen (Carlsson et af. 1985). This volume is processed digitally to perform different types of 3-D reconstructions. It is this approach that will be discussed herein. A more detailed account of previous works, with relevance to this method, is given in Carlsson and Aslund (1987). A later, independent realization of the present method, based on a different scanning technique (by moving the object table), has been presented by van der Voort et al. (1985). The method makes it possible to visualize the depth structure of a specimen and to carry out microphotometry in three dimensions. Flying spot scanning is performed by means of a laser beam. When it scans a semitransparent specimen it penetrates through it without damaging it. The flying spot moves along a set of parallel lines in the focal plane of the microscope. Reflected or fluorescent light from the illuminated spot is recorded. Using the depthdiscriminating property of confocal scanning (Wilson and Sheppard 1984), an optical section representing a thin slice at the focal plane in the specimen is obtained. This means that a sharp picture is extracted which in an ordinay micro-

228 N. Aslund et al. : Three-dimensional digital microscopy scope would be blurred by light from regions above and below the focal plane. By repeating this sectioning sequentially. refocusing the microscope between the recordings, a stack of digital images representing the 3-D structure of the specimen is obtained. The sections can be displayed slice by slice with a display system for digital images, or they can be used to produce images that represent projections of the object from arbitrary directions. A hostcomputer, communicating with the instrument system, is used to calculate these projections. The ability of the system to record consecutive slices of an object without losing the geometrical relationship between them is fundamental for performing threedimensional analysis. The user is presented with a nondamaging, optodigital microtome. The field of applications of the new technique is growing very rapidly. The feasibility of the method has been confirmed in a number of studies both in biology and medicine, for example, for studying cell colony growth, neuron morphology, and plant embryology. The technique can also be used in physics and chemistry (e.g., for surface studies). Before presenting a technical description, some features of immediate interest to a user will be described briefly. Some of these features are also useful in conventional twodimensional applications. General Features of PHOIBOS System An optical microscope, a high-quality research microscope, is incorporated in the PHOIBOS system (Fig. 1). No modifications of the microscope are necessary and it can be used in the usual manner to view a specimen. The scanner can work either in a confocal or a nonconfocal mode. In the latter case not only laser-generated images but also images obtained with the ordinary light sources of the microscope can be recorded. Any image that can be seen in the microscope, through its eyepieces, can thus be recorded as a digital image. The images are visible on a TV monitor. At the same time they are also accessible to a computer, for digital image processing and quantitative measurements, both geometric and photometric. The repeatability of the scanner is very high. Due to this, images of the same field of view, obtained by different illumination techniques, may be compared, combined, superimposed, etc., pixel by pixel, to take advantage of the complementary information that they may represent. An example of this is given in Aslund et af. (1983). The maximal area that is scanned is a square inscribed in the field of view as seen through the eyepieces. However, the field to be analyzed can be made larger. By repositioning the table, other subregions may be added to make a montage. For this purpose a motorized, computercontrolled object stage is used. Contrary to what happens when slices are produced in an ordinary microtome, the orientation of the sections produced by the instrument is never lost. When looking at them on a monitor, it is possible to flip instantaneously from one image to another. This gives an impression similar to changing the focal setting in an ordinary microscope. However, there is a very important difference. All information in the image seen on the monitor is in focus. If the images in the stack are added together into one single image that covers the entire depth of the stack, all the information in this picture is still sharp. The restriction of a limiting depth of focus, characteristic to optical microscopy, has been removed. The system is easy to use, even for an inexperienced operator. and sample preparation is often simplified compared with conventional microscopy. Since it works without destroying the specimen applications out of reach of traditional methods are opened up. In addition, there is the new possibility offered by digital processing and display. This will be discussed in some detail in a later section. FIG I Overview of the PHOIBOS system. Technical Description of the Scanner In a confocal scanning laser microscope a focused laser beam illuminates the specimen point by point (Wilson and Sheppard 1984). The reflected or fluorescent light from the specimen is focused onto a pinhole aperture in front of a detector. Figure 2 shows the function of the PHOIBOS scanner ( Aslund et af. 1983; Carlsson et af. 1985, Carlsson and Aslund 1987). The laser beam enters the microscope via a beam splitter and two scanning mirrors controlled by a computer. The phototube of the microscope is used as the entry for the laser beam. The laser beam is focused onto the specimen by the microscope objective and is deflected by the two scanning mirrors which displace the

N. Aslund el al. : Three-dimensional digital microscopy 229 Work FIG. 2 Schematic diagram of the system. Tape focused beam in two perpendicular directions. To move the beam along a scanning line, it is deflected by a mirror attached to a galvanometric scanner (A, Fig. 2). A mirror moved by a stepping motor is used to deflect the beam from line to line (B, Fig. 2). The time required to scan a 256 by 256 picture is 5.1 seconds. The function of the beam splitter is to reflect the beam of light from the laser onto the scanning mirrors and transmit the returning light from the specimen onto the detector. In fluorescence microscopy the returning light is of different wavelength than the incoming light. Therefore a dichroic mirror can be used as beam splitter. A dichroic mirror has a specific cut-on wavelength where transmission becomes more dominant than reflection. In the scanner it is mounted on a wheel that can hold several such beam splitters, each with a different cut-on wavelength. There is also a barrier filter placed in front of the PM-tube. The combined effect of the beam splitter and the barrier filter is to block the exciting laser light from reaching the detector. (This bamer filter is removed when the instrument is used to record reflected rather than fluorescent light.) In this way, the instrument can be adapted for different fluorescent stains by changing laser wavelength and beam splitter. In fact several fluorescent stains can be used in the same specimen since the specimen does not have to be moved while changing laser wavelength and beam splitter. In fact several fluorescent stains can be used in the same specimen since the raypath is not affected by changing laser wavelength and beam splitter. Thus, the pixel-to-pixel correspondence is retained. Several pinhole apertures are mounted on a wheel similar to that used for the beam splitters to allow for rapid change of aperture size. In this way it is possible to optimize the relationship between light intensity and depth resolution. The fluorescent light is often weak, and to improve the signal-to-noise ratio the detected signal is integrated in an analog circuit prior to analog-to-digital conversion. The integration time may be selected up to a limit set by the total time spent for scanning one pixel in the specimen. As has been shown theoretically (Sheppard and Wilson 1978), the depth resolution that can be obtained with confocal scanning is not as good as the lateral resolution. Thus it is not necessary to make the spacing between different sections as small as the spacing between the pixels in the plane. As an example, we have obtained a lateral resolution of 0.25 pm and a depth resolution of - 1.0 pm with a Zeiss Planapo 100/1.3 at an excitation wavelength of 458 nm and a fluorescent wavelength of 550 nm. A microcomputer coordinates the different functions of the scanner. It handles the collection of pixel data, communication with a host computer, the image memory, and a user keyboard with joystick. Another microcomputer acts as an intelligent stepping motor controller. There are four stepping motors in the scanner, controlling the slow scanning mirror, the focus, and the object stage. The pixel data is stored in a digital image display memory while the specimen is being scanned. A picture of the specimen gradually appears on a TV screen during scanning. Communication with a host computer is accomplished with

230 N. Aslund et al. : Three-dimensional digital microscopy either a IEEE-488 (GPIB) parallel interface or a IEEE-802 Ethernet serial interface. Both interfaces are capable of transmitting and receiving data and commands from the host computer at high rates. The host computer interface allows the user to transmit the images to a larger computer for further processing. All functions of the scanner can be initiated by commands from the host interface. The operator can use the scanner in several ways. It can be used as an ordinary microscope during the search for objects and areas to digitize. The most basic way to take advantage of the digital and confocal aspects of the scanner is to use the keyboard and joystick box. Sixteen function keys give access to low-level operations like scanning an image and displaying it. The focus setting can be changed step by step with function keys or more coarsely with the joystick. The object stage can also be controlled by the joystick. One of the function keys allows the operator to obtain a vertical section through the specimen by refocusing the microscope between each linescan instead of moving the slow scanning mirror. When using the host computer interface, all actions can be initiated by the host computer. The operator thus controls the instrument through interaction with the host computer instead of using the keyboard and joystick box. In this case a menu-based program is used to communicate with the scanner. The collected data may be further processed by the host computer and the scanner is not involved at all. Data Processing The data processing part of the PHOIBOS system consists, at present, of software to be run on a standard computer, with possibility to add an array processor. Basically the software comprises routines for storage, display and filtering. The data storage routines provide means for storing image data together with relevant information about the recordings. An example of this is seen in Figure 3, which shows the structures of data concerning an original volume from the microscope. Creation and access to these data structures are performed through high-level subroutine calls. An observation of great importance is, that in many cases the data matrix is very sparse. The interesting data is concentrated in well defined parts of the volume with empty space surrounding it. This gives a possibility to reduce the requirements on data storage and the amount of computation. The reduction in storage is achieved by neglecting the uninteresting elements of the volume. The remaining elements are described by vectors (Forsgren 1987). Each vector gives the position of the element and its value, which is normally the intensity at the point. The number of vectors may be large, but this should be compared with the size Section series J Section 1 series number xsize number of sections xy/z resolution section member 1 ysize number of bits in voxel table x-position table y-position table z-position start coordinate x start coordinate y increment in x increment in y laser wave-length barrier filter wave-length aperture size integration time number of accumulations photomultiplier voltage magnification photometric offset FIG. 3 Information structures of the original volume. of the original volume: a 5lZ3 cube occupies 128 Megabytes of storage if 8 bits are sufficient for intensity information. One vector requires 4 or 8 times more data storage space than a single element of the raster volume (depending on whether 8 bits are sufficient or not for storing the coordinate information). But if the matrix is sparse, substantial reductions are possible. By filtering the sparse information of the volume, fulther reductions can be made. Vector representation has been used for all pictures shown in this article. The storage reductions are in the range 5-65. The vector storage form has some merits and some drawbacks. As the coordinates are explicit, geometrical transformations are very easily accomplished. Random search in the volume is on the other hand more difficult as it requires a large amount of comparisons. Many different methods have been developed for visualizing 3-D stacks of images (Carisson et al. 1985, Brakenhoffetaf. 1985, Lenz 1986). A simple approach is to look at the individual slices in consecutive order. This can give very much information and is excellent for tracking individual features. It is important though, that the change from one slice to another is instantaneous. A memory buffer that allows flipping between six 512* pictures or twenty four 2562 pictures has been used in the studies referred to in this article. A very good means for understanding 3-D structures is to calculate projections ofa volume. The projections used are orthographic projections. A simple example of such a

N. hund et al. : Three-dimensional digital microscopy 23 1 projection is obtained by adding the images of the stack. This gives a projection image corresponding to projection rays that are perpendicular to the sections. Rather than just summing the intensity values along the ray, other schemes may be used to define the projection image. Further, by geometrical transformations performed by the computer the volume may be rotated before the projection image is calculated. This means that the direction of the transversing rays, and thus the viewing direction, can be arbitrarily chosen. Such geometrical transformations are particularly simple to perform when vectors are used to describe the object. An example of a projection image is seen in Figure 4. Two projections from somewhat different directions constitute a stereo pair (see Figs. 6 and 8). By generating a stack of consecutive projections one can create the impression of a rotating object by displaying the images in fast succession. To emphasize surfaces of the object, a high-pass frequency filter such as a gradient can be used. Its effect is to enhance regions of rapid change in intensity. An illustrative Of this is the enhancement Of the boundaq and the cell of a neuron Seen in Figure 5. By depth coding, coarse depth information is conveyed in a single picture. The intensity in the picture is inversely FIG. 5 A lateral interneuron from the spinal cord of a lamprey. The neuron was stained with the fluorescent dye Lucifer Yellow. Specimen area 95 X95 pm. The data has been filtered with a gradient operator of size 3 x 3 x 3 and the resulting response is summed along the projection rays. Objective: Zeiss Planapo 100/1.3. Excitation wavelength: 458 nm. Specimen provided by P. Wallen, Karolinska Institute, Stockholm. proportional to the position in depth. Bright areas, as illustrated in Figure 7d, are interpreted as being closer to the viewer. More detailed depth and surface information can be displayed very clearly by using projection images obtained by surface shading techniques. A model of a diffuse surface is used which means that the intensity in the projection image is calculated as n*v z = zmax - (ill (v( + zo FIG. 4 An edge cell from the spinal cord of a lamprey. The cell was stained with the fluorescent dye Lucifer Yellow. Specimen area 95 x95 pm. The picture has been obtained by summing the intensity values along the projection rays. Objective: Zeiss Planapo 100/1.3. Excitation wavelength: 458 nm. Specimen provided by P. Wallen, Karolinska Institute, Stockholm. where Z is the 3-D surface normal and V is the viewing direction. A simpler way than to calculate the surface normal in three dimensions is to use an approximation obtained from a depth-coded projection image as described by Gordon and Reynolds (1985). The result is not as good as true three-dimensional surface shading, but the surface structure is clearly seen, as exemplified in ~i~~~ 6. Using several depth cues hopefully reminds the user that they all are computer generated, based on rather simple models, and that careful is necessary. In Figure 7, a comparison between different depth cue

232 N. hund ef al. : Three-dimensional digital microscopy FIG. 6 A motoneuron from the spinal cord of a lamprey. The neuron was stained with the fluorescent dye Lucifer Yellow. Specimen area 95 X95 pm. Shaded picture ofa surface determined by thresholding the original data. Stereo angle 6PObjective: Zeiss Planapo 100/1.3. Excitation wavelength: 458 nm. Specimen provided by P. Wallen, Karolinska Institute, Stockholm. FIG. 7 Panels c. d and e on next page

N..hind et a/. : Three-dimensional digital microscopy 233 FIG. 7 The same specimen as in Figure 5. Specimen area 240 ~ 240 pm. Figure (a) shows a projection of the original data, (b) with gradient filtering, (c) with surface extraction, (d) depth coding, and (e) surface shading. Objective: Zeiss Planapo 40/1.0. Excitation wavelength: 458 nm.

234 N. Aslund er af. : Three-dimensional digital microscopy FIG. 8 Chinese hamster V-79 cells attached to a microcarrier (a collagen-coated dextran sphere with a radius of 100 pm). The specimen is stained with propidium iodine. Specimen area 220 X220 prn. Depth resolution is about 2 pm. The original data has been gradient filtered to enhance surfaces of the objects. Crosses indicate cell nuclei found by the computer. Stereo angle 6P Objective: Zeiss Planapo 4011.O. Excitation wavelength: 458 nm. Specimen provided by E. Szolgay-Daniel, The Gustaf Werner Institute, Uppsala. schemes is made, and the very different aspects of the object that they reveal can be seen. Applications As has been emphasized above, a benefit of confocal scanning is that the contributions from out-of-focus regions on an optical section are strongly attenuated. However, there is another phenomenon that may influence the imaging, and that is not removed by confocal scanning, namely the absorption that the imaging light may undergo on its way back and forth through the specimen. This absorption will depend on the characteristics of the specific specimen that is studied. An early experiment, reported in Carlson et a1. (1985), indicates that in practice these effects may be of little consequence when the method is used for visualization. The experiment was camed out on a neuron cell from a lamprey, that had been prepared by microinjection of a fluorescent stain, Lucifer Yellow. The images were not seriously distorted by any shadowing or other effects due to absorption. This is also the case for the images we present. This does not mean that such effects can be neglected when using the method for photometry rather than for visualization. Also, when the method is used for visualization. one should keep in mind that such shadowing may appear. In the figure legends examples of applications of the new technique are described. A project involving not only visualization but also analysis in 3-D is shown in Figure 8. Object segmentation and counting of the number of cell nuclei situated on a microcarrier were performed (Szolgay- Daniel et al. 1985). References Agard D, Sedat J: Three-dimensional architecture of a polytene nucleus. Nature 302, 676-681 (1983) Aslund N, Carlsson K, Liljeborg A, Majloff L: PHOIBOS, a microscope scanner designed for microfluoremetric applications, using laser induced fluorescence, Proc. of 3rd Scandinavian Conference on Image Analysis, Copenhagen, 338-343 (1983) Brakenhoff G, van der Voort H, van Spronsen E, Linnemans W, Nanninga N: Three-dimensional chromatine distribution in neuroblastoma nuclei shown by confocal scanning laser microscopy. Nature 317, 748-749 (1985) Boyde A, Petran M, Hadravsky M: Tandem scanning reflected light microscopy of internal features in whole bone and tooth samples. J Microsc 132. 1-7 (1983) Boyde A: Stereoscopic images in confocal (tandem scanning) microscopy, Science 230, 1270-1272 (1985) Boyde A: Colour-coded stereo images form the tandem scanning reflected light microscope (TRLSM), Jof Microsc 146, 137-142 (1987) Carlsson K, Danielsson P, Lenz R, Liljeborg A, Majloff L, Aslund N: Three-dimensional microscopy using a confocal laser scanning microscope. Opt Letr 10, 53-55 (1985) Carlsson K, Aslund N: Confocal imaging for 3-D digital microscopy. Appl Opt 26, 3232-3238 (1987) Cox I J, Sheppart C J R: Digital image processing of confocal images, Image and Vision Computing 1, 52-56 (1983)

N. hund et al. : Three-dimensional digital microscopy 235 Davidovits P, Egger M: Scanning laser microscope for biological investigations. Appl Opt 10, 1615-1619 (1971) Forsgren P-0: Using vector representation to display 3D data from a confocal laser scanning microscope. Proc 5th Scand Conf on Image Analysis, Stockholm, Sweden (1987) 367-374 Gordon D, Reynolds A: Image shading of 3-dimensional objects. Comput Vision, Graph Image Proc 29, 361-376 (1985) Howard V, Reid S, Baddeley A, Boyde A: Unbiased estimation of particle density in the tandem scanning reflected light microscope, J of Microsc 138, 203-212 (1985) Lenz R: Reconstruction, processing and display of 3-D images. PhD dissertation no. 151, Linkoping University, Sweden (1986) Petran M, Hadravsky M, Egger M, Galambos R: Tandem-scanning reflected-light microscope. J Opt Soc Am 58,661-664 (1968) Sheppard C, Wilson T: Depth of field in the scanning microscope. OptLeft3, 115 (1978) Szolgay-Daniel E, Larsson B, Brunnstrom K, Forsgren PO: A new method for colony formation of Chinese hamster cells on microcamers. Proc 2nd Int Syrnp on Neutron Capture Therapy. Tokyo University, Tokyo (1985) 353-358 van der Voort H T M, Brakenhoff G J, Valkenburg J A C, Nanninga N: Design and use of a computer controlled confocal microscope for biological applications, Scanning 7, 66-78 (1985) Wijnaendts van Resandt R, Marsman H, Kaplan R, Davoust J, Stelzer E, Stricker R: Optical fluorescence microscopy in three dimensions: Microtomoscopy. J Microsc 138, 29-34 (1985) Wilson T, Sheppard C: Theory and Practice of Scanning Optical Microscopy. Academic, London (1984) SCANNING Vol. 9,6 (1987)