Spectral imaging microscopy (imaging spectroscopy) systems

Transcription

1 Spectral imaging microscopy (imaging spectroscopy) systems Bartek Rajwa Assistant Professor Purdue University

2 Color (RGB) imaging! Bayer mask uses a color filter array that passes red, green, or blue light to selected pixel sensors, forming interlaced grids sensitive to red, green, and blue! The missing color samples are interpolated using a demosaicing algorithm. Foveon uses an array of layered pixel sensors, separating light via the inherent wavelengthdependent absorption property of silicon, such that every location senses all three color channels. Spectral imaging 2

3 LANDSAT system Spectral imaging 3

4 224 bands AVIRIS Airborne Visible/Infrared Imaging Spectrometer nm spectral range AVIRIS uses diffraction gratings with two sets of CCD arrays, one with silicon chips to sense in the visible range and the other with Indium- Antimony (InSb) chips for wavelengths in the Near-IR to Short- Wave-IR range An oscillating mirror scans the scene. Mounting AVIRIS on a NASA aircraft (ER-2), produces a spatial resolution of about 20 meters Spectral imaging 4

5 1960 the successful launching of the first weather satellite, TIROS the National Academy of Science through its National Research Council formed a committee on "Aerial Survey Methods in Agriculture 1964 Purdue, UC/Berkeley, and Michigan consortium starts developing methods for remote sensing 1966 First multispectral pattern recognition methods capable of identifying individual crop species demonstrated by the Purdue researchers (Professor David Ladgrebe). Let s talk about remote sensing Spectral imaging 5

6 Remote sensing Spectral imaging 6

7 Hyperspectral data cube Moffett Field, California, at the southern end of the San Francisco Bay. AVIRIS acquired the data on August 20, 1992 when it was flown on a NASA ER-2 plane at an altitude of 20,000 meters (65,000 feet). Spectral imaging 7

8 210 bands unmixed into 7 endmembers Spectral imaging 8

9 Spectral imaging web resources The illustrations are copied from The article Spectral Imaging and Linear Unmixing authored by! George McNamara - Miller School of Medicine, 1450 Northwest 10th Avenue (R-134), University of Miami, Miami, Florida, ! Jeffrey M. Larson and Stanley A. Schwartz - Nikon Instruments, Inc., 1300 Walt Whitman Road, Melville, New York, ! Michael W. Davidson - National High Magnetic Field Laboratory, 1800 East Paul Dirac Dr., The Florida State University, Tallahassee, Florida, Spectral imaging 9

10 Spectral imaging in microscopy Linear unmixing can be employed with fluorescence spectral imaging due to additive properties of fluorescence emission spectra Illustrations courtesy of Zeiss Inc, and M. Davidson, FSU Spectral imaging 10

11 Lambda stack Spectral imaging 11

12 Spectral imaging Spectral imaging 12

13 Spectral microscopy hardware Sequential spectral imaging (Zeiss LSM 700, Leica, LCTF systems, AOTF systems) Simultaneous spectral imaging (Zeiss Meta, Nikon) Illustrations courtesy of Zeiss Inc, and M. Davidson, FSU Spectral imaging 13

14 Nikon spectral imaging system Spectral imaging 14

15 Other multispectral arrangements in confocal microscopy Spectral imaging 15

16 210 bands unmixed into 7 endmembers 16

17 Slide courtesy of Timo Zimmermann, EMBL, Heidelberg Spectral unmixing Spectral imaging 17

18 Multispectral small animal imaging Combined macro- and microimaging of bidirectional axillary lymphatic flows from the mammary pad (courtesy R. Levenson). Maestro in-vivo imaging system. Slide courtesy of Cri (now Perkin Elmer) Nude mice with two different species of autofluorescence and three subcutaneous fluorochrome signals 18

19 Mixing and unmixing 100 balls 50 cubes Spectral imaging 19

20 Can we unmix the mixed signals? Two types of photons: cubes, and balls Signals 80% 70% We want balls to go mostly to the ball detector, and cubes in the cube detector 30% 20% Detectors Ball detector Cube detector 4/28/16 Multispectral flow cytometry 20

21 Mixing, sorting and sieving 50 cubes 100 balls We want balls, not cubes a =! # " $ &;M = %! # " $ & % 4/28/16 Multispectral flow cytometry 21

22 Unmixing vs. compensation Filter A Filter B Spillover r = Ma + n In spectral unmixing the M matrix is columnwise normalized to 1 r! Sa + n, S = { s i,j = m i,j / m j,j, m j,j = max(m j )} In cytometry-style compensation the spillover matrix is normalized to its diagonal Spectral imaging 22

23 Matrix notation!!!"#!"$" = # $ %!"%!"&& Ball detector Cube detector 4/28/16 Multispectral flow cytometry 23

24 What is the definition of true signal? Unmixing Compensation a 1 s 1 " $ r 1 = pa 1 + (1! q)a 2 # %$ r 2 = (1! p)a 1 + qa 2 & r ) ( 1 + ( r ' 2 + = & p 1! q ( 1! p q * '( ) & + ( * + ( ' a 1 a 2 ) + + * #! = " " #!! #" =! + #!! % $ % &! = " " $!! &" = $!! +! '" ( '!!! # ('!!( ) * = )! *) * + "", + $!, +!",!! "!! " " "! "! " The a 1 and a 2 (abundances) defined for unmixing are different than s 1 and s 2 (compensated signals) defined for compensation! 4/28/16 Multispectral flow cytometry 24

25 Unmixing requires matrix inversion!!!! = "!!"#!"$"!%!" # $! = # $ %!"&!"'& %! %& Matrix multiplication "!"#!"$# " %"&!!"'# "%!# $ % $ % =! $ % &!"(!")' &!"& %"' ' &! %' 4/28/16 Multispectral flow cytometry 25

26 Matrix inversion - example Spectral imaging 26

27 Matrix inversion Spectral imaging 27

28 Unmixing (or how to fix a bad sieve) Given the result and the sieve characteristics described by the M matrix, can we figure out the a vector? Yes, we can r = [95,55]; M =! # " Columnwise normalization $ &; S = %! # " Diagonal normalization $ & % a U = M -1 r! # a U = % $ a C = S -1 r! # a C = % $ 1.4 "0.6 " "0.28 " & # ( ) % ' $ & # ( ) % ' $ & # ( = % ' $ & # ( = % ' $ & ( ' & ( ' 4/28/16 Multispectral flow cytometry 28

29 Sieved (yet, still mixed) result r=[95,55] 55 objects= 35 cubes + 20 balls 95 objects=80 balls + 15 cubes Spectral imaging 29

30 Linear mixing of fluorescence signal r = Ma + n OMG! Math!! r vector of observations (pixel vector) M target spectral signature matrix (pure endmembers) a vector of abundances (or abundance fractions) n noise (error) The LMM is subject to two constraints on the entries of a.! To be physically meaningful, the nonnegativity condition requires all abundances to be nonnegative such that a i!0! Second, as a way of accounting for the entire composition of a mixed pixel, the full additivity condition requires "a i =1. Spectral imaging 30

31 From n channels to two 2 colors Unmixed image of spectrally similar fluorochromes in tobacco BY!2 cells stained with MitoTracker Red and Congo Red, imaged using the Zeiss META spectral imager. Journal of Microscopy Volume 214, Issue 2, pages , 26 APR 2004 DOI: /j x

32 Mixing of signals (multspectral case) a 1 a 2 a 3 a Source of signal M r Detectors r 1 r 2 r 3 r 4 r 5 4/28/16 Multispectral flow cytometry 32

33 Overdetermined (multispectral) case M =! $ # & # & # & # &, a =! # " & # & # & "# %& $ % r=[ ] Can we go back from r to a? Can we unmix a from r? Spectral imaging 33

34 FC spectral unmixing deals with the problem by finding the leastsquare estimation of a. Therefore: Overdetermined case? â LS = M T M ( )!1 M T r For square M we got simply â LS = M!1 r Spectral imaging 34

35 Kernel PCA (cosine kernel) 4+5'()*+,*-./' &!"#!"$!# $ # "$ "#! Pure endmemebers form a simplex!! Theory of convex sets! Simplest endmember estimation requires only PCA (or kpca)!"#!"$!# $ # "$ "# "%&'()*+,*-./' & Spectral imaging 35

36 PCA Dutch Boy paint cards Colors difficult to distinguish by visual inspection Spectral imaging 36

37 Reflection example PCA plot Spectral imaging 37

38 Non-negative matrix factorization (NMF) Known in chemometrics as "self modeling curve resolution". Deceptively simple problem: given a matrix X find nonnegative matrices W and H that minimize the function!"!#" $ = #!!"!! Spectral imaging 38

39 Woolfe, F. et al.., Autofluorescence Removal by Non-Negative Matrix Factorization. IEEE Transactions on Image Processing 20, doi: /tip Autofluorescence removal (a) Human breast biopsy stained with cy5 bound to estrogen receptor. The fluorescence due to cy5 is mixed with tissue AF. (b) The AF is removed by our NMF algorithm. Spectral imaging 39

40 Rabinovich, A., Agarwal, S., Laris, C.A., Price, J.H. & Belongie, S. Unsupervised color decomposition of histologically stained tissue samples. Advances in Neural Information Processing Systems (2003). NMF in histopathology Color unmixing using Non-negative Matrix Factorization (NMF) and comparison to ground truth measurements. (a) Tissue sample single stained with diaminobenzidine (DAB); (b) Measured optical density of single stained, DAB, tissue sample; (c) Same tissue sample as in (a) but a second stain, hematoxylin is added to the sample; (d) Estimated optical density of DAB using NMF. (e) Estimated optical density of hematoxylin using NMF. The error measure is computed by comparing images (b) and (d). Spectral imaging 40

41 Spectral unmixing on excitation side Tunable light sources Supercontinuum white laser AOTF and white light source Two images of Convallaria majalis taken at excitation wavelengths of (a) 620 nm and (b) 530 nm. Each image is 355 "m x 355 "m. (c) A plot of fluorescence intensity versus excitation wavelength for several different structures within the sample (see (a) for locations). Illustrations courtesy of C.F. Kaminski, University of Cambridge, UK Spectral imaging 41

42 Emission/Excitation spectral microscopy A mixture of 6 differently colored beads was recorded. Left: false-color projection of the beads recorded by an excitation-emission lambda-square scan. Right: pseudo three dimensional view of the excitation-emission map (fluorescence landscape). One can easily locate the peaks of the six different fluorochromes Spectral imaging 42

43 Thank you! Spectral imaging 43