Intelligent agents (TME285) Lecture 4,

Transcription

1 Intelligent agents (TME285) Lecture 4, Image processing for IPAs + Advanced C# programming

2 Assignment, Stage 1 Note, again, that to complete Stage 1, you must have a discussion with us, based on your draft, submit the final version of your planning document. Many have now had the discussion, but I have only received a few final versions so far (deadline: Today!) Some have not even had the discussion the last chance (to get the 5p) is in the breaks of today s lecture (and you must then finalize and send the document later today).

3 Assignment, Stage 1 Formatting Use the headers in the assignment, i.e. make sure to cover the topics (task, dialogues, similar agents etc.) described there. For the dialogues: List the required dialogues, with the name of the dialogue (e.g. GreetingDialogue) as well as a brief general description (not a usage example) of what it will do. For the visualization, show (schematically) how the entire GUI will look which application windows will be shown, where will they go on the screen etc. etc.

4 Assignment, Stage 2 Simplification: For Stage 2, it is sufficient that you just send the document (i.e. prior discussion is certainly allowed but not required) provided that you follow the instructions, i.e. writing a 2-6 pages long document describing the dialogues in detail (dialogue item by dialogue item). To get some examples, see Hazel s dialogues. giving specific usage examples for each dialogue, describing (briefly) the derived DialogueItem classes that you intend to implement.

5 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

6 Image processing in IPAs Any IPA with a camera will require some image processing. Typical image processing tasks for an IPA are Face detection and recognition, Gesture recognition, Object detection (in general), Facial emotion detection. These are all fairly advanced concepts that rely on the basic image processing operations described next.

7 Digital images and color spaces Picture elements (pixels) can be represented in different color spaces: RGB (Red-Green-Blue) Three components, each typically one byte (0 255) Fourth component (alpha channel) determines the level of transparency (where applicable). Grayscale Gray value Γ in [0,255] (Γ = R = G = B) Typical conversion:

8 Digital images and color spaces Picture elements (pixels) can be represented in different color spaces: YCbCr Used, for example, in skin pixel detection (see p. 52) Luma and chrominance (blue and red) Common conversion betwwen RGB and YCbCr:

9 YCbCr components Misprint! Should say Cr

10 Image histograms Compact representation of image contents. For each color channel, count the number of pixels at each value 0, 1, 255:

11 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

12 Basic image processing Common operations: Changing contrast and brightness Grayscale conversion Binarization Image convolution: Sharpening, blurring and edge detection Histogram stretching Integral images and connected components (Morphological image processing)

13 Contrast and brightness Transformation to change both contrast and brightness: A value > 255 must be set to 255, and values below 0 must be set to 0. Problem: Setting suitable values of α and β. Often better to use histogram stretching (see below).

14 Grayscale conversion Reduces the amount of information (from 3 color channels to 1). General expression: Common values: f r = 0.299, f g = 0.587, f b =

15 Binarization Simplest version: In a grayscale image, set pixels with gray level below a given threshold to 0 (black) and all other pixels to 255 (white). In practical applications, one must often handle brightness variations over the image; see below (Adaptive thresholding).

16 Convolution In convolution, one passes an N x N matrix (the convolution matrix) over every pixel in an image, changing the value of the center pixels based on an elementwise product of the convolution matrix and the image pixels under it: Here, ν = (N-1)/2, and N is assumed to be odd.

17 Blurring and sharpening Example 1: With the matrix the center pixel will be an average of the 9 pixels covered by the convolution matrix, resulting in a distinct blurring:

18 Blurring and sharpening Example 2: With the matrix the center pixel will emphasized relative to its neighbors, resulting in sharpening:

19 Blurring and sharpening

20 Histogram stretching Histogram stretching is applied in order to enhance the contrast of an image. Method: First generate the grayscale histogram, Then normalize the histogram and generate the cumulative histogram:

21 Histogram stretching Method: then find the bin index j Low as the smallest j such that H c (j) > p, and the bin index j High as the largest j such that H c (j) < 1-p. Then, set any pixel with gray level below j Low to black and any pixel with gray level above j High to white. Finally, for all other pixels, set the new gray level as:

22 Histogram stretching

23 Edge detection Can be implemented as a dual convolution (usually also preceded by blurring) However, this somewhat time-consuming process can be approximated as follows (see p. 54 in the compendium):

24 Integral image Integral images are useful when one needs to compute the pixel sums over many regions in an image. The integral image (element) I(i, j) is defined as the sum of all pixels above and to the left of (i, j): The integral image can be computed as

25 Integral image Once the integral image has been obtained, one can find the sum of pixels in any rectangular area with one addition and two subtractions, as If any index is < 0 (can happen if the requsted sum involves the left edge or the top), set the corresponding term to 0.

26 Integral image: Example

27 Connected components In many tasks, such as face detection, one may need to find connected components, i.e. regions of pixels (in a binarized image) that are connected to each other. Detailed method not given here (good references available online):

28 Connected components: Example

29 Morphological image processing In morphological image processing, one passes a shape (the structuring element) over the image. Then, the value of a given pixel (usually the center pixel) is changed if some conditions are met. This topic is a bit outside the scope of the scope, but see pp in the compendium. Two common morphological operators are erosion and dilation; see the example on the next slide.

30 Morphological image processing

31 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

32 The ImageProcessing library Part of the IPA libraries. Most important classes ImageProcessor Camera

33 The ImageProcessor class The ImageProcessor class contains fast implementations of several basic image processing tasks:

34 The ImageProcessor class Rather then accessing the pixels of the image (or, rather, its bitmap) directly via GetPixel() and SetPixel(), the methods in the ImageProcessor class use very fast pointer operations. However, such operations involve direct memory access (rather than the managed memory access in.net) and are therefore referred to as unsafe. One must take great care when applying such operations. This topic will be described further in the next lecture!

35 The ImageProcessor class Furthermore, some methods in the ImageProcessor class makes use of parallel processing. This, too, is a type of operation that requires careful implementation, and which will be described later in this lecture. For now, we will focus on the use of the ImageProcessor class!

36 The ImageProcessor class Generates the image processor and locks the bitmap in memory. Sequence of image processing operations Releases the locked memory, making the processed bitmap available. Actively disposes the image processor. Should be called if, for example, you generate many image processors in a short time such that the garbage collection in.net may have a hard time keeping up.

37 The ImageProcessor class Constructor and Lock() method:

38 The ImageProcessor class Sample method (details given next lecture)

39 The ImageProcessor class Release() method: NOTE! Call Release() before trying to access the ProcessedBitmap! See also the example above!

40 The Camera class This class makes use of a CaptureDevice class that, in turn, uses the DirectShow library (placed as a DLL in the ImageProcessorLibrary (/bin/debug) Usage example:

41 The Camera class In order to access the camera bitmap, one must run a separate thread that obtains (reads) the camera image at regular intervals. Threading will be described below.

42 The CameraSetupControl class This is a UserControl (a graphical component generated by a user, in this case me!) for showing and modifying camera settings:

43 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

44 Advanced image processing Many different topics could be considered here. Topics of particular relevance to IPAs: Adaptive thresholding, Motion detection (and background subtraction), Face detection and face recognition.

45 Adaptive thresholding In adaptive thresholding, one uses a varying binarization threshold, in order to get the best possible binarization. Several methods exists. Here, Sauvola s method will be described. Fixed binarization threshold Sauvola s method

46 Adaptive thresholding In Sauvola s method, the binarization threshold is computed (locally) as 1,2 where k is a parameter, s is the standard deviation over a j x j region around the current pixel, m is the mean, and R is the maximum standard deviation over all j x j areas. 1. Note: Small misprint in Eq.(4.27): Should read σ instead of s. In both cases, I mean the standard deviation. 2. Note: The implementation (accidentally included in the IPA libraries) in the ImageProcessor is a debug version in which r is a fixed input parameter. Do not use! (But you can implement your own method, starting from that method, if you wish).

47 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

48 Motion detection Motion and gesture detection is relevant in many IPAs. An important special case is background subtraction. A simple approach is to take an image of the area in front of the agent before the user sits down, and then, with the user in front, take any pixel as foreground that fulfil: Current gray level at pixel (i, j) Background gray level This approach will not work well though, due to noise, variations in illumination etc.

49 Motion detection A better approach is to use exponential Gaussian averaging: Initialize the average (μ) as Current gray level at pixel (i, j) and initialize the variance at pixel (i, j) as the variance over the pixels surrounding that pixel.

50 Motion detection Then update the average and variance as where

51 Motion detection One can then take as foreground any pixel that fulfils for some suitable value of α.

52 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

53 Face detection and recognition Many, though not all, methods for face detection are based on skin pixel detection.

54 Face detection and recognition Once the skin pixels have been detected, one can process the information to find the bounding box of the face. The Vision application, described on pp shows one example of how one can do this. You may start from this example and then make modifications as necessary. Drawback: Skin pixel detection is quite dependent on the illumination level. A more advanced approach is the Viola-Jones face detection algorithm (see p. 66 and the references).

55 Face detection and recognition General face recognition is quite difficult, and typically requires a large training set. Examples of methods (pp and references) the eigenface method (in which faces are defined as linear combinations of facial templates) or...local binary patterns, or artificial neural networks.

56 Face detection and recognition General advice: Before starting with your implementation, carefully define the scope of your algorithm, In this case (IPAs) it is often sufficient that the agent can detect a single face, seen from the front. the agent can distinguish one (or perhaps a few) users from other people (non-users), using specific features found in those few faces. Thus, rather than uncritically implementing a generalpurpose method for either face detection or face recognition, consider the scope first (and discuss with us).

57 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

58 The ImageProcessing application This application allows the user to test sequences of basic image processing tasks:

59 The VideoProcessing application This application demonstrates how the camera class is used (in particular, see the CameraViewControl). It also shows how to use the CameraSetupControl. Finally, it demonstrates basic background subtraction, using exponential Gaussian averaging.

60 Advanced C# programming Appendices A4 and A5 + CommunicationLibrary (async callback)

61 Threading Many computer programs are multi-threaded, meaning that different operations runs in parallel (at least from the user s point-of-view) in separate threads. A common application of multi-threading is to have one thread responsible for updating the GUI and interacting with the user and, one thread for doing computationally intensive work in the background.

62 Threading By default, in a standard Windows forms application in C#, the operation take place in the GUI thread. Thus, one has to make explicit use (see below) of threading in order to achieve the situation described in the previous slide. Appendix A.4 in the compendium.

63 Single-thread example Consider now the first example in Appendix A.4. This example (and the next) is implemented in the ThreadingExample application in the DemonstrationSolution available in the IPASrc folder. In the single-thread example (Listing A.10 in the compendium) the user (unwisely) runs a heavy computation in the GUI thread. As a result, the GUI becomes frozen (does not respond) during the computation.

64 Single-thread example

65 Multi-threading example Consider now the second example in Appendix A.4. Here (see listing A.11 in the compendium) the user starts the heavy computation in a separate thread. As a result, the GUI remains responsive during the calculation.

66 Multi-threading example Instantiates the computationthread. Starts the computationthread, in which it executes the ComputationLoop()

67 Multi-threading example There is a price to be paid though: One has to be careful with cross-thread communication. For example, any operation involving the GUI will require access to the GUI thread (see the next slide) Whenever one wishes to access the GUI thread from another thread, one must use the BeginInvoke() 1 method (which is available in any graphical component 2). 1. There are some different variants though. Sometimes, one instead uses Invoke(). Here, however, we shall only use BeginInvoke(). 2. Graphical components (forms, buttons, text boxes etc.) are derived from the Control base class, which is where the BeginInvoke() method resides.

68 Multi-threading example Thread-safe visualization of progress: Simplying somwhat, one can say that this method asks for permission to access the GUI thread, which is required to carry out any action in the GUI One can then carry out the actual GUI operation (in this case: Showing progress information on the screen) using the same method as in the single-thread example.

69 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

70 Asychronous callbacks BeginInvoke (see above) is used for making an asynchronous method call. In the particular case of GUI updates, often one does not need to know when the method completes its operation. However, there are case where one does, for example in the client-server code in the CommunicationLibrary. In those cases one can use a construct called an asynchronous callback method. Basically, one tells C# to report ( call back ) once the method (called asynchronously) completes its work.

71 Asychronous callbacks: Example An asynchronous call: Start receiving connection requests from clients and then Specifying the callback method immediately exit from AcceptClients() so that the program can do other things while waiting for clients to request a connection. The Begin<Something> method is matched by an End<Something> method (example: BeginAccept EndAccept), which returns some customized information, in this case the client socket: Repeat the call, in order to continue listening (asynchronously) for clients.

72 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

73 Concurrent reading and writing In programs that involve asynchronous operations (e.g. multi-threading) one often needs to handle the possibility of concurrent reading and writing. For simple types (e.g. Int) there is no problem: Reading and writing to an Int is a so-called atomic operation, meaning (essentially) that reading and writing cannot occur concurrently. However, this is not the case for more complex objects (e.g. generic lists). Appendix A.5 in the compendium.

74 Concurrent reading and writing ConcurrentAccessExample in the DemonstrationSolution. Begin with a list containing 10 integers, all equal to 1. Then start two threads (running in parallel): One thread that first adds the integer 1 to the list (so that it contains 11 elements), and the removes the first 1 in the list (so that it again contains 10 integers, all equal to 1). One thread that simply computes the length of the list = element sum. If those operations occured in order, the length would always be 10. However, since the threads are asynchronous, the length check can occur between the addition and the deletion!

75 Concurrent reading and writing Incorrect approach: If AddElement() and GetCheckSum() are called asynchonously, the length check in the latter method can occur between Add() and Remove()! Length check (GetCheckSum) may happen here, since AddElement and GetCheckSum are called from different threads (see the code in the MainForm.cs of the example).

76 Concurrent reading and writing Correct approach (one possibility among several): Lock object (used below) The AddElement() method has acquired the lock. Any other code that tries to acquire it must wait until AddElement(). releases the lock:

77 Concurrent reading and writing Locking should only be used when necessary. There is also a TryEnter() method in the Monitor class, which can be used for operations that are not crucial (such as displaying information on the screen). In that case, one can tell C#: show information IF you can acquire the lock, otherwise just skip it. See for example the TryGetItems() method in Memory (AgentLibrary.Memories), which is used when obtaining memory items to be displayed in the MemoryViewer.

78 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

79 Locked bitmaps Pixel values (colors) can be manipulated directly using GetPixel() and SetPixel() (in the Bitmap class). However that method is slow! Alternative: Lock the bitmap in memory to carry out fast pointer operations.

80 Locking a bitmap Locked bitmaps and releasing the lock (after completing the desired image processing operations):

81 Locked bitmaps In most C# code, the code is managed (by the.net framework), meaning that.net handles allocation (and de-allocation) of memory etc. Direct memory access (such as e.g. pointer operations) is unmanaged meaning, among other things, that the programmer is responsible for handling memory resouces etc. In C#, such code is indicated using the unsafe keyword:

82 Locked bitmaps Unsafe code to follow! = 3 for 24-bit color, 4 for 32-bit color Pointer to the first byte of the bitmap Bytes per row (in the bitmap) Note: bytes ordered as B-G-R (not R-G-B)

83 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

84 Parallel computation in C# One can certainly use a normal for-loop for locked bitmaps. However, in order to speed up the computation even more, one can use a parallel for-loop (Parallel.For). Ordinary For-loop Parallel For-loop Note, however, that this must be done with care. When running a parallel for-loop one must be aware that the operations may occur in any order.

85 Parallel computation in C# Will not work! Possible outcome: Two different threads might read the current value, and then add one, each assuming that they have incremented the previous value by one (so that the result is +1 rather than +2)! Correct procedure: Use the Interlocked class in order to handle the addition in an atomic (thread-safe) way.

86 Recommendation and notes Unsafe code Unsafe code is uncommon in C#. Here, you only need to use it in connection with image processing. There are plenty of examples in the ImageProcessor class. Parallel processing Fast, but risk of error. Be careful! If you ever you parallel for-loops, first write the code with a standard for-loop and then check carefully that the parallel version gives the same result. You are not required to use parallel for-loops.

87 Today s learning goals After this lecture you should be able to Describe color spaces and image histograms, Implement and use various elementary image processing operations, Describe and use the ImageProcessing library, Implement and use adaptive thresholding, Implement and use motion detection, Describe methods for face detection and recognition Describe and use elementary multi-threading Explain the concept of asynchronous callbacks Describe and use methods for concurrent access Describe and use locked bitmaps Explain and use the concept of parallel computation in C#

88 To do (for you!) Run (and then step through, with breakpoints) the two applications in the ImageProcessingSolution. Study the ImageProcessor and Camera classes. Study the examples (in the DemonstrationSolution) regarding Threading Concurrent access