Semi-Automatic People Counting in Aerial Images of Large Crowds

Transcription

1 Semi-Automatic People Counting in Aerial Images of Large Crowds Christian Herrmann, Juergen Metzler, Dieter Willersinn Fraunhofer Institute of Optronics, System Technologies and Image Exploitation, Fraunhoferstraße 1, Karlsruhe, Germany ABSTRACT Counting people in crowds is a common problem in visual surveillance. Many solutions are just designed to count less than one hundred people. Only few systems have been tested on large crowds of several hundred people and no known counting system has been tested on crowds of several thousand people. Furthermore, none of these large scale systems delivers people s positions, they just estimate the number. But having the position of people would be a large benefit, since this would enable a human observer to carry out a plausibility check. In addition, most approaches require video data as input or a scene model. In order to generally solve the problem, these assumptions must not be made. We propose a system that can count people on single aerial images including mosaic images generated from video data. No assumptions about crowd density will be made, i. e. the system has to work from low to very high density. The main challenge is the large variety of possible input data. Typical scenarios would be public events such as demonstrations or open air concerts. Our system uses a model-based detection of individual humans. This includes the determination of their positions and the total number. In order to cope with the given challenges we divide our system into three steps: foreground segmentation, person size determination and person detection. We evaluate our proposed system on a variety of aerial images showing large crowds with up to several thousand people. Keywords: people counting, crowd analysis, aerial image 1. INTRODUCTION There is a large interest in the number of participants at a public event. On the one hand, authorities need the number for security and staff planning aspects. On the other hand, the organizer often measures the success of the event by the number of participants. One traditional method for counting people is to divide the event area into squares of known size. In each square of the grid a trained person estimates the person density. Another conventional method is to position persons with tally counters at all entrances to the event area and let them count the entering people. The drawbacks of these methods are the more or less extensive need of staff and the divergences and inaccuracies of the results when comparing them. Divergences of 50 percent between the two presented methods are not uncommon. Furthermore, once a counting is completed it cannot be reproduced or checked for its correctness afterwards. Hence, divergences between counting results cannot be solved. Because of that, a solution based on aerial images offers the possibility to improve the counting. Using an image as basis for people counting enables to check the counting result anytime. In addition, image processing techniques promise to reduce the workload of people counting and to deliver a more reliable result. A robust people counting system addressing the mentioned problems needs to fulfill two requirements: The first is to return the number of people present in the image as accurately as possible and the second is to return the position of each counted person to achieve traceability. Although these requirements seem consequent, no known system is able to achieve this task on large crowds. Junior et al. 1 distinguish three major approach categories in their survey about people counting: object-based, pixel-based and texture-based. Object-based approaches work with some kind of a person model like head 2, 3 or whole body 4, 5. These approaches provide the number and positions of people, but generally need good image resolution of the person and the people must not occlude each other. These restrictions limit the maximum number of people such systems can count. The test images for these systems contain mostly below 20 people. Lin et al. 2 try to overcome this drawback by estimating the size Electro-Optical Remote Sensing, Photonic Technologies, and Applications VI, edited by G. W. Kamerman, O. Steinvall, K. L. Lewis, R. C. Hollins, T. J. Merlet, M. T. Gruneisen, M. Dusek, J. G. Rarity, G. J. Bishop, J. Gonglewski, Proc. of SPIE Vol. 8542, 85420Q 2012 SPIE CCC code: /12/$18 doi: /

2 , I Jr. 1, e ano. st, 7412:451 " Pk V 4 Figure 1. Image of a large demonstration in Stuttgart, Germany. The crowd shown in this figure contains about 12,000 people. Courtesy of Polizeipraesidium Stuttgart. of a crowd out of a few head detections. So the number of people can indeed be estimated for larger crowds, but without giving the positions of people. Unfortunately, they could not verify their estimated numbers for large crowds due to the lack of ground truth data. Early counting systems from Velastin 6 and Davies 7 use the simple pixel-based methods as their structure is simpler. In a constant scene, segmentation is provided by background subtraction. Based on the segmentation the number of people is determined by the number of foreground pixels. Hou et al. 8 use similar pixel-based methods on larger crowds. Wu et al. 9 estimate the number of people in crowds by local texture analysis mainly based on the gray level dependence matrix. The texture gives an indication about the local crowd density, which again is turned into the number of people. There are approaches which do not even count the people, but only determine the crowd density in a texture-based way 10. It is also possible to combine pixel-based and texture-based methods to count people 11. The combination promises to achieve better results on dense crowds. However, the pixel-based and texture-based methods altogether have one drawback, they just estimate the number of people, but they do not give their positions. So they are very much the same as the traditional non-image based counting methods with respect to a visual plausibility check. The novelty of the system we present in this contribution is its capability to count people in large crowds containing hundreds or thousands of people (see Fig. 1) and to give the positions of people. Another system aspect is that its operation does not require a particular imaging technique. Our purpose is to analyze all images which satisfy certain simple conditions. Due to the large variability of possible input data, the system has to overcome the following challenges: JPG

3 Single image Just one image is available for the counting task. The image is taken by a digital camera or it may be a mosaic image generated from video. Arbitrary crowd density Crowd density reaches from single or loose groups of people to a densely packed crowd, e. g. in front of a stage. Large number of people Crowds can contain thousands of people. Arbitrary person size The person size in the image can vary heavily from image to image. Outdoor scenes Disturbances by shadows or different lighting conditions have to be considered. Unknown scenes The location is variable and not known beforehand. The only requirement for the input image is that it must be an aerial view of the crowd. The image does not need to be taken from an airborne platform. Each camera location that provides an approximately vertical view on the crowd will suffice. This view ensures a homogeneous representation of the crowd in the image. Thus lamp poles, mobile cranes and high-rise buildings are, for example, suitable camera positions. 2. COUNTING SYSTEM As we do not only want to extract the number of people present in the image, but also their positions, a modelbased approach is necessary. However, a model-based system alone is not able to cope with the given images, as a single person might barely be visible in a dense crowd. Because of that, we choose a design following the approach of Schofield et al. 5. In this approach, persons are searched after a foreground segmentation by a rectangular person model. This model can tolerate occlusions between the persons. In addition to these two steps, we need a third one to determine the person size in the image. This is necessary because of the different scenes in the case at hand, whereas Schofield s approach solely works with a constant scene and camera perspective. Hence, the person size is not changing from one image to another. For a generally applicable solution, this restriction needs to be overcome. The three steps of our proposed system are shown in Fig. 2. Segmentation Person size determination People counting Figure 2. Major processing steps of our counting system. After successfully loading an image (Fig. 3a), the first step is a manual segmentation supported by several tools (Fig. 3b). An automatic background segmentation is not provided as unknown scenes with no background model are assumed. In the second step, the person size has to be determined. For this, the user tags several persons with a rectangular frame. This frame must have the same size as the tagged person (Fig. 3c). The person size gives the size of the person model for counting and localizing the persons in the third step. In this step no user interaction is necessary. Finally, the result is displayed as rectangles on the original image and the people s positions are listed (Fig. 3d). 2.1 Foreground segmentation As only single images are provided as input data, segmentation is quite a challenging task. Usual approaches like background subtraction or learning are not available, unlike in systems using video data. To overcome this problem, a manually supported segmentation approach is chosen. The system provides three different tools the user can utilize to get a foreground mask. These are a simple rectangular marking tool, a region detection tool and a color selection tool. As examples, we use regions of the image of a large demonstration shown in Fig. 1. The rectangle tool is a simple tool to quickly mark large regions of background. In general, there are regions in images of crowded public events with no persons in them. There might also be large buildings that could be

4 41:1=2) óane Sprache ; G F R -- s,ycak,j.,yi.'lolüll G F R 238e1 Subje14z1119 rg813 ;prame NOW A,, G F R.. Subgexnrsm óa.i Sprache G F R suyexnrsm [ g 5 3 gl imi (a) (b) (c) (d) ubjekgrbbebenimmen AwM1ltlerS bjene ANOm2I 26e]a61ung Figure 3. Processing of a simple example. (a) Input image showing 6 persons. (b) Segmentation mask. (c) Manually tagged persons for person size determination: black rectangles. (d) Counting result: white rectangles. ir +r, e0 fi í II (a) (b) (c) (d) (e) (f) Figure 4. Image regions of Fig. 1 and the respective tool output. Segmented background regions are surrounded by white frames. (a),(b) In- and output of rectangular marking tool. (c),(d) In- and output of region detection tool. (e),(f) In- and output of color selection tool. eliminated out of the foreground this way. In a short period of time and with a few mouse clicks, it is possible to segment large areas of the image containing no people (see Fig. 4b for an example). As the capabilities of the basic rectangle tool are limited, we provide another tool for segmenting larger regions of background. After excluding the areas showing no people, the areas with sparse people have to be processed. Typical examples for these regions are streets or lawn at the outer parts of a public event. In general, the ground has quite a homogeneous appearance which can be used for segmentation. The region tool detects a connected region simply by clicking into it. This is done by a region growing algorithm starting with the pixel marked by the user as seed point. Further pixels are included into the region if they are similar enough to a neighboring pixel that already belongs to the region. A typical example with uniform background and sparse people is given in Fig. 4c. The output of the region tool is shown in Fig. 4d. The most difficult task is still remaining: segmentation of dense crowds. Such image regions show just some small spots of ground between people. These spots are not connected to each other so the previously introduced region detection tool is quite ineffective here. However, spots close to each other generally show a similar color

5 t t 10! i t It ti Àt.. If r r r Figure 5. Iterative person detection. Each column gives the current data of one processing step in the counting process. First row shows the input images with so far counted persons, second row shows the foreground masks M and last row shows the density maps D. which can be used for segmentation. We design our color tool in a way that it marks all pixels with a similar color to a seed pixel as background (Fig. 4f). Pixels are set as background if their distance to the seed point does not exceed a threshold. This will prevent wrong segmentations of similarly colored pixels that do not belong to the same ground area. The result of the segmentation is a foreground mask M indicating all the people in the image (Fig. 5, second row, first image). 2.2 Person size determination One remaining problem is the unknown size of the people. The size s has to be measured as width w and height h in pixels. With this size we can adapt our model for person detection to the image at hand. Because we demanded the input images to have a vertical view on the scene, size differences between the persons in one image are insignificant. Thus, we can use one model for a whole image and do not have to modify it according to the position. A good way to get the person size s in the image is a user supported approach. The user marks several persons by drawing a frame around each person like in Fig. 3c. The final size s is calculated as average of all marked person sizes. According to experience, a small amount of 5 to 10 people is sufficient to reliably determine the person size s. This results in a small user effort and high accuracy. 2.3 Person detection With the foreground mask M and the person size s, we start a model-based counting. The main idea is to detect persons in the foreground M with a person model P and remove them afterwards out of the foreground so they

6 will not be detected again. Removing a person could simply be done with a foreground subtraction of the person model. We use a rectangular person model of exactly the same size s as a person. The likeliness of the model being at a person s position is measured by the foreground density inside the rectangle. Calculating the density for all possible positions of the model results in a density map D. We store the density at the position of the model s upper left corner pixel. As not all image pixels ensure that the model is completely included in the image, the remaining density values at the border are set to zero. The density at a valid corner pixel p x,y is calculated as follows: D p = 1 w h x+w y+h m i,j, (1) i=x j=y w is the person s width, h is the person s height, both in pixels and m i,j is the value of the foreground mask M at position (i, j). The most likely position of a person is at the position of the largest density value v in D. So this maximum is searched and if v exceeds a threshold S, a person is assumed there. For each found person the total person counter n c is increased by one and the position (x i, y i ) is stored to a list. Since an already found person should not be detected twice, it will be removed from the mask M. This is done by declaring all foreground pixels in the person model rectangle to background pixels. After the reduction of M, the density map D has to be updated at all according positions. This whole process is repeated iteratively until the maximum density v no longer exceeds the threshold S. This process is shown step by step in Fig. 5 for a small example. As result, we get the total number of detected persons and their respective image coordinates. 3. RESULTS We measure the performance of our system in three categories: user effort, computational effort and accuracy. In each category the system is compared to a totally manual counting of the people shown in the image. Manual counting is done similar to the person size determination step: each person is tagged by a rectangular frame. A simple tool for this task allows to load an image and draw rectangles around the people with the mouse. Our test dataset includes 16 images showing a crowd or part of a crowd. They contain between 2 and 12,000 people. We included the image with 2 people for comparison purposes. Most of the images contain about one hundred people. Furthermore, the actual number n a of people in the image has to be known for testing purposes. In most cases, this reference count must carefully be done by hand which means that individuals need to be distinguishable in the image. We determined the actual number n a either from context knowledge or as average from multiple thorough counting cycles by hand. The exact number of people in an image is hard to determine due to occlusions. Unlike the manual counting, which will be compared to the performance of our system, the reference counts were not made as fast as possible, but as accurate as possible. For measuring the accuracy, we take the relative difference between the counted number n c and the actual number n a of persons in the image: d = n c n a n a 100%. (2) The algebraic sign of the divergence d indicates if the number of persons is over- (positive) or underestimated (negative). Fig. 6 shows the divergence in dependency of the actual number of people in the image. The manual counting reaches an average divergence of about 4 percent, whereas our system reaches an average divergence of 24 percent. Although this is not as good as the manual count, it is better than the performance of traditional non-image based methods. In addition, the semi-automatic system tends to produce better counting results with higher numbers of persons in the image. User and computational effort are measured as necessary time in seconds. The user effort t u for our semiautomatic system contains the time for foreground segmentation t f and person size determination t s. Within the computational effort t c all processing times of the person counting step are accumulated. For manual counting there exists of course no computational effort, only the user effort t u. Because the processing duration depends a lot on the input image, we define a normalized version t for each time measure t. With division by the actual

7 75 Divergence in % Number of persons Figure 6. Counting divergence d in dependency of the number of persons n a in the image. Compared between manual counting (boxes) and semi-automatic counting (crosses). Overall effort in s Number of persons User effort in s Number of persons Computational effort in s Number of persons (a) (b) (c) Figure 7. Time effort per person in dependency of the number of persons n a in the image. Compared between manual counting (boxes) and semi-automatic counting (crosses). (a) Overall effort t o, (b) User effort t u, (c) Computational effort t c. number of persons n a in the image, the time for counting one person is calculated: t u = t u n a, t c = t c n a. (3) This allows to compare time measures between different images. The results of the time measurements are shown in Fig. 7. Manual counting generally takes about 1 second per person. But with higher numbers of persons, according to our experience from about 500 persons on, manual counting takes more and more time per person as concentration and attention of the user tend to fade. This explains the single outlier at about 850 persons. For this reason, we did not count the images with more than one thousand people by hand. The actual number of persons for these images was given by their providers. The semi-automatic system shows the inverse trend with an increasing number of people in the image. Due to the decreasing user effort t u, the overall effort t o decreases. One reason is the constant effort for person size determination t s which is independent of the image size and the number of people. In addition, large images of respective crowds can be segmented more efficiently. In this case, larger regions in the image can be segmented in one step. The computational effort is of no consequence to the overall effort, even though, it seems to increase with higher number of persons. In summary, our semi-automatic system reliably exceeds the counting speed of the manual count if the image contains more than one hundred persons.

8 The speedup increases with the number of persons. Counting the image with 12,000 people takes about 9 minutes on the semi-automatic system, whereas manual counting would take more than 3 hours if the optimistic counting speed of 1 second per person is assumed. This results in a speedup factor of at least 20 for the semi-automatic system. 4. CONCLUSION We built an image-based system for counting large crowds. The counting is faster than counting the people in the image by hand and reaches an accuracy that is comparable to the accuracy of the traditional non-image based methods. Because our system provides the people s positions in addition to the number of people present in the image, an ordinary person can check if the counted number is plausible. Having the positions also enables a post-processing of the counting result. If at some spot in the image the segmentation failed, it is possible to delete counted persons there or add them manually. The still rather large user involvement in the system might be reduced by an automated segmentation. This challenge will be the next step to a fully automated counting system. Even though, this might not turn out as an easy task on single images as input data. REFERENCES [1] Junior, J., Musse, S., and Jung, C., Crowd analysis using computer vision techniques, IEEE Signal Processing Magazine 27(5), (2010). [2] Lin, S., Chen, J., and Chao, H., Estimation of number of people in crowded scenes using perspective transformation, IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans 31(6), (2001). [3] Zhang, X. and Sexton, G., Automatic human head location for pedestrian counting, in [Sixth International Conference on Image Processing and Its Applications, 1997], 2, , IET (1997). [4] Zhao, T. and Nevatia, R., Bayesian human segmentation in crowded situations, in [IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Proceedings], 2, , IEEE (2003). [5] Schofield, A., Stonham, T., and Mehta, P., Automated people counting to aid lift control, Automation in Construction 6(5-6), (1997). [6] Velastin, S., Yin, J., Davies, A., Vicencio-Silva, M., Allsop, R., and Penn, A., Automated measurement of crowd density and motion using image processing, in [Seventh International Conference on Road Traffic Monitoring and Control, 1994], , IET (1994). [7] Davies, A., Yin, J., and Velastin, S., Crowd monitoring using image processing, Electronics & Communication Engineering Journal 7(1), (1995). [8] Hou, Y. and Pang, G., People counting and human detection in a challenging situation, IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans 41(1), (2011). [9] Wu, X., Liang, G., Lee, K., and Xu, Y., Crowd density estimation using texture analysis and learning, in [IEEE International Conference on Robotics and Biomimetics, ROBIO 06.], , IEEE (2006). [10] Hinz, S., Density and Motion Estimation of People in Crowded Environments Based on Aerial Image Sequences, International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 38(1-4-7/W5), on CD (2009). [11] Li, W., Wu, X., and Zhao, H., New techniques of foreground detection, segmentation and density estimation for crowded objects motion analysis, Information and Media Technologies 6(2), (2011).

9 Year: 2012 Author(s): Herrmann, Christian; Metzler, Jürgen; Willersinn, Dieter Title: Semi-automatic people counting in aerial images of large crowds DOI: / ( Copyright Society of Photo-Optical Instrumentation Engineers. One print or electronic copy may be made for personal use only. Systematic reproduction and distribution, duplication of any material in this paper for a fee or for commercial purposes, or modification of the content of the paper are prohibited. Details: Society of Photo-Optical Instrumentation Engineers -SPIE-, Bellingham/Wash.: Electro-Optical Remote Sensing, Photonic Technologies, and Applications VI : 24. Sept. 2012, Edinburgh, United Kingom Bellingham, WA: SPIE, 2012 (Proceedings of SPIE 8542) ISBN: Paper 85420Q