Reading Barcodes from Digital Imagery

Transcription

1 Reading Barcodes from Digital Imagery Timothy R. Tuinstra Cedarville University Abstract This document was prepared for Dr. John Loomis as part of the written PhD. candidacy exam by Timothy R. Tuinstra. This report provides a look at the automatic detection and decoding of barcodes found within imagery. Such techniques could conceivably be used in the retail, manufacturing, inventory and travel industries as well as others. A general description of barcodes is included as well as a review of some of the techniques used to automatically locate barcodes within larger images containing additional objects. The author will also present some results from implementing a variant of these techniques. I. INTRODUCTION Barcodes have become very ubiquitous in the world around us. The most common barcodes that we see are the Universal Product Codes (UPC) used to tag merchandise in the local grocery stores. Use of laser barcode scanners has made checking out more quick and convenient for most of us. Other barcodes are seen on library books, and even luggage is tagged at the airport with barcodes containing routing and other information. Barcodes were invented in 1949 by Bernard Silver and Norman Woodland, at that time graduate students at the Drexel Institute of Technology. Silver and Woodland received a patent for their invention in Barcodes, however, were not used in retail business until 1967 when the first barcode scanner was installed by RCA in a Kroger store in Cincinnati. The UPC code was not standardized until later and the first UPC code was scanned in a supermarket in Troy, Ohio using a scanner built by the National Cash Register (NCR) company. The pack of gum scanned is today in the Smithsonian Museum in Washington DC. The principle behind barcoding is to encode alphanumeric characters in bars of varying widths. While many barcodes today are of the rectangular variety known to grocery shoppers world-wide, original barcodes consisted of concentric circles of ink of varying widths. In this report the basic theory of barcoding will be described, along with the decoding of UPC barcodes. In addition we will discuss how barcodes may be found in a larger image through a process known as barcode localization. II. THE BASIC THEORY OF BARCODES The basic premise of barcodes is that alphanumeric information may be coded into bars and spaces of varying widths printed on a product label for instance. Barcodes use a one dimensional coding scheme. The height of the barcode provides added redundancy to the system if part of the symbol becomes damaged or occluded. In this work we will focus on the Universal Product Code (UPC) but many of the results could easily be extended to other coding schemes such as the Code 39 system which has been used by the U.S. government and specifically the Department of Defense to maintain inventories. UPC codes are the codes that are commonly placed on merchandise in stores. The UPC code contains a code indicating the manufacturer of the product as well as a number indicating the product family and product. UPC barcodes contain no price information, but product information is passed after scanning to the in-store product database and the appropriate price is retrieved. A good description of UPC barcodes can be found in [2], [5]. UPC codes contain twelve decimal numbers from 0-9. The first eleven digits are data and the twelfth digit is a checksum to ensure that the code was scanned correctly. Some smaller sized products such as cans of soda may contain eight digit UPC barcodes which are called zero-suppressed barcodes. These codes are achieved by strategically suppressing zeros in the original twelve digit code. Each digit in the UPC barcode is encoded using two bars and two spaces. The bars are usually a dark color that absorbs light and the spaces are a lighter color tends to reflect and scatter light. Each digit in the barcode is represented by a four number codeword encoded in the width of the bars and spaces. This codeword is divided into seven segments and a bar or a space in a segment can be one, two, or three units wide. Naturally the sum of the widths of the bars and spaces making up a single digit must be seven units. An example annotated UPC symbol is shown in Figure 1. The codewords for UPC barcodes is shown in Table I. UPC barcodes also contain several guard codewords both at the ends of the symbol and in the middle. At the beginning and at the end are the codewords [1, 1, 1]. Also, between the first six digits and the second six digits in a UPC that is not zero-suppressed is the codeword [1, 1, 1, 1, 1]. UPC barcodes are designed in such a way so that the direction of the scan does not matter. This is very important for ease of use and speed especially at the checkout counter. This is achieved by encoding the digits in such a way that the reversal of any codeword is not a valid code. For example for the digit 0 we have the codeword

2 [3, 2, 1, 1]. If this was read backwards it would be [1, 1, 2, 3] which is not a valid codeword as can be seen. A scanner therefore just has to look for a string of invalid codewords to detect the fact that the code was scanned backwards. Guard Patterns Number System check sum Manufacturer Product ID Fig. 1. Example UPC code. TABLE I UPC WIDTH CODES. Decimal Number Width Code 0 3, 2, 1, 1 1 2, 2, 2, 1 2 2, 1, 2, 2 3 1, 4, 1, 1 4 1, 1, 3, 2 5 1, 2, 3, 1 6 1, 1, 1, 4 7 1, 3, 1, 2 8 1, 2, 1, 3 9 3, 1, 1, 2 The barcode checksum exists to ensure that the correct data was scanned. The calculation of the checksum proceeds by taking the first 11 digits and computing the sum of digits in even positions and the sum of the digits in odd positions. This can be represented as O = X[1] + X[3] + X[5] + X[7] + X[9] + X[11] (1) E = X[2] + X[4] + X[6] + X[8] + X[10] (2) where X[i] is the digit in the i th position. We then compute Z = O + 3E. (3) The checksum digit is then calculated by computing CS = 10 mod(z, 10) (4) where mod is the modulus operation indicating the remainder of dividing Z by 10.

3 A. Theory III. DECODING BARCODES Barcodes are decoded based on their intensity profile. In many cases this intensity profile detected using the familiar cross shaped laser scanner seen in grocery stores. The intensity profile is the one-dimensional function across the barcode needed for proper decoding. The two-dimensional nature of the barcode is important, however, if there is any damage to the barcode. It would be nice if the profile of the barcode were perfectly nicely behaved, but noise and packaging imperfections cause non-constant intensity levels in the bar and space regions. Sometimes this noise and distortion can be severe enough to actually cause decoding errors. In order to minimize these kinds of effects then, a one-dimensional median filter was chosen to minimize noise. This non-linear filter was chosen over a linear filter because it does better at preserving the boundaries between bars and zeros which is precisely what needs to be preserved for decoding. A length 3 median filter seemed to work well for all of the barcodes encountered during experimentation. A noisy profile along with a filtered profile is shown in Figure Noisy Profile Filtered Profile Fig. 2. Example of a noisy profile with the median filtered profile. Following the median filtering, the next step was to upsample the profile. This is extremely important because the relative widths of the bars and spaces needs to be computed, and the higher the resolution the better this can be done. For these experiments an upsampling of 4:1 was used and worked in a satisfactory way. The median filtering and the upsampling are essentially pre-processing steps. Once pre-processing is completed, the one-dimensional profile was passed to the decoder. The decoder normalized the profile and performed a thresholding to create a binary function. The main job of the decoder is to step through the binary profile data and to look for transitions. The first four transitions found are deemed to be the guard word. Once the guard word is passed, each additional set of four transitions can be considered to be one codeword. The distances between the transitions are measured and the sum of these computed over four transitions. It is then possible to compute the width of each bar or space relative to the total width of the codeword. These can be multiplied by seven and rounded giving a codeword for each digit. Some UPC barcodes are intentionally compressed so that they fit on smaller products such as soda cans. These codes follow the UPC-E standard and are called zero-suppressed codes. Essentially, these codes take the twelve digit standard UPC code and compress it down to a six digit code. This is done by strategically removing unnecessary zeros. An additional feature of the UPC-E code is that the checksum digit is encoded within the polarity of each codeword. In other words, each of the six codewords can be either in forward or reversed direction. Table II gives the proper decoding scheme for these checksum digits. A description of the UPC-E standard can be found in [6]. At this point it is important to check to see whether the barcode was read in the proper direction. This is simply a task of checking to see whether the codewords produced are proper UPC barcode codewords. If this is not the case, the intensity profile is reversed and the decoding process repeated.

4 TABLE II CHECKSUM CODING IN UPC-E SYMBOLS. 1 INDICATES ONE POLARITY AND 0 INDICATES THE REVERSE. NOTE THAT THE POLARITIES MAY BE FLIPPED. Checksum Code

5 B. Results In order to test the results of the barcode decoder designed, seven example barcodes were decoded. The decoding scheme described above and used for these experiments worked very well. The UPC codes used along with their original and filtered profiles are shown in Appendix A. Table III shows the UPC numbers decoded from each of the symbols shown. Also, a 1 in the check column indicates that the code passed the checksum test. All of the sample UPC barcodes were read correctly including one UPC-E or zero-suppressed barcode. TABLE III RESULTS OF DECODED UPC BARCODES. UPC Decoded Number Check

6 A. Theory IV. FINDING BARCODES The process of localizing barcodes found in a larger scene that contains many other image features may be cast essentially as a texture segmentation problem. Several characteristics of barcodes differentiate them from other features that may be visible within imagery. The most basic feature is that they contain strong bandpass spatial frequency content in one direction and not in the other. We should therefore expect that there are strong gradients in one direction as well. These are features that we can exploit to accomplish the barcode segmentation task. Since spatial frequency content is a strong indicator of the presence of barcodes, it may be possible to use a Fourier transform or a discrete cosine transform (DCT) of the image to localize the barcodes. However, Fourier transform methods give an indication of the frequency content of the entire image and not just for specific regions. What is required then is a method for finding local frequency information. Jain and Chen make use of multi-scale, mult-directional, Gabor filtering to find specific regions of high frequency content[1]. Another possibility would be to use the discrete wavelet transform (DWT) to localize regions of high spatial frequencies. This was suggested by Oktem [4]. A simple and practical approach was taken by Viard-Gaudin et al when they made use of the idea of directional gradient density [3]. This is the approach implemented by this author with some slight variations to be described later. The principle behind this particular algorithm is that in the barcode region, we should expect a high density of unidirectional gradients. This is a key point since there may be other features in the image such as text which provide a strong gradient magnitude response, but the gradient direction is not unidirectional. To estimate the image gradient in the x and y directions, the Sobel kernels were employed. Viard-Gaudin et al make use of four Sobel kernels oriented at 45 to attempt to locate barcodes oriented in any of eight different directions. In the implementation for this project only two orthogonal Sobel kernels were employed. The code written here could be modified to cover the more general case. Once the gradient is computed, two different operations are performed. The first involves thresholding the gradient to keep only regions of the image where the gradient energy is strong. Barcodes should exhibit strong gradient magnitude. The second operation is more computationally intensive and involves a nonlinear neighborhood operation on the gradient direction information to find whether a majority of the gradients in a neighborhood point in the same direction. To do this a two-level image is formed where a pixel is given the value 1 if the maximum gradient direction is 0 and a 2 if the maximum gradient direction is 90. The neighborhood operation then looks at each square neighborhood around a given pixel to determine whether a majority of pixels point 0 or 90. The difference threshold is passed as a parameter to the barcode finder software. To find the initial binary image then, the gradient is found, the directional density is found and both are thresholded. A logical and operation is then performed on a per pixel basis. At this point morphological processing is performed to clean up the segmented image. Examples of what these binary images look like will be shown in the results section. The morphological processing performed on this binary image segmentation is composed of several steps. The first step is to do a hit-or-miss operations using lines as the structuring element. What is found is that the areas of the binary image containing the barcodes contain long lines. The hit-or-miss operation tends to select these regions while removing areas that have a few pixels turned on here and there. The length of the structuring element line is a parameter that can be passed to the software. After the hit-or-miss operation, a dilation using a large square structuring element is performed. The purpose of this dilation is to try to connect the lines of the barcode which are not presently connected. Naturally this also accentuates any non-barcode regions that are turned on, but these can be removed later using other criteria. A square structuring element was used because it matches the shape of the barcodes and tends to give things a rectangular shape. If we were allowing for barcodes to be oriented in other angles, a circular structuring element might have been used. The next step was to perform morphological erosion to get rid of long thin lines within the image. Many product labels have lines on them and these are preserved in the line hit-or-miss and in the dilation, but will not be preserved when an erosion using an even larger structuring element is used. Now that the barcode regions are connected, the only result here is to shrink them and this is taken into account by immediately performing one last dilation after removing all the regions that are deemed too small or too large. These parameters are also passed to the code and should be based on a priori knowledge concerning the size of the barcodes. One final operation performed was a solidity test which compared the number of pixels turned on in a region to the convex hull of the region. This yields a fraction which gives a feel for how much open space remains in the image. This is especially good for removing large regions such as text which may have survived up to this time but have lots of holes in them. This worked very successfully in a number of the images to remove text regions which remained. At this point if all is well, only barcode regions remain and these can be excised from the original image and passed to the barcode decoder.

7 There are a number of parameters which are passed to the barcode finder software. A Table of the values used for these examples is shown in Table IV. Future research could be done to help determine how to automatically optimize these parameters by taking advantage of knowledge about the size of the barcodes in the image etc. TABLE IV BARCODE FINDER SOFTWARE PARAMETERS Parameter Value Directional Density Neighborhood size 3x3 Directional Density Threshold 0.67 Gradient Threshold 0.35 Length of Hit-or-Miss Line 5 pixels Size of Dilation Structuring Element 17x17 Region Size Threshold 25,000 pixels Solidity Threshold 85%

8 B. Results Software to implement the algorithm described above was written and tested on two images, one containing four UPC barcodes and one containing three barcodes. In addition to the barcode localization algorithm described above, code was added to extract all of the barcodes, rotate them if they are oriented at 90 and pass them to the decoder and the checksum module. This worked well with all but the zero-suppressed UPC barcode being read correctly. Shown in Figure 3 and Figure 4 are the two images containing multiple UPC barcodes Fig Test image #1 for finding barcodes Fig Test image #2 for finding barcodes. The barcode localization algorithm was run on these two images producing the output segmented images shown in Figure 5 and 6. These are very pleasing results. The individual barcodes were then extracting by taking a rectangular subimage for each barcode from the upper left pixel to the lower right pixel for each barcode. These subimages were returned in a structure for decoding. Prior to returning the subimages, their orientation (vertical or horizontal) was detected by taking the gradient using the Sobel kernals and checking to see which orientation had a stronger total magnitude. Vertical barcodes were rotated so that all barcodes had the same orientation. Since scan direction is irrelevant because of the decoder software, upside down barcodes do not matter so this is not an issue. Extracting the intensity profile was a delicate issue. It turns out that some of the barcodes had defects in the middle which caused inaccurate intensity profiles and hence decoding errors. An example of a damaged barcode is shown in Figure 7. Note the reflection across the middle of the code. It is therefore not sufficient to simply take the middle row of pixels as the intensity

9 Fig. 5. Barcode segmentation results for Test image # Fig. 6. Barcode segmentation results for Test image #2. profile. Nor was it appropriate to take an average of all of the rows because some of the codes were at an angle slightly out of axis and averaging them would have smeared them into unreadibility. The solution to this problem was to try the middle row as the intensity profile, to decode it and to see how many codewords were found. If there were any unreadable codewords in the decoded data, a different row was selected until the code decoded correctly. This worked well and led to the ability to read all of the previously unreadable codes. This is a perfect example of the height redundancy built into the barcode symbols. All barcodes read in this fashion from the larger image were error checked by verifying the checksum value decoded. The barcode numbers read from the two images are shown below in Tables V and VI. TABLE V UPC NUMBERS READ FROM TEST IMAGE #1. UPC Decoded Number Check

10 Fig. 7. Example of a barcode corrupted by a reflection. TABLE VI UPC NUMBERS READ FROM TEST IMAGE #2. UPC Decoded Number Check

11 V. CONCLUSION Barcodes can be read from digital imagery. Barcode localization is possible by using texture segmentation techniques and binary image morphology. By getting a good intensity profile of the segmented barcode image, we may decode the codes accurately. VI. APPENDIX A Below in Figures 8 through 14 are shown the UPC barcode symbols used for testing the decoder Original Profile Filtered Profile Fig. 8. UPC Original Profile Filtered Profile Fig. 9. UPC 2.

12 250 Original Profile Filtered Profile Fig. 10. UPC 3. Original Profile Filtered Profile Fig. 11. UPC 4.

13 Original Profile Filtered Profile Fig. 12. UPC Original Profile Filtered Profile Fig. 13. UPC 6.

14 Original Profile Filtered Profile Fig. 14. UPC 7.

15 VII. APPENDIX B Figure 15 displays the initial binary image resulting from thresholding the gradient and taking the logical and of this and the results of thresholding the directional density Fig. 15. Initial binary image. Figure 16 shows the results of doing a morphological hit-or-miss operation on the initial binary image with linear structuring elements Fig. 16. Step 1: Result of morphological hit-or-miss operation. Figure 17 shows the results of performing a dilation on the results of Step 1. Notice how the barcode regions have been connected. Figure 18 shows the results of eroding the binary segmentation to remove long tail structures. Figure 19 shows the results of applying a size constraint on the segmentation. We do not desire to keep any regions that are smaller than a typical barcode. Notice how at least one text region is kept here, but it may be removed by applying a solidity constraint. Figure 20 shows the results of applying the solidity constraint on the segmentation. What is left is the four barcodes!

16 Fig. 17. Step 2: Result of morphological dilation Fig. 18. Step 3: Result of erosion to remove tails.

17 Fig. 19. Step 4: Result of removing small regions Fig. 20. Step 5: Result of removing non-solid regions. Final Segmentation!

18 REFERENCES [1] A. K. Jain and Y. Chen, Bar Code Localization Using Texture Analysis, Proceedings of the Second International Conference on Document Analysis and Recognition, pp , [2] T. Pavlidis and J. Swartz and Y. P. Wang, Fundamentals of Bar Code Information Theory, Computer, vol. 23, no. 4, pp , [3] C. Viard-Gaudin and N. Normand and D. Barba, A Bar Code Location Algorithm Using a Two-Dimensional Approach, Proceedings of the Second International Conference on Document Analysis and Recognition, pp , [4] R. Oktem, Bar Code Localization in Wavelet Domain by Using Binary Morphology, Proceedings if the IEEE 12th Signal Processing and Communications Applications Conference, pp , 4. [5] How UPC Bar Codes Work, viewed 4 October, 5, [6] UPC-E Symbology, viewed 4 October, 5,