2018 International Conference on Physics, Computing and Mathematical Modeling (PCMM 2018) ISBN: 978-1-60595-549-0 A Smart Home Design and Implementation Based on Kinect Jin-wen DENG 1,2, Xue-jun ZHANG 1,2,*, Yue WANG 3 and Mu-jun LIU 4 1 School of Computer, Electronics and Information, Guangxi University, China 2 Guangxi Key Laboratory of Multimedia Communications and Network Technology, Guangxi University, Nanning 530004, China *Corresponding author Keywords: Smart home, Kinect somatosensory device, Skeleton recognition, SVM. Abstract. We proposed a smart home cloud system based on the Kinect camera combining control unit of STM32 with SVM classifier and networking technology. Human skeleton-based identification and posture recognition is fulfilled with Kinect depth sensor, from which users can enter different positions according to their preferences for system to learn, ultimately allow appliances have accurate predictive features. Instead of using any third-party media to complete the home appliances control, our system applies face recognition and bone matching technology to open the door, or alarm stranger invasion. Together with child fall/rain protection window, home electric power detection and web controller, it can enhance the intelligent level and security capability of the current smart home system. Introduction With the development of internet technology and mobile internet terminal platform, technologies such as Internet of Things (IOT) are gradually mature, that leads smart home to step into people's lives. In order to improve the intelligent and security level and achieve autonomous control, we designs a smart home [1] automation system based on WiFi data transmission IOT technology and kernel controller STM32. The introduction of Kinect somatosensory device, obtains the human skeleton information which adopts a new identification method [2], and achieves a high recognition rate. The system has the following benefits: low cost, big profit, high practical marketing value. Composition and Design Block Diagram This smart home cloud system mainly consists three parts: the control terminal, the communication link, and the smart user port [3]. The control terminal mainly takes the STM32 as the processing center which collects, analyzes, processes data according to different pre-set conditions. It sends out control instruction signal, and uploads the processing result to the user port through the link, and then provide real-time feedbacks through the APP to the user. WiFi module and cloud server platform for remote data exchange is used in communication link, so that the control terminal and the client can achieve real-time human-computer interaction; the client is used to receive and display system data sent by the control terminal, intuitively shows the status of the home system with a graphical interface, and enable the user to remotely control the home through the interactive interface. The overall system diagram is shown in Figure 1.1, while the system in the family application is shown in Figure1.2. Kinect Technology Application in the System Nowadays, effective identification becomes more and more important. Existing biometrics have limitations; this paper realizes the design of the acquisition and control system of access control system with Kinect as the core component. Due to the body's bones belong to the biological properties of the physical, which has high stability that cannot be fake. Obtaining the human skeleton information from Kinect as the authentication to judge the legitimacy of visitors, Kinect is a 399
three-dimensional body sensor [4], through its infrared CMOS camera and RGB camera together to form a 3D depth sensor, cooperate with its internal processing chip, the "depth field" imaging could be obtained for bone recognition [5], motion tracking, and face detection. Figure 1.1. The overall system diagram. Figure 1.2. The system in the family application. Identification Based on Human Skeleton Data and Methods to Open the Door The system has 3 ways to open the door, including bones, fingerprints, mobile phones, combined with an automatic alarm. Its original skeleton recognition system was based on WPF developed by VS2013 using C# language that included 6 modules: skeleton data entry, training model, adjustment parameters, character recognition, gesture recognition and real-time character image display. Identifying Bone Structure Principle Inside the Kinect, a skeleton was represented by 25 joints, as shown in Figure 2.1. As one walks into Kinect's view, it automatically looked for 25 knuckles of an individual (standing pose), represented by three-dimensional coordinates (x, y, z). In the system, we eventually need the individual three-dimensional coordinates of 25 nodes, the implementation process was as follows: 1) Opened KinectSensor first, then got BodyFrameSource and ColorFrameSource data sources via KinectSensor for color image data and skeleton data. 2) In the Kinect C # API, event mode was used to access skeletal and color image data, defining frame arrival events. 3) Opened colorframereader and bodyframereader, to obtain the ColorFrame, BodyFrame objects. 4) Opened the color image data stream and skeleton data stream, got the color image and bone data. 5) Processed color image data and converted it to jpg format. 6) Extracting the three-dimensional coordinates of the 25 coordinate points and coordinated conversion using coordinatemapper, and then drew the skeletal frame using WPF Canvas layout. 7) The color images and bone images displayed in real time to the same window, to achieve the skeleton structure recognition. Selection and Optimization of Skeletal Features Data Set Construction. We collected a set of character skeletal data of different height and gender and randomly selected 30 people's data as a sample. Due to the instability of Kinect sensor, the identification of some points might be erroneous, making the distance unstable, such as the distance between ankle and foot. Thus, we chose a relatively stable group of 30 different skeletal fragments as candidate features. Features Selection. The coordinates of the skeletal joint point were determined by the Kinect's three-dimensional coordinate system. As shown in Figure 2.2, the origin (0,0,0) of the Kinect's 3D coordinates was located at the center of the Kinect's infrared camera. According to the extracted coordinates of 25 human skeletal joints, each joint was connected to form a segment of bone, different skeletal features were obtained by calculating different skeletal distances. As shown in Figure 2.3, Dist i represented one of the skeletal features we selected. Assuming that the coordinates of the two joint points a and b are respectively (x 1,y 1,z 1 ) and (x 2,y 2,z 2 ), the distance between the two joint points was calculated as: 400
(1) We recombined 25 joints to make different kinds of features by calculating the distance between different joint points. Finally, the skeleton feature vector could be expressed as: F = (Dist1, Dist2,..., Distn) Where Dist n represented the distance of a skeletal segment. (2) Figure 2.1. Joints position. Figure 2.2. Three-dimensional coordinate system. Figure 2.3. Skeleton feature selection. SVM Classifier Support Vector Machine (SVM), a classifier first proposed by Vapnik [6] in 1995, can improve the generalization ability of classifiers by finding the structural risk minimization. The system uses SVM algorithm for people recognition. SVM Optimization and Feature Selection. The characteristics play a very important role on the performance of the algorithm. By combining different features [7] using 10-fold cross-validation, we found features combination with the highest recognition rate by changing the number of samples, kernel functions, the number of feature points and the parameters c and g multiple times. Figure 2.4 shows SVM optimization result, the recognition rate reaches 97.059% and the top 50% of features combinations is the optimal features. Training and Character Recognition In this paper, 30 human skeletal data were used as training samples, SVM was used to train the classification model using the optimal parameters obtained above. In the stage of character recognition, new skeletal sample data are scanned and inputted into the trained SVM model. The output was the classification result of the sample as shown in Figure 2.5. Figure 2.4. SVM parameter optimization accuracy curve. Figure 2.5. Kinect figure skeletal identification flow chart. System Realization and Experimental Result Analysis In this paper, smart home system using C/S architecture with WiFi Ad hoc network, through the home gateway to establish real-time data control link between control center and client which could achieve various types of home information. The system structure was shown in Figure 3.1. 401
Figure 3.1. The system structure. Figure 3.2. The character recognition system interface. The Function that Console should be Completed. The system control terminal using STM32F103 as the main control chip. Kinect as control terminal sensor, connected with the PC. Data and order were transferred through serial communication between STM32F103. It had many functions: automated appliance controlling, integrated information display, skeleton information matching to open the door and stranger alarm, intelligent child-drop prevention automatic windows, temperature and humidity information integrated display. Online Real-time Character Recognition System Based on Kinect. The system had realized the intelligent access control has three advantages: Skeleton recognition makes up for the deficiency of existing recognition; Using anti-interference ability wireless WiFi signal; Realizing phone as remote control. Figure 3.2 shows the recognition system interface. Home application control using posture recognition based on Kinect. Kinect 3D technology [8] was used to in gesture recognition. According to different users habits to develop different home application control. As shown in Figure 3.3 and Figure 3.4, each command switch corresponded to the posture represent a system custom operating instruction. Client Implementation In order to facilitate system main- tenance and function expansion, we designed a mobile client based on Android operating system and JAVA language. It had the advantages of simple interface, convenient operation and low cost. We could see the mobile client control interface, the APP face recognition login screen and responding screen to voice commands on phone in Figure 3.4. Figure 3.3. Electric control method. Figure 3.4. APP face recognition login. Figure 3.5. Custom operation instruction. Experimental Results Analysis In order to tested the accuracy of character identification, 2 masters of 10 persons, 6 masters of 30 persons and 10 masters of 50 persons were tested respectively, and each species was measured 10 times and average the results was in the Table 3.2. Among them, TN = Number of master data classified as master; FN = Number of stranger data classified as master; TP = Number of stranger data correctly classified as they are; FN = Number of master data classified as stranger; Sensitivity = TP/(TP+FN); Specificity = TN/(TN+FP); PPV = TP/(TP+FP); Accuracy = (TP+TN)/(TP+FN+TN+FP). Experiments show that this character recognition the accuracy of this character identification is high. 402
Table 3.2. The accuracy of character identification. Table 3.3. The accuracy of difference instuctions sheet. In order to test the reliability and stability of posture recognition system, 100 measurements were made for each command under the same conditions. Table 3.3 summarized the accuracy of difference instuctions, which demonstrates that this gesture recognition works robust in our experiments. Conclusion The results of the study indicate that compared with smart home products in the market, the advantages of this system are embodied in security, scalability and low cost. With the successful introduction of Kinect, users simply neeed to give a gesture to control the indoor facilities which effectively improve the intelligence of smart home level. This means it get rid of the dependence on the mobile internet side and free up users hands. The next step will be further used Kinect technology to improve people's intelligent living standards by combining smart home with smart health care. Acknowledgments This work was supported in part by two research support from the National Natural Science Foundation of China (Nos.81460274 and 81760324). References [1] Shi Y, Xie W, Xu G, et al. The smart classroom: merging technologies for seamless tele-education[j]. IEEE Pervasive Computing, 2003, 2(2): 47-55. [2] Zollhöfer M, Martinek M, Greiner G, et al. Automatic reconstruction of personalized avatars from 3D face scans[j]. Computer Animation and Virtual Worlds, 2011, 22(2 3): 195-202. [3] Sun, C., Y. Hsieh and P. Georgescu, A model for HIV transmission with two interacting high-risk groups. Nonlinear Analysis-Real World Applications, 2018. 40: p. 170-184. [4] Zhang Z. Microsoft kinect sensor and its effect[j]. IEEE multimedia, 2012, 19(2): 4-10. [5] Pal D H, Kakade S. Dynamic hand gesture recognition using kinect sensor[c]. Global Trends in Signal Processing, Information Computing and Communication (ICGTSPICC), 2016 International Conference on, 2016: 448-453. [6] Sotiris V A, Peter W T, Pecht M G. Anomaly detection through a bayesian support vector machine[j]. IEEE Transactions on Reliability, 2010, 59(2): 277-286. [7] Ghaddar, B. and J. Naoum-Sawaya, High dimensional data classification and feature selection using support vector machines. European Journal Of Operational Research, 2018. 265(3): p. 993-1004. [8] Loeb, H., et al., Automated recognition of rear seat occupants' head position using Kinect 3D point cloud. Journal of safety research, 2017. 63: p. 135-143. 403