Sign Detection of Future Malicious Behavior for Nuclear Security
公開日:
カテゴリ: 第13回
1. Introduction
After Fukushima Daiichi accident, the importance of nuclear security is increased, especially sabotage by insider should be considered. In response to the increasing threats to Nuclear Power Plant, human malicious behavior detection is necessary for nuclear security. Hand motion has high contribution to human activity thus it’s significant in the detection of inside malicious behavior. For this purpose, Kinect v2 is proposed to be used. Fig.1 Microsoft Kinect v2
Kinect v2 is the new version of game controller technology introduced by Microsoft. It is composed of two cameras, namely a RGB and an infrared (IR) camera (as shown in Fig.1), The RGB camera captures color information with a resolution of 1920x1080 pixels, while the IR camera is used for the real-time acquisition of depth frame with a 512x424 pixels resolution. Also, Kinect v2 provide a new function called body index frame which can be used to segment human body from background.
However, there is no hand specific data available for gesture recognition, although it does include information of the joints between hands and arms. [1] Time series data analysis is a useful method in abnormal behavior detection. In this case, we capture hand motion time series and distinguish normal and abnormal behavior. However, abnormal behavior detection isn’t enough for nuclear security, sign detection of future malicious behavior should be considered. In this research, we tracked hand motion by using Kinect v2. The hand region is segmented and distinguished by depth data and body index frame and developed a real time system in capturing fingertip positions. Moreover, to derive future hand motion, AANN (Auto-Associate Neural Network) will be used. Hand motion are detected by capturing the fingertips positions. Most previous fingertip tracking algorithms usually can track the fingertips for only stretched fingers and the fingertips of bend fingers are usually difficult to be detected. In this research, stretched fingers and bend fingers of both left and right hands were classified as different parts and not only the fingertips of stretched finger but also bend fingers can be detected. The hand motion system of this research consists of three main parts: Hand Region Classification, Hand Contour Pixels Capturing, Fingertips Calculation. We developed our system using Visual Studio 2015 with C# as programming language and body index data, depth data and skeleton data of Kinect v2 were used to calculate fingertip positions.
2.1 Hand Region Classification
At first, body regions should be segmented from the background.
2. Hand Motion Detection For this purpose, Kinect body index frame is used. Body index frame is the new function provided by Kinect v2, which has 512 ×424 resolution frame and at most six body can be detected. For each pixel in body index frame, Kinect system set an index: 1) If the pixel belongs to the region of human body, the index is 0~5; 2) If the pixel belongs to the background, the index is 255. By using body index frame, human body can be segmented from background. After body segmentation, we have to do hand region classification. For each hand, stretched hand region and bend hand region should be divided. As shown in Fig.2, the distance from each hand region to Kinect camera are different, in this case, depth data can be used for hand region classification. Let D(n) be the set of pixels’ depth in Kinect depth frame, I(n) be the set of pixels’ body index in Kinect body index frame, dmax be the wrist point which is in the farther distance to the Kinect Camera, dmin be the wrist point which is in the closer distance to the Kinect Camera and d is the threshold to calculate bend hand pixels. For each pixel’s depth D(i) in D(n): 1) If D(i) ? dmax and D(i) > dmax ? d, meanwhile, I(i) ≠ 255, then I(i) can be classified as the farther distance stretched hand pixels; 2) If D(i) ? dmax ? d and D(i) > dmin, meanwhile, I(i) ≠ 255, then I(i) can be classified as the farther distance bend hand pixels; 3) If D(i) ? dmin and D(i) > dmin ? d, meanwhile, I(i) ≠ 255, then I(i) can be classified as the closer distance stretched hand pixels; 4) If D(i) ? dmin ? d, and D(i) ≠ 0, meanwhile, I(i) ≠ 255, then I(i) can be classified as the closer distance bend hand pixels; Finally, pixels can be divided into four parts: ● Left stretched hand pixels; ● Right stretched hand pixels; ● Left bend hand pixels; ● Right bend hand pixels. Fig.4 Results of hand region classification (red pixels: stretched hand region; yellow pixels: bend hand region; black pixels: background) 2.2 Hand Contour Pixels Capturing Hand contour can be used to calculate the stretched fingertips. For each pixel, it has four neighbor pixels: left neighbor pixels, right neighbor pixel, above neighbor pixel and below neighbor pixel. And if a pixel is stretched hand pixels and more than one of its neighbor pixels are background pixel, it can be classified as hand contour pixels. (as shown in Fig.5) Fig.2 Hand region classification by depth To reduce computing complexity, hand pixels should be limited in some specific region. For this purpose, we use “rectangle limitation algorithm”. The algorithm can be summarized as follow: 1) Get center points of both left and right hand; 2) Build four rectangles from center point with width W and height H for each hands; (The values of W and H depends on the distance from body to the Kinect camera) 3) Only the pixels within this region can be classified as hand pixels; (As shown in Fig.3) Fig.3 Rectangle limitation algorithm used for reducing computing complexity The classification results are shown in Fig.4. - 128 - Fig.5 Hand Contour Pixels Capturing 2.3 Fingertips Calculation There are two kinds of fingertips to be detected: stretched fingertips and bend fingertips. Stretched fingertips are calculated by K-curvature algorithm based on hand contour while bend fingertips are calculated by K-means Clustering Algorithm. A. Stretched Fingertips Calculation Let P(n) be the set of hand contour pixels, for each pixel P(i) in the P(n), K-curvature algorithm determines the angle between the two lines that start at P(i) and end k points away in either direction P(i - k), P(i + k). Let V1 be the vector from P(i - k) to P(i) and V2 be the vector from P(i) to P(i + k), so the angle can be calculated as: If the angle α is less than a constant value, than P(i) can be determined as convex hull. However, there’re two kinds of convex hull: fingertip and groove point. Let P(u) be the middle point of P(i - k) and P(i + k), S be the center point of hand, d1 be the distance from P(u) to S and d2 be the distance from P(i) to S. Fingertip and groove point can be distinguished as follows: 1) If d1 > d2, P(i) is groove point; (Fig.6) 2) If d1 < d2, P(i) is fingertip; (Fig.7) Fig.6 Groove Point of Hand Fig.8 Result of stretched fingertips detection in real time B. Bend Fingertips Calculation On the step of hand region classification, the bend hand pixels have already been classified as shown in Fig.4. The K-means clustering algorithm is applied to partition all the bend finger pixels to each finger. The k-means method aims to minimize the sum of squared distances between all points and the cluster center [2], and partition n observations into k clusters, C1, C2, ..., Ck. The value k depends on the results of stretched fingertips calculation. For each hand, if the number of stretched fingertips we captured is s, the value of k here would be set as 5 ? s. After the pixels of each finger are classified, the fingertip can be located by finding the point which maximize the distance to the center of the hand region. 3. Hand Motion Prediction Hand motion detection is not enough for nuclear security, find the sign and derive future hand motion is necessary. For this purpose, Auto-Associate Neural Network (AANN) will be used. AANN is basically a neural network whose input and target vectors are the same [3] (see Fig.9). AANN can be used for nonlinear principle component analysis. It consists of an input layer, 3 hidden layers, and an output layer. The first of the hidden layers is the mapping layer with dimension greater than the number of input vectors. The second hidden later is called the bottleneck layer. The dimension of this layer is required to be the smallest in the network. The third hidden layer is called the de-mapping layer and has the same dimension as the mapping layer. - 129 - Fig.7 Fingertip of Hand The stretched fingertips detection system was implemented in real time and the results were very encouraging as shown in Fig.8. prediction by using AANN. For future work, the real time system in detecting both stretched and bend fingertips will be developed. Moreover, future malicious behavior prediction by AANN will be implemented. Reference [1] Y. Li, “Hand gesture recognition using Kinect”, Proc. Int. Conf. Software Eng. Service Sci., 2012, pp. 196?199. [2] Ray S, Turi RH, “Determination of number of clusters in K-means clustering and application in colour image segmentation”, Proceedings of the 4th international conference on advances in pattern recognition and digital techniques (ICAPRDT‘99), Calcutta, India, pp 137?143. [3] M. Kramer, “Nonlinear principal components analysis Fig.9 Auto-Associate Neural Network using auto associative neural networks”, AIChe J., 37(2) : 233-242,1991. For training, raw time-series data of fingertip positions will be used as input data, and the principle components will be output. [4] J. Baek and S. Cho, “Bankruptcy Prediction for credit risk Finally, the trained AANN is used as a hand motion predictor. using an auto associative neural networks in Korean firms”, 4. Conclusion and Future Work in: IEEE International Conference on Computational Intelligence for Financial Engineering, Hong-Kong, 2003. Hand motion is significant to detect and predict malicious behavior for nuclear security. Fingertip positions are important feature for hand motion. In this research, we developed a real time system to detect stretched fingertip position and proposed a new algorithm to detect bend fingertips by using Kinect v2. Also, we explored the possibility of future malicious behavior - 130 -“ “Sign Detection of Future Malicious Behavior for Nuclear Security “ “,Shi CHEN,,Kazuyuki DEMACHI,,Tomoyuki FUJITA,,Yutaro NAKASHIMA,,Yusuke KAWASAKI
After Fukushima Daiichi accident, the importance of nuclear security is increased, especially sabotage by insider should be considered. In response to the increasing threats to Nuclear Power Plant, human malicious behavior detection is necessary for nuclear security. Hand motion has high contribution to human activity thus it’s significant in the detection of inside malicious behavior. For this purpose, Kinect v2 is proposed to be used. Fig.1 Microsoft Kinect v2
Kinect v2 is the new version of game controller technology introduced by Microsoft. It is composed of two cameras, namely a RGB and an infrared (IR) camera (as shown in Fig.1), The RGB camera captures color information with a resolution of 1920x1080 pixels, while the IR camera is used for the real-time acquisition of depth frame with a 512x424 pixels resolution. Also, Kinect v2 provide a new function called body index frame which can be used to segment human body from background.
However, there is no hand specific data available for gesture recognition, although it does include information of the joints between hands and arms. [1] Time series data analysis is a useful method in abnormal behavior detection. In this case, we capture hand motion time series and distinguish normal and abnormal behavior. However, abnormal behavior detection isn’t enough for nuclear security, sign detection of future malicious behavior should be considered. In this research, we tracked hand motion by using Kinect v2. The hand region is segmented and distinguished by depth data and body index frame and developed a real time system in capturing fingertip positions. Moreover, to derive future hand motion, AANN (Auto-Associate Neural Network) will be used. Hand motion are detected by capturing the fingertips positions. Most previous fingertip tracking algorithms usually can track the fingertips for only stretched fingers and the fingertips of bend fingers are usually difficult to be detected. In this research, stretched fingers and bend fingers of both left and right hands were classified as different parts and not only the fingertips of stretched finger but also bend fingers can be detected. The hand motion system of this research consists of three main parts: Hand Region Classification, Hand Contour Pixels Capturing, Fingertips Calculation. We developed our system using Visual Studio 2015 with C# as programming language and body index data, depth data and skeleton data of Kinect v2 were used to calculate fingertip positions.
2.1 Hand Region Classification
At first, body regions should be segmented from the background.
2. Hand Motion Detection For this purpose, Kinect body index frame is used. Body index frame is the new function provided by Kinect v2, which has 512 ×424 resolution frame and at most six body can be detected. For each pixel in body index frame, Kinect system set an index: 1) If the pixel belongs to the region of human body, the index is 0~5; 2) If the pixel belongs to the background, the index is 255. By using body index frame, human body can be segmented from background. After body segmentation, we have to do hand region classification. For each hand, stretched hand region and bend hand region should be divided. As shown in Fig.2, the distance from each hand region to Kinect camera are different, in this case, depth data can be used for hand region classification. Let D(n) be the set of pixels’ depth in Kinect depth frame, I(n) be the set of pixels’ body index in Kinect body index frame, dmax be the wrist point which is in the farther distance to the Kinect Camera, dmin be the wrist point which is in the closer distance to the Kinect Camera and d is the threshold to calculate bend hand pixels. For each pixel’s depth D(i) in D(n): 1) If D(i) ? dmax and D(i) > dmax ? d, meanwhile, I(i) ≠ 255, then I(i) can be classified as the farther distance stretched hand pixels; 2) If D(i) ? dmax ? d and D(i) > dmin, meanwhile, I(i) ≠ 255, then I(i) can be classified as the farther distance bend hand pixels; 3) If D(i) ? dmin and D(i) > dmin ? d, meanwhile, I(i) ≠ 255, then I(i) can be classified as the closer distance stretched hand pixels; 4) If D(i) ? dmin ? d, and D(i) ≠ 0, meanwhile, I(i) ≠ 255, then I(i) can be classified as the closer distance bend hand pixels; Finally, pixels can be divided into four parts: ● Left stretched hand pixels; ● Right stretched hand pixels; ● Left bend hand pixels; ● Right bend hand pixels. Fig.4 Results of hand region classification (red pixels: stretched hand region; yellow pixels: bend hand region; black pixels: background) 2.2 Hand Contour Pixels Capturing Hand contour can be used to calculate the stretched fingertips. For each pixel, it has four neighbor pixels: left neighbor pixels, right neighbor pixel, above neighbor pixel and below neighbor pixel. And if a pixel is stretched hand pixels and more than one of its neighbor pixels are background pixel, it can be classified as hand contour pixels. (as shown in Fig.5) Fig.2 Hand region classification by depth To reduce computing complexity, hand pixels should be limited in some specific region. For this purpose, we use “rectangle limitation algorithm”. The algorithm can be summarized as follow: 1) Get center points of both left and right hand; 2) Build four rectangles from center point with width W and height H for each hands; (The values of W and H depends on the distance from body to the Kinect camera) 3) Only the pixels within this region can be classified as hand pixels; (As shown in Fig.3) Fig.3 Rectangle limitation algorithm used for reducing computing complexity The classification results are shown in Fig.4. - 128 - Fig.5 Hand Contour Pixels Capturing 2.3 Fingertips Calculation There are two kinds of fingertips to be detected: stretched fingertips and bend fingertips. Stretched fingertips are calculated by K-curvature algorithm based on hand contour while bend fingertips are calculated by K-means Clustering Algorithm. A. Stretched Fingertips Calculation Let P(n) be the set of hand contour pixels, for each pixel P(i) in the P(n), K-curvature algorithm determines the angle between the two lines that start at P(i) and end k points away in either direction P(i - k), P(i + k). Let V1 be the vector from P(i - k) to P(i) and V2 be the vector from P(i) to P(i + k), so the angle can be calculated as: If the angle α is less than a constant value, than P(i) can be determined as convex hull. However, there’re two kinds of convex hull: fingertip and groove point. Let P(u) be the middle point of P(i - k) and P(i + k), S be the center point of hand, d1 be the distance from P(u) to S and d2 be the distance from P(i) to S. Fingertip and groove point can be distinguished as follows: 1) If d1 > d2, P(i) is groove point; (Fig.6) 2) If d1 < d2, P(i) is fingertip; (Fig.7) Fig.6 Groove Point of Hand Fig.8 Result of stretched fingertips detection in real time B. Bend Fingertips Calculation On the step of hand region classification, the bend hand pixels have already been classified as shown in Fig.4. The K-means clustering algorithm is applied to partition all the bend finger pixels to each finger. The k-means method aims to minimize the sum of squared distances between all points and the cluster center [2], and partition n observations into k clusters, C1, C2, ..., Ck. The value k depends on the results of stretched fingertips calculation. For each hand, if the number of stretched fingertips we captured is s, the value of k here would be set as 5 ? s. After the pixels of each finger are classified, the fingertip can be located by finding the point which maximize the distance to the center of the hand region. 3. Hand Motion Prediction Hand motion detection is not enough for nuclear security, find the sign and derive future hand motion is necessary. For this purpose, Auto-Associate Neural Network (AANN) will be used. AANN is basically a neural network whose input and target vectors are the same [3] (see Fig.9). AANN can be used for nonlinear principle component analysis. It consists of an input layer, 3 hidden layers, and an output layer. The first of the hidden layers is the mapping layer with dimension greater than the number of input vectors. The second hidden later is called the bottleneck layer. The dimension of this layer is required to be the smallest in the network. The third hidden layer is called the de-mapping layer and has the same dimension as the mapping layer. - 129 - Fig.7 Fingertip of Hand The stretched fingertips detection system was implemented in real time and the results were very encouraging as shown in Fig.8. prediction by using AANN. For future work, the real time system in detecting both stretched and bend fingertips will be developed. Moreover, future malicious behavior prediction by AANN will be implemented. Reference [1] Y. Li, “Hand gesture recognition using Kinect”, Proc. Int. Conf. Software Eng. Service Sci., 2012, pp. 196?199. [2] Ray S, Turi RH, “Determination of number of clusters in K-means clustering and application in colour image segmentation”, Proceedings of the 4th international conference on advances in pattern recognition and digital techniques (ICAPRDT‘99), Calcutta, India, pp 137?143. [3] M. Kramer, “Nonlinear principal components analysis Fig.9 Auto-Associate Neural Network using auto associative neural networks”, AIChe J., 37(2) : 233-242,1991. For training, raw time-series data of fingertip positions will be used as input data, and the principle components will be output. [4] J. Baek and S. Cho, “Bankruptcy Prediction for credit risk Finally, the trained AANN is used as a hand motion predictor. using an auto associative neural networks in Korean firms”, 4. Conclusion and Future Work in: IEEE International Conference on Computational Intelligence for Financial Engineering, Hong-Kong, 2003. Hand motion is significant to detect and predict malicious behavior for nuclear security. Fingertip positions are important feature for hand motion. In this research, we developed a real time system to detect stretched fingertip position and proposed a new algorithm to detect bend fingertips by using Kinect v2. Also, we explored the possibility of future malicious behavior - 130 -“ “Sign Detection of Future Malicious Behavior for Nuclear Security “ “,Shi CHEN,,Kazuyuki DEMACHI,,Tomoyuki FUJITA,,Yutaro NAKASHIMA,,Yusuke KAWASAKI