computer_vision_kinect_t2
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revision | |||
computer_vision_kinect_t2 [2016/07/14 14:35] – joaomatos | computer_vision_kinect_t2 [2016/07/14 15:13] (current) – joaomatos | ||
---|---|---|---|
Line 7: | Line 7: | ||
===== Introduction : How the Kinect works ===== | ===== Introduction : How the Kinect works ===== | ||
+ | {{ :: | ||
In simple words , the Infrared projector sends infrared lights to the room , if you cold see these lights you would see a lot of dots projected in all your Kinect field of view. The Infrared camera analyze the pattern of the dots project into all the surfaces that intercepts the Infrared ray and by the Kinect software processing the Kinect can tell how the surface is ( if it is a flat surface , if it has curves , etc..) and also it can say how far is the projected dot from the camera (That' | In simple words , the Infrared projector sends infrared lights to the room , if you cold see these lights you would see a lot of dots projected in all your Kinect field of view. The Infrared camera analyze the pattern of the dots project into all the surfaces that intercepts the Infrared ray and by the Kinect software processing the Kinect can tell how the surface is ( if it is a flat surface , if it has curves , etc..) and also it can say how far is the projected dot from the camera (That' | ||
Line 12: | Line 13: | ||
The Depth stream obtained by the OpenNI library from the Kinect has stored the depth value of each pixel. So if you know in what pixel the object that you want to know the distance is (By analyzing the Colored Stream for example , or By running your object detection algorithm first ) , you only have to see the value of this same pixel on the Depth image and you get the distance (after doing some calculations). | The Depth stream obtained by the OpenNI library from the Kinect has stored the depth value of each pixel. So if you know in what pixel the object that you want to know the distance is (By analyzing the Colored Stream for example , or By running your object detection algorithm first ) , you only have to see the value of this same pixel on the Depth image and you get the distance (after doing some calculations). | ||
- | | + | The problem is that the RGB camera and IR camera are not on the same place , so the object located in the (x,y) pixel on the Depth Stream is not necessarily the same object located in the same (x,y) pixel of the Colored Stream. Fortunately we can use the OpenNI library to convert pixels from the Depth stream to the Colored Streams ( The values will be pretty close , so if your object is more than approximately 10x10 pixels , you can measure the distance of this object to the kinect using the pixel that is in the center of the object on the colored stream , and see the depth value of this pixel on the Depth Stream ( because the deviation is small and you will continue to be over the object surface on the Colored Stream ). For small object , you might need to use the Coordinate conversor. |
- | + | ||
- | For the OpenNI2 for example , you don't need to do any conversion , because you can start the device (kinect) with the command " | + | |
| | ||
+ | For the OpenNI2 for example , you don't need to do any conversion , because you can start the device (kinect) with the command " | ||
| | ||
+ | One thing that you have to pay attention is how to get the data from the Depth Stream. | ||
+ | |||
+ | You can get the depth data using this: (it will create the depth array, single row , with all the depth values) | ||
+ | |||
+ | < | ||
+ | short* depth = (short*)depthFrame.getData(); | ||
+ | </ | ||
| | ||
+ | The depth data is stored into an one row array (so trying to access the depth of the (x,y) pixel will not work) . The pixel that you want to measure (you have the x,y location in the depth/ | ||
+ | < | ||
+ | index = ( PixelCoordinateY*DepthStreamResolutionX ) + PixelCoordinateX | ||
+ | </ | ||
+ | Finally you get the depth value of your object using: | ||
+ | |||
+ | < | ||
+ | short* depth = (short*)depthFrame.getData(); | ||
+ | depthObject = depth[index]; | ||
+ | </ | ||
---- | ---- | ||
- | ===== Using OpenNI to get the depth view ===== | + | ===== Using OpenNI to get the depth image ===== |
+ | |||
+ | **Initializing :** Lets try to open any Kinect connected to the PC , if you did't follow the installation instructions right you will not be able to start the Code and you will get an error message. | ||
+ | |||
+ | < | ||
+ | openni:: | ||
+ | openni:: | ||
+ | auto ret = device.open(openni:: | ||
+ | if (ret != openni:: | ||
+ | throw std:: | ||
+ | } | ||
+ | |||
+ | </ | ||
+ | |||
+ | **Starting the Streams that you want:** (on this case Colored and Depth). We will initialize the " | ||
+ | |||
+ | < | ||
+ | openni:: | ||
+ | depthStream.create(device, | ||
+ | depthStream.start(); | ||
+ | |||
+ | openni:: | ||
+ | colorStream.create(device, | ||
+ | colorStream.start(); | ||
+ | |||
+ | device.setDepthColorSyncEnabled(true); | ||
+ | device.setImageRegistrationMode(openni:: | ||
+ | </ | ||
+ | |||
+ | **Useful Variables: | ||
+ | < | ||
+ | cv::Mat depthImage; | ||
+ | cv::Mat colorImage; | ||
+ | </ | ||
+ | |||
+ | Getting the frames: Now we introduce the infinite loop to keep getting frames from both streams . We need to create variables to store this frames , you do it with openni:: | ||
+ | |||
+ | < | ||
+ | while (1) { | ||
+ | |||
+ | |||
+ | // Create frames from depth and colored streams. | ||
+ | openni:: | ||
+ | depthStream.readFrame(& | ||
+ | |||
+ | openni:: | ||
+ | colorStream.readFrame(& | ||
+ | |||
+ | </ | ||
+ | |||
+ | |||
+ | **Storing the frames in the Mat variable:** Now we want to store these frames into " | ||
+ | |||
+ | < | ||
+ | |||
+ | //CREATING THE COLORED IMAGE | ||
+ | const openni:: | ||
+ | ColorFrameBGR.create(initialcolorFrame.getHeight(), | ||
+ | memcpy(ColorFrameBGR.data, | ||
+ | |||
+ | cv:: | ||
+ | |||
+ | //CREATING THE DEPTH IMAGE | ||
+ | |||
+ | depthImage = cv:: | ||
+ | CV_16U, | ||
+ | |||
+ | </ | ||
+ | |||
+ | Now we have two images , " | ||
+ | |||
+ | | ||
+ | |||
+ | ---- | ||
+ | |||
+ | ===== Convert Depth coordinate to World coordinate ===== | ||
+ | |||
+ | There are two ways to get the World Coordinate from the depth image. First we can use an OpenNI function and get it straight , or we can use some algebra , and knowing the Kinect camera parameters we can find where the object is in relation with the camera referential . | ||
+ | | ||
+ | |||
+ | **The OpenNI function is:** | ||
+ | |||
+ | < | ||
+ | |||
+ | |||
+ | float x = 0; | ||
+ | float y = 0; | ||
+ | float z = 0; | ||
+ | |||
+ | openni:: | ||
+ | 200, | ||
+ | |||
+ | </ | ||
+ | |||
+ | you need to first initialize three variables to store the World coordinates given by the function (x,y,z). The Coordinate Converter will use the depthStream . The third and forth arguments are the pixel location of your object (on this case i want to know the world coordinate of the object located on the pixel (200,200). The fifth argument is the same object pixel but given in Depth Image Pixel (just use depthImage.at< | ||
+ | |||
+ | **Algebra Approach:** You can follow the math presented [[http:// | ||
computer_vision_kinect_t2.txt · Last modified: 2016/07/14 15:13 by joaomatos