3D interior modeling using Kinect v2

After having the all powerful camera Kinect, there are just so much thing that we can explore. The first project I done is the 3D interior modeling for my laboratory.

In this project, the Kinect is going to be able to track an object moving in the room and tell the location of that object(x and y coordinate).

To be able to get it done, we need a Kinect, a windows environment with visual studio and Kinect SDK.

Since this article is only for demo purpose of Kinect, I won't get on much detail onto the tracking part but if you do want it you can either contact me or there are a whole lot of information online about it using openCV.

To be able to find the exact location of an object we can do a simple math. As the picture shown below(apologize for my poor drawing), A is the location where you mount your Kinect, the green dot is the object that you want to locate, B is the distance from the Kinect to the object, C is the height that you measured from ground to Kinect, and F is the distance you find from the Kinect to a point that shares the same x coordinate value as the object(D is perpendicular to the wall and right below A).

So after you find all those values above, you can easily calculate the X and Y coordinate values of the object. However, the problem now become how to find the distance value B and F(you measure C with rule and calculate D).

Here comes the powerful Kinect, it has not only the RGB camera but also embedded an inferred sensor that gives you the distance information. All you need to do is read them from the Kinect depth frame source.

First you init your Kinect sensor, color frame reader and depth frame reader and those are the interface you need to get access to each of the frame. And after that, use the AcquireLatestFrame function to get the latest frame and use CopFrameDataToArray function to copy the data you acquire into an array. You can do the same thing fro both color frame and the depth frame.

The data we collected there is a 2D array full of information of distances. Each pixel contains 16 bit of information telling you the distance from the Kinect to this pixel. Now we just need to find which pixel in this array has our object and the connecting point of D and E.

Notice here I find is a big challenge that we are going to face. Since our tracking program is simply returning the x, y coordinate value indicating the object is shown in which pixel in the color frame, we have to convert that into the depth frame. Luckily Kinect offered us a way to do that, CoordinateMapper->MapColorFrameToDepthSpace.

So here is a part of my program showing how to acquire the depthFrame and how to map it into the color frame:

if (SUCCEEDED(hr)){

hr = pDepthFrame->AccessUnderlyingBuffer(&nBufferSize_depth, &pBuffer_depth);

pDepthFrame->CopyFrameDataToArray(424 * 512, pDepthPoints);

DepthSpacePoint* m_pDepthCoordinates = new DepthSpacePoint[1920 * 1080];

m_pKinectSensor->get_CoordinateMapper(&m_pCoordinateMapper);

m_pCoordinateMapper->MapColorFrameToDepthSpace(512 * 424, (UINT16*)pDepthPoints, 1920 * 1080, m_pDepthCoordinates);

DepthSpacePoint& rPoint = m_pDepthCoordinates[1920*origin.y+origin.x];

if ((rPoint.X >= 0) && (rPoint.Y) >= 0) {

int depth = pDepthPoints[(int)((512 * rPoint.Y + rPoint.X))];

cout << "depth = " << depth << endl;

}

depthImg_show = ConvertMat(pBuffer_depth, depth_width, depth_height, 0, 9999);

}

A couple thing that you might want to pay attention to:

1. the size of RGB fram is 1920* 1080 but the size of depth frame is only 512* 424.

2. Kinect is not going to give you successful frame every time, so keep reading and refreshing. Also in the depth frame, there are possibility that some pixel has null value, so handle it(distance can be out of bound).

3. I used a windows onMouse function to let you click on the frame and return you the distance between the Kinect and that pixel, that's where the rPoint comes from.

4. Remember to release your frame after you are done, you don't want to crash you program and your memory may leak.

If you have any questions or you want to know more about this project, feel free to contact me.