Since I will be working on the vision aspect of this Darwin project, I will be engaging OpenCV to process the images received from the camera.

OpenCV are libraries of files that languages like C/Cpp and Python can use as long as their source codes have the link to the directory of OpenCV. Hence, compare to Matlab which is a standalone software that can also process images, OpenCV is more attractive a choice to outsource the image processing to from Darwin because it will take up less computation memory.

I will want to be able to obtain the coordinates of each rung of the ladder which Darwin can use to project its next step in its ascend of the ladder.

I will be using Visual Cpp 2010 Express as my platform for practicing OpenCV.

I refer to here for the installation of the relevant software application and this tutorial, which is related to the previous, to practise and understand some of the codes from the OpenCV library.

There were some error with using OpenCV2.1 and OpenCV2.2 in this case.

• Here states that OpenCV2.1 do not work with VCpp 2010.
• Here states that there is a bug in OpenCV2.2 that is preventing webcams from working with this version of OpenCV. However, mine still does not work after fixing this bug.

I have changed to use OpenCV2.4.3 instead. I followed the tutorial here to install it properly with VCpp 2010. And I got the program code to run!

NOTE: The OpenCV functions used in this sample code are for C programming, which is not really applicable to my project on Darwin as we will be mainly using Cpp. For Cpp, OpenCV uses a different API.

I have also prepared a MS Power Point presentation for a step by step installation of OpenCV.

opencv_tutorial.pptx

This is my code to recognize rectangles and output the coordinates of the center of them.

I will explain the main functions that are critical to make this possible. Refer to here for the documentation of the Cpp API of OpenCV.

You may need to go here for a orientation of the OpenCV libraries and built-in functions.

Here are some of the critical parts of the codes:

VideoCapture cap(0); // open the default camera
if(!cap.isOpened())  // check if we succeeded
    return -1;
...
while(1){
cap >> original;  //constantly obtain new frame from camera
 
Mat original, pyr, timg, gray;//declare Mat objects that can be used to store image datas
...
// down-scale and upscale the image to filter out the noise
pyrDown(original, pyr, Size(original.cols/2, original.rows/2));
pyrUp(pyr, timg, original.size());

Note: pyrDown and pyrUp can operate on both gray(single channel) and colored(3 channels of Blue Green and Red in the standard BGR order used in OpenCV) images.

inRange(timg,// input timg as the image to be processed
	Scalar(0,  0,  0),// min filtering value (if color is greater than or equal to this)
	Scalar(90,90,90),// max filtering value (if color is less than this)
	inrange);//save filtered image to inrange

Note: Resultant filtered image will be a single channel of a matrix of pixels of value of wither 255(white) or 0 (black).

Inside findSquares function:

Canny(image, gray, 50, 200, 5);
//Detect edges on the input "image" passed by reference from the main program into this findSquare function
//save the newly processed to "gray"
//with lower threshold of 50 and higher threshold to 200
//with the aperture size of the kernel at 5

Note*: Input image to Canny function must be a single channel image. It need not be grayscale.

dilate(gray, gray, Mat(), Point(-1,-1));
//take input image, that is the 1st "gray"
//"blur" the image (I am not too sure of the term that should be used)
//save processed image to 2nd "gray"
//using the kernel of default aperture size of 3 as defined by "Mat()"
//at default start point specified by "Point(-1,-1)"
 
vector<vector<Point> > contours;
//declaration of a vector of a vector of X and Y points
//the nested vector of points is the list of coordinates of point that make up ONE contour
//the outer vector of vectors of points is the list of ALL the contours detected
...
findContours(gray, contours, CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE);
//input image is "gray"
//processed image is saved to contours
//in the designated mode

Note*: Input image to findContours function must be a single channel image. It need not be grayscale.

for( size_t i = 0; i < contours.size(); i++ )
{
    approxPolyDP(Mat(contours[i]), approx, arcLength(Mat(contours[i]), true)*0.02, true);
	//for each detected contour, if they approximate to a polygon according to Ramer–Douglas–Peucker algorithm
	//the vertices of X and Y coordinates are stored as a list in the vector "approx"
 
	//if the polygons are squares or rectangles they must be
    if( approx.size() == 4				//4sided
	&& fabs(contourArea(Mat(approx))) > 1000	//big enough (eliminate noise)
	&& isContourConvex(Mat(approx)) )		//convex
    {
        squareCount++;
	//taking count of the number of rectangles detected
 
	double maxCosine = 0;			
        for( int j = 2; j < 5; j++ )
        {
            double cosine = fabs(angle(approx[j%4], approx[j-2], approx[j-1]));
			//store cosine of the angle between 3 adjacent vertices into the variable "cosine"
            maxCosine = MAX(maxCosine, cosine);
        }
 
        if( maxCosine < 0.3 )
		squares.push_back(approx);
		//if the angles are above cos^-1(0.3) = 72 degrees,
		//the 4 sided polygon characterized by the current contour is considered a square
		//the "push_back" function stores vector of 4 2D points into the "squares" vector
 
	double sum_x=0, sum_y=0;
	for(int k=0; k<approx.size(); k++)
	{
		sum_x+=approx[k].x;
		sum_y+=approx[k].y;
	}
	cout << "Square #"<< squareCount << "\nX: "<< sum_x/4 << "\t\tY: " << sum_y/4 <<endl;
	//print the coordinates of the center of each detected rectangles
    }
}

Here is a sample run of my code to filter out black rectangles and output the X and Y coordinates of the centers of the squares in the output MS-Prompt window.

Some food for thought:

• Code is supposed to outline black rectangles, but it can also encircle the “negative” areas.
• Two or more rectangles may be identified for the same rectangle, resulting in double counting.
• This is after all done using Windows Visual Cpp which is not applicable on Darwin.
• The speed of computating the codes and data will need to be looked into.

I improved on the previous code and tested it out on an improvised ladder.

Here is my code and here is a video of it.

Video explanation:

• I created a program that creates trackbars for me to use and adjust the colour to filter out easily.
• I shined a light to test the effect of different light intensity on the colour filtered out.
• Initially the ladder was right in front of the back ground. I shifted the ladder away from the background such that there was some space between them.
• During the actual detection of the ladder rungs, it was not so effective and I attributed it to the sides of the ladder. They form groove edges which disfigure the rectangular shape of the spaces in between the ladder rungs which were to be detected.
• I cheated and covered the sides so that smooth rectangles were formed. Detection of the spaces in between the ladder rungs improved.
• I went on to detect the actual ladder rungs rather than the spaces in between them.
• However, when I removed the covers that were covering the sides of the ladder, non of the rungs can be detected.

In conclusion, the ladder needs to be modified to allow easier detection of the ladder rungs. This can be done by:

• Making a new ladder that has smooth sides so that the corners of each possibly detectable rectangular shape of the spaces in between the runs or the rungs themselves form a smooth rectangle.
• Colour the rungs a different colour and leave the sides untouched. Filter out the colour of the rungs only
• Create a region of interest in the centre of the camera. The detection algorithm will carry out these central pixels only instead of the whole frame.

This algorithm depends on contour points (white circles in the picture at the top right hand corner of the insert below) detected to plot rectangles.

However, as can be seen circled in red, when there is a smooth straight line, the points do not appear, and hence on the ladder rungs which has pretty smooth horizontal edge features, points are hard to come by and a rectangle is hard to detect.

This means that one of the adjustments I made during my previous trial run, which was intended to smoothen the sides of the spaces in between ladder rungs by covering up the sides, should make the ladder rungs more difficult to be detected. However the opposite occur.

Why?

I made the adjustments again and found out that I did not use a cover with a smooth vertical side to cover up the sides.

Therefore, as can be seen below, there were more points detected along the vertical side of the covers, hence improving chance of a rectangle being detected.

(I made the left cover slanted to achieve more points along vertical and prove my point)

When the ladder is tilted. the detection of rectangles improved greatly (detecting the ladder rungs instead of the spaces in between ladder rungs now by using black colored covers).

This is so as along the ladder rungs, contour points are more easily detected as the rung edges are not horizontal to the camera width.

Below is a picture of it.

This algorithm which uses contour points and the approxPolyDP() function is not effective to detect lines which are perfectly vertical or horizontal to the camera, like the ladder rungs.

It is thus not effective to detect the ladder rungs.

I found out the best way is to perform canny edge detector on a grayscale image, and then the findContours() function, rather than filtering out the colour of interest to work on.

...
...
cvtColor(original,convert,CV_RGB2GRAY);
Canny(convert, canny, 500, 800, 5);
findContours(canny, contours, CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE);
...
...

Here are some pictures.