DASL Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
===== Tutorial 6 =====
 
 On this tutorial you will learn how to use the Threshold together with tracking algorithm to track an object by its contour. The object will be defined as the biggest contour found.The idea is to mix Tutorial 4 ( Detecting edges and contours ) and Tutorial 5 ( Threshold ). 

 I recommend you to type the code on your own to get familiarized with the program language. If you have trouble , the original code is attached bellow ( Running on Visual Studio 2015 + OpenCV 3.1 ) * Check the installation guide to make sure that you linked all the OpenCV modules to your Visual Studio.

{{::edge_and_contour_webcam.rar|}}

----

===== Tracking an object by its contour =====
{{ ::edge_and_contour_webcam.jpg?direct |}}

<Code:c++ linenums:1>
//Initializing the modules that will be used
#include <sstream>
#include <iostream>
#include <opencv/highgui.h>
#include <opencv/cv.h>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>


#define _CRT_SECURE_NO_WARNINGS

//defining the namespace used ( no need to use cv:: or std:: before the API's)
using namespace cv;
using namespace std;

//Values to be used on the HSV trackbar
//these will be changed using trackbars
int iLowH = 0;
int iHighH = 179;

int iLowS = 0;
int iHighS = 255;

int iLowV = 0;
int iHighV = 255;

//default capture width and height
const int FRAME_WIDTH = 640;
const int FRAME_HEIGHT = 480;

//Function to create a window with the Trackbars to apply the Threshold.

void createTrackbars() {

	//Open the window to display the Trackbars
	namedWindow("Trackbars", CV_WINDOW_AUTOSIZE);

	//Hue values (0 - 179)
	cvCreateTrackbar("LowH", "Trackbars", &iLowH, 179);
	cvCreateTrackbar("HighH", "Trackbars", &iHighH, 179);

	//Saturation Values (0-255)
	cvCreateTrackbar("LowS", "Trackbars", &iLowS, 255);
	cvCreateTrackbar("HighS", "Trackbars", &iHighS, 255);

	//Value (0-255)
	cvCreateTrackbar("LowV", "Trackbars", &iLowV, 255);
	cvCreateTrackbar("HighV", "Trackbars", &iHighV, 255);

}


//Function to apply the erode and dilate features.
void DilateAndErode(Mat &image) {

	//Defining the erode and dilate properties
	//the erode element chosen here is a 3x3 piexels rectangle.
	//Change the Size argument to optimize your threshold. 
	//dilate with 8x8 size element to make the threshold object more visible

	Mat erodeElement = getStructuringElement(MORPH_RECT, Size(3, 3));
	Mat dilateElement = getStructuringElement(MORPH_RECT, Size(8, 8));

	//Apply erode and dilate
	erode(image, image, erodeElement);
	dilate(image, image, dilateElement);


}





int main(int argc, char *argv[])
{


	//Variables to store the opened image , a gray version , edge and cedge
	Mat coloredimage;
	Mat edge;
	Mat HSV;
	Mat threshold;


	//Edge detection Threshold 
	int edgeThreshold = 10;


	//auxiliar variables to control the actions of the program
	char key = 0;
	bool useFeatures = true;



	//Call the function to create the window with the trackbars
	createTrackbars();
	//Initialize the webcam capture
	VideoCapture capture;
	capture.open(0);

	//set height and width of capture frame
	capture.set(CV_CAP_PROP_FRAME_WIDTH, FRAME_WIDTH);
	capture.set(CV_CAP_PROP_FRAME_HEIGHT, FRAME_HEIGHT);

	//Until the User press 'q' the loop will run
	//Get the image from the webcam -> convert to HSV -> Threshold the image
	//using the HSV max and min set on the Trackbar window.
	//The threshold image will be used into the findcontours function to find the contours.
	//The largest contour will be draw on the webcam frame.
	while (key != 'q')
	{


		//Get the image from the webcam
		capture >> coloredimage;
		//Convert the frame from BGR (RGB) to HSV
		cvtColor(coloredimage, HSV, COLOR_BGR2HSV);


		//filter the HSV image using the minimum and maximum values set on the 
		//Trackbars window using the inRange function.		
		inRange(HSV, Scalar(iLowH, iLowS, iLowV), Scalar(iHighH, iHighS, iHighV), threshold);

		if (key == 'm')
		{
			useFeatures = !useFeatures;
		}

		//If the key M is pressed the dilate and erode features will be activated.
		if (useFeatures)
		{
			DilateAndErode(threshold);
		}

		//Show Threshold image
		imshow("Threshold", threshold);

		//Canny is a openCV function  (Canny Algorithm to edge detection )
		//Canny(source image , output image , threshold 1 , threshold 2 (usually 3 times threshold1) , kernel size )
		Canny(threshold, edge, edgeThreshold, edgeThreshold * 3, 3);


		//Initializing two vectors to be used on the findContours function
		vector<vector<Point> > contours;
		vector<Vec4i> hierarchy;

		//The function findContours  Use the output of the Canny function as input

		findContours(edge, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE);

		
		//Find the largest contour to be draw on the image
		int largest_area = 0;
		int largest_contour_index = 0;

		for (int i = 0; i < contours.size(); i++)
		{
			double a = contourArea(contours[i], false);  //  Find the area of contour
			if (a > largest_area)
			{
				largest_area = a;
				largest_contour_index = i;                //Store the index of largest contour
			}
		}


		//draw the largest contour on the image		
		drawContours(coloredimage, contours, largest_contour_index, Scalar(0, 0, 255), 2, 8, hierarchy, 0, Point());


		// Show image with contours in a new window.	
		imshow("Find Contours", coloredimage);

		key = waitKey(25);

	}



	return 0;


}
</Code>

----

===== Understanding the code =====

<Code:c++ linenums:17>
//Values to be used on the HSV trackbar
//these will be changed using trackbars
int iLowH = 0;
int iHighH = 179;
 
int iLowS = 0;
int iHighS = 255;
 
int iLowV = 0;
int iHighV = 255;
 
//default capture width and height
const int FRAME_WIDTH = 640;
const int FRAME_HEIGHT = 480;
 
//Function to create a window with the Trackbars to apply the Threshold.
 
void createTrackbars() {
 
	//Open the window to display the Trackbars
	namedWindow("Trackbars", CV_WINDOW_AUTOSIZE);
 
	//Hue values (0 - 179)
	cvCreateTrackbar("LowH", "Trackbars", &iLowH, 179);
	cvCreateTrackbar("HighH", "Trackbars", &iHighH, 179);
 
	//Saturation Values (0-255)
	cvCreateTrackbar("LowS", "Trackbars", &iLowS, 255);
	cvCreateTrackbar("HighS", "Trackbars", &iHighS, 255);
 
	//Value (0-255)
	cvCreateTrackbar("LowV", "Trackbars", &iLowV, 255);
	cvCreateTrackbar("HighV", "Trackbars", &iHighV, 255);
 
}

</Code>

 The code begins with the same idea of image threshold. Declaring the variables to be used on the Trackbars windows and defining the function to create the slider bars. You can see the commentaries  [[http://www.daslhub.org/unlv/wiki/doku.php?id=opencv_tutorials_t5|here]]


----

<Code:c++ linenums:32>
//Function to apply the erode and dilate features.
void DilateAndErode(Mat &image) {
 
	//Defining the erode and dilate properties
	//the erode element chosen here is a 3x3 piexels rectangle.
	//Change the Size argument to optimize your threshold. 
	//dilate with 8x8 size element to make the threshold object more visible
 
	Mat erodeElement = getStructuringElement(MORPH_RECT, Size(3, 3));
	Mat dilateElement = getStructuringElement(MORPH_RECT, Size(8, 8));
 
	//Apply erode and dilate
	erode(image, image, erodeElement);
	dilate(image, image, dilateElement);
 
 
}
</Code>

 Again the same idea of the image threshold code. To track an object first we need to obtain a binary image with only the object visible - and the morphological transform helps to optimaze the visibility of the thresholded object in the binary image by filtering the noise and expanding the object contour.After obtaining the binary image , we can apply the edge and contour detector and we can track the object by its contour - because the object contour will be the only contour shown in the binary image.


----

<Code:c++ linenums:76>
int main(int argc, char *argv[])
{
 
 
	//Variables to store the opened image , a gray version , edge and cedge
	Mat coloredimage;
        Mat edge;
	Mat HSV;
	Mat threshold;
 
 
	//Edge detection Threshold 
	int edgeThreshold = 10;
 
 
	//auxiliar variables to control the actions of the program
	char key = 0;
	bool useFeatures = true;

</Code>

 Our main program start with the variables declaration.The coloredimage variable will store the web camera frame , the edge will store the output from **"Canny"** edge detector function , the HSV will store the HSV conversion from RGB done by the **"cvtColor"** function and the threshold will store the image output after the threshold done by the **"inRange"** function.

 Also we declare the key variable to control the loop ( to end the program when 'q' is pressed) and a Boolean to control the on/off of the morphological transformations dilate and erode.


----

<Code:c++ linenums:98>
//Call the function to create the window with the trackbars
	createTrackbars();
	//Initialize the webcam capture
	VideoCapture capture;
	capture.open(0);
 
	//set height and width of capture frame
	capture.set(CV_CAP_PROP_FRAME_WIDTH, FRAME_WIDTH);
	capture.set(CV_CAP_PROP_FRAME_HEIGHT, FRAME_HEIGHT);

</Code>

 We call the **"createTrackbars()"** function to create a window with the slider bars to be used to threshold the image ( adjust the H S and V values to make the image all black only with the object silhouette in white ).

 The web camera video is initialize using the **"VideoCapture**" function and the variable that will store the web camera video is 'capture' , opened in **"capture.open(0)"**.  We set the capture resolution as 640x480 pixels using the **"capture.set"** function. 


----

<Code:c++ linenums:108>
//Until the User press 'q' the loop will run
	//Get the image from the webcam -> convert to HSV -> Threshold the image
	//using the HSV max and min set on the Trackbar window.
	//The threshold image will be used into the findcontours function to find the contours.
	//The largest contour will be draw on the webcam frame.
	while (key != 'q')
	{
 
 
		//Get the image from the webcam
		capture >> coloredimage;
		//Convert the frame from BGR (RGB) to HSV
		cvtColor(coloredimage, HSV, COLOR_BGR2HSV);

</Code>

 The main loop of the program will stop when 'q' is pressed on the keyboard. We first need to obtain the colored frame that will be thresholded - this frame is obtained by **"capture >> "** and is stored on the 'coloredimage' variable. 

 Using the **"cvtColor"** function we convert from BGR to HSV using the code **"COLOR_BGR2HSV"** on the last argument. The output will be stored on the second argument which is the 'HSV' variable.


----

<Code:c++ linenums:122>
//filter the HSV image using the minimum and maximum values set on the 
		//Trackbars window using the inRange function.		
		inRange(HSV, Scalar(iLowH, iLowS, iLowV), Scalar(iHighH, iHighS, iHighV), threshold);
 
		if (key == 'm')
		{
			useFeatures = !useFeatures;
		}
 
		//If the key M is pressed the dilate and erode features will be activated.
		if (useFeatures)
		{
			DilateAndErode(threshold);
		}
 
		//Show Threshold image
		imshow("Threshold", threshold);

</Code>

 Now the threshold is done by the **"inRange"** function . The minimum and maximum values are set by hand sliding the bars on the Trackbar window . After the threshold will be generated a binary image that will be stored on the last argument which is the 'threshold' variable. 

 The 'm' key on the keyboard can be pressed to toggle on/off the morphological transformations dilate and erode. When 'm' is pressed the key variable will store the 'm' value and inside the first conditional statement , **"useFeatures = !useFeatures"** the useFeatures Boolean will switch from TRUE to FALSE and vice-versa. Whenever the value is TRUE , it will enter in the second conditional statement , which calls the **"DilateAndErode()"** function , using the threshold image as argument. The result from this function will be stored on the same threshold variable ( the image will be replaced by the image after the morphological transformations) and it will be displayed in a new window called "Threshold" ( using the **"imshow"** function ).


----

<Code:c++ linenums:140>
//Canny is a OpenCV function  (Canny Algorithm to edge detection )
		//Canny(source image , output image , threshold 1 , threshold 2 (usually 3 times threshold1) , kernel size )
		Canny(threshold, edge, edgeThreshold, edgeThreshold * 3, 3);
</Code>

 As we used on the Tutorial from [[http://www.daslhub.org/unlv/wiki/doku.php?id=opencv_tutorials_t4|Edge and Contour detection]] , the "Canny" function can be used to find all the edges in a given image. It will use the thresholded image and the output will be saved on the 'edge' variable. This output will be the input for the functions to find the contours.


----

<Code:c++ linenums:145>
//Initializing two vectors to be used on the findContours function
		vector<vector<Point> > contours;
		vector<Vec4i> hierarchy;
 
		//The function findContours  Use the output of the Canny function as input
 
		findContours(edge, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE);

</Code>

 To use the function to find the contours "findContours" we need to declare first a vector that will store all the contours points , and a vector to represent the hierarchy argument - the hierarchy vector contain as many element as the number of contours found (if we know its size , we know how many contours were found) , and contains information about the image topology . For the most basic applications the other arguments will not be changed , if you want to understand more you can read [[http://docs.opencv.org/2.4/modules/imgproc/doc/structural_analysis_and_shape_descriptors.html?highlight=findcontours|Here]].

 The "findContours" function will take the image with the edge detected as input and will give a vector with the contours points stored on the 'contours' variable. Now we need to draw these contours found in our image.


----

<Code:c++ linenums:153>
		//Find the largest contour to be draw on the image
		int largest_area = 0;
		int largest_contour_index = 0;
 
		for (int i = 0; i < contours.size(); i++)
		{
			double a = contourArea(contours[i], false);  //  Find the area of contour
			if (a > largest_area)
			{
				largest_area = a;
				largest_contour_index = i;                //Store the index of largest contour
			}
		}

</Code>

 After applying the **"FindContours"** function we obtained a vector with the points of every contour detected. Even with a good threshold image , this function can detect that are not the object contour - it can separate the object contour into more than one contour (it will find a contour that will be very close to the object contour and small contours next to the object if the silhouette have some noise).

 For sure we know that the object that we want to track will be the biggest contour on the binary image , so we can use a **"for"** loop to find the largest contour on the 'contour' vector and store it to be drawn on the image. The for loop goes from 0 to the size of the 'contour' vector. It will read every contour area using the **"contourArea"**  function , which will read the contour[i] points and calculate the area. The second argument "false" will deactivate the flag that will return a area-signed value ( depending if it is calculated clockwise or counter-clockwise ). Using "false" will return only the absolute value. Whenever the area calculated is greater than the previous , the conditional statement will save the the largest area and the index that where is this contour ( we want to save where the points that represent the largest contour is so we can drawn it later ).


----
<Code:c++ linenums:172>
//draw the largest contour on the image		
		drawContours(coloredimage, contours, largest_contour_index, Scalar(0, 0, 255), 2, 8, hierarchy, 0, Point());
 
 
		// Show image with contours in a new window.	
		imshow("Find Contours", coloredimage);
 
		key = waitKey(25);
</Code>

 Finally we can draw the biggest contour on the coloredimage variable - which will show the web camera frame with plus the biggest contour . The** "drawContours"** will do the job. The first argument is the output image and the second argument is the contour vector. The third argument is important here , as it will define which contour will be drawn , and we already have the largest contour index saved on the 'largest_contour_index' variable. If you want to draw all the contours found , you can use -1 on this argument (Not recommended , the algorithm can found a lot of contours and the program can be very slow). The fourth argument will define the color of the contour.

 We show the image with the web camera frame and the contour drawn using the **"imshow"** function and update the key variable value to check if the loop will continue / the morphological transformations will be used or not.
\\
\\
The video below demonstrates the program running in real time.
{{youtube>h3T0-a4YmPs?medium}}