Kinect + Emgu

Sorry about the inactivity but having and a daytime job and girlfriend sometimes leaves us without time.

Started to play with Computer Vision and Kinect, didn’t done a big thing but I was completly unaware of how CV works and after reading a while about it I hear that Emgu is a great wrapper for openCV library.
In fact you can do some face detection without any worries at all.
So let’s do a hands-on on this library
For this example you need
and to make things even more easier Coding4Fun Kinect Toolkit
This last one is great for helping to pass the kinect raw data to bitmaps
I strongly recommend to read the Emgu documentation before using it… it may be tricky
So let’s start by creating a new windows project, after that it’s time to add the references.
Add the Microsoft.Research.Kinect reference that can be found on the .NET references and then the Coding4Fun dll, and all the dlls needed for Emgu.
references
As you can see Emgu as lots of references and on the bin folder you should add the unmanaged libraries like
cvextern.dll, opencv_calib3d220 and so on… As I said before check the Emgu documentation to know what to do…
On the design view I added a EmguCV.UI.ImageBox and called it imageBox1.
On the code I added
using Emgu.CV;
using Emgu.CV.UI;
using Emgu.CV.Structure;
using Emgu.Util;
using Emgu.CV.CvEnum;
using Microsoft.Research.Kinect.Nui;
using Coding4Fun.Kinect.WinForm;
This is all the using you will need to a simple a Face Detection.
To make this work we will need 3 objects
Kinect.Nui.Runtime,
Emgu.CV.Image
and since the face detection will use the HaarCascade Algorithm we will use the HaarCascade object provided by Emgu
HaarCascade algorith need an xml file from where it will learn the patterns to recognize a face, there are a lot of files for face detection over the internet and you can create your own following this guide as an example (there are lot’s of other guides covering this topic)
so
haar = new HaarCascade("haarcascade_frontalface_alt_tree.xml");

will populate the the haarCascade object with that file.

Then it’s time to initialize the kinect. For now I will only need the RGB camera so I will use the RuntimeOption.UseColor.
We also need to create the event VideoFrameReady in order to get the frames we are working with and finally open the VideoStream.
For those who are not familiar with VideoStream.Open, it will support 4 parameters, the ImageStreamType, poolSize (the minimun value is 2, this will work as a backbuffer for storing data in memory), the ImageResolution and finaly the ImageType:
kinect.Initialize(RuntimeOptions.UseColor);
kinect.VideoFrameReady += new EventHandler(kinect_VideoFrameReady);
kinect.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
and this is it… the complete constructor should look like this:
public Form1()
{
      InitializeComponent();
      haar = new HaarCascade("haarcascade_frontalface_alt_tree.xml");
      kinect.Initialize(RuntimeOptions.UseColor);
      kinect.VideoFrameReady += new EventHandler(kinect_VideoFrameReady);
      kinect.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
}
Now when there is a VideoFrameReady the event will be fired and call kinect_VideoFrameReady method. it will receive the sender and an ImageFrameReadyEventArgs where you can find the image data generated by the sdk
The first task will be convert the captured image to a bitmap and convert it again to a Emgu image so it can be processed
 Image kinImage = new Image(e.ImageFrame.ToBitmap());
this does the trick!
then using the Emgu colored image we will process that nasty face detection
So we will convert the the Colored Emgu Image to a Grayscale one (better for image processing):
Image grayframe = nextFrame.Convert();
and then we will retrieve all the faces detected on that frame using the haarCascade:
 grayframe.DetectHaarCascade(
                                    haar, 1.4, 4,
                                    HAAR_DETECTION_TYPE.DO_CANNY_PRUNING,
                                    new Size(grayframe.Width / 8, grayframe.Height / 8)
                                    )[0];
and finally for each face detected we will get the face rectangle and draw it over the initial image,
foreach (var face in faces)
{                         
    nextFrame.Draw(face.rect, new Bgr(0, double.MaxValue, 0), 3);                    
}
and finally set the image to the imagebox1.
imageBox1.Image = nextFrame;
for the lazy ones here is who like to copy paste without understanding the code (can’t figure why they do it) here it goees:
void kinect_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
      {
            Image kinImage = new Image(e.ImageFrame.ToBitmap());
            using (Image; nextFrame = kinImage)
            {
                if (nextFrame != null)
                {
                    // there's only one channel (greyscale), hence the zero index
                    Image grayframe = nextFrame.Convert();
                    var faces =
                            grayframe.DetectHaarCascade(
                                    haar, 1.4, 4,
                                    HAAR_DETECTION_TYPE.DO_CANNY_PRUNING,
                                    new Size(grayframe.Width / 14, grayframe.Height / 14)
                                    )[0];
                    foreach (var face in faces)
                    {
                        nextFrame.Draw(face.rect, new Bgr(0, double.MaxValue, 0), 3);
                    }
                    imageBox1.Image = nextFrame;
                }
            }
        }
So as you can see easy code 😀
you can download the .cs file with all the code needed to do it
Here is a small demo of how it works, i’m sorry about the low framerate (my capture software sucks) and bad light conditions but it’s already 2am and my girlfriend is sleeping next to me so it’s better to don’t have more light.
But it’s nice too see that it still can detect my face in such bad lightning conditions :)
with good lightning, the detection occurs flawlessly

The first demo

If you want any piece of code please just comment and I’ll post it here 😉

Ok, so here is my first demo… Simple stuff, basically I have 2 views, one with the RGB VideoStream and another with the DepthFieldStream.

Pretty similar to the one of the Channel9 tutorial with some  minor changes like I’m not using 3 diferent distance keys to draw the colors and implemented a grayscale colouring depending on the distance…. so it becomes a very smooth…

Here’s a vid about it

Shell Scripting

For the first time in 10 years of programming someone asked for a shell script to do a bunch of operations.

I had never done such thing but I tought it should be easy.

One thing that helped me a lot was this cheat sheet. Extremelly usefull for some basic operations.

Also I was faced with a problem: How to get a dump of a pg database that is on a remote machine?

Well, I found a nice approach thanks to www.stackoverflow.com website. So here is the a little sample on how to perform that:

ssh $USERNAME@$HOSTNAME "pg_dump -f /dev/stdout -t tb1 -t tb2 -t tb3 dbname" > /export/bin/dbexport.sql err=$? if [ $err -ne 0 ]; then   echo "Dump failed with error code ${err}!" fi

Was this usefull or what?

Codebits 2011 is comming…

The greatest geek-nerd-techy event in Portugal is about to happen!

It’s only 2 days untill the codebits start!

Yes! It will have a 48 hours programming challenge
Yes! It will have a geeky quiz
Yes! It will have O’Reilly books
Yes! It will have lot’s of food an beverages
Yes! It will have arduinos
Yes! It will have Nuclear Tacos
YES! CODEBITS FTW!

and NO, there shouldn’t be no sleep!

Be Aware!

meo|kinect @ codebits 2011

SAPO Codebits 2011 was great, as always, but I found my project presentation a “little bit” compromised by some evil events

First of all I will talk about the project itself and then I’ll talk about the disastrous presentation.

Meo|Kinect

Probably you don’t know what “Meo” is unless you’re at Portugal.
Meo is a triple play service provided by Portugal Telecom (the main telco in Portugal). It gives you IPTV, broadband ADSL from 6 to 200Mbps and Voice Service.

To watch IPTV the hardware required is a Router Thomson and a Scientific Atlanta’s KMM3000 IPTV Set or similar model!
These boxes contain a WinCE as base OS with Microsoft MediaRoom. Microsoft MediaRoom is a extremelly closed software with a lot of license policies around it. Only the service providers and special licenced developers have access to the MediaRoom SDK so basically this is a Black Box.

 

It is possible to send commands that are remote like keys to the box with telnet and send  post via an http a request to open applications.

 

Now that you are more aware of what is the Meo Service I can go on and explain what was my concept:
One day before the contest on the codebits webpage there was a post mentioning that there would be a team helping to develop for this boxes and giving full support, also I saw that a microsoft team would be there too to help on Kinect based projects.

1+1= meo|Kinect

So I was interested in interfacing the kinect gestures with the meo box using a PC in the middle!

 

Since day one Gonçalo Basto (one of my team members) was doing some widgets for the box in order to explore the kinect use in the box with interactive widgets. We talked several times with PT Inovação, Microsoft,  sapo.labs teams and our ideias should be partially changed and the presentation on the main stage seemed a “little bit” compromised. They said that it wouldn’t be  impossible but would require some work from the organization to do it. “Keep in touch we will try to handle that, also try to talk with the organization”.

We put a lot of efford on that part reducing the development time and the solutions were getting delayed time after time.

We got the interface recognizing the gestures using hotspots on the computer screen that would send nice and clean commands to the box and time to end the project was near at the end when I finally went to test it on a test environment with all the required hardware and network configurations.

It was a success. But we were only 4 hours away from the presentation and nothing was done on stage.

We went there to test the main stage environment 1 hour before and they sent someone to put a network cable on the router and do the magic but it wasn’t working at all

I talked with Celso about our issues and he said… “that’s too late for such things” and I replied that we passed 3 days trying to get someone to help us. Celso grabbed his cellphone and said “I’ll see what I can do for you although I can’t promise you nothing it’s less than a 1 hour for the presentations”…

 

I was desperate….the team wasn’t motivated, I was thinking about giving up, and someone from the organization told me, “we will make that happen”

Miguel Vicente from Microsoft, which I most send kudos to him, was always giving us good advices and telling to not give up, PT Inovação Team was really interested in make that happen too. And suddenly magic things started to happen.

The organization moved our project to the last one on the list so we could win some time, PT Inovação sent a guy to do the set up, and we where literally laid down in front of the stage configuring all the necessary component to make the presentation… even if it was untested.

We finished all the set 5 minutes before our turn on stage and our presentation guys was insecure has I have never saw him. We didn’t believe that it would work on stage and IT DID WORK, but our orator was under a lot of pressure believing that it would never work and he was speechless…. all the presentation we trained, all the talk about making Interactive TV a new concept, the speech recognition features, the new market that could be possible to gain with such features… all was left behind and all we heard about it was: “As you can see I can go to the menus, and do some workout as I do it!”

The presentation was a failures dispite the demo worked after all.

I give my sincere thanks to all the Codebits organization to make the impossible happen, to PT Inovação for giving us a big help and for having one of the top techs on MediaRoom in Portugal laid on the ground in front of 800 persons just to help to configure the DNS hosts, to Miguel Vicente for all the motivation he gave us, and to my team that was under a great pressure due to all our missfortune events.

Soon I will publish more about this project including videos of it working.

It was a great codebits and a great challenge.

Smart Defrag

During this whole week I was trying to defrag my office’s laptop, everyday at night I was trying one more time running the windows disc defragmenter but the results were poor, and the red color was still dominating the graph.

Today I decided to google a little bit about defrag tools and I found this one quite interesting

Smart Defrag 2:

Until now these are the pros and cons about this software

pros:
-low resource consumption
-small application
-extremelly fast
-efficient
-freeware

cons:
-Be carefull when installing it has toolbars and other stuff that asks to be installed!

The usability is simple and there are just few and precise options what turn this software very intuitive.

Smart Defrag allows us to do 3 types of defrag:

-Simple Defragmentation
-Defrag+Optimization (fast)
-Defrag+Optimization(complete)

Basically you can choose the speed of the defrag and it’s  eficiency. It’s curious because this software analyses your files and directories in order to know which ones are must used and the optimization will move them to the fastest areas of your hard drive.

Also you have the possibility to keep the disc defragmenting in background all the time, and it will not consume too much resources on the machine and also the boot defragmentation that is an option to allow to defrag files that are usually locked by the OS!

From now on, this will be a must have on my computers!

You can download it here.

Arduino + Webcam = Volumetry

One of the projects I have worked occasionally in recent months was finding a way to measure materials volumes with “low-cost”! I thought about the challenge and started to get down to work and got amazing results using an ordinary webcam and a sensor with an Arduino. To begin with let’s talk about hardware: – Arduino + Distance Sensor Sharp GP2Y0A02 (20-150cm): I must say I thought it was the easiest but failed miserably. To achieve acceptable results (about 1 cm of deviation in distances near the limit of the sensor) I have had to follow a set of procedures that’s tricky because the sensor tends to return lots of noise and so we do not have accurate results. First I do 10 readings and determine the average value, then determine the upper and lower limit values ​​from this first average, capture another 10 and make the average ​​of all the values. I must say it worked really well

float GetAverage(float * _values, uint8_t size)
{
	float avg = 0;
	for(int i = 0; i< size; i++)
	{
		avg += _values[i];
	}
	avg = avg / size;

	float setMax = avg * 1.15;
	float setMin = avg * 0.85;
	float precisionAvg;
	int counter = 0;
	for(int i = 0; i < size; i++)
	{
		if((setMin < _values[i]) && (_values[i] < setMax))
		{
			precisionAvg += _values[i];
			counter++;
		}
	}
	return precisionAvg / counter;
}

To achieve volumetric measuring I used Emgu library and a few filters  to find the contours of the object based on a mask. In other words we must have a mask from the background in order to make the exclusion of what it already was there and just stay with the new object in the image. We use a few tricks of image processing, some trigonometric calculations and voila here is the result

Resizing the size of all the components in a java.awt.Frame

When we are used to use a programming language, sometimes we get our tasks a little bit automated.

I come from a background in C# and I am now doing some tasks in Java and I found some concept differences very quickly!

Task: Change the size of all the components inside a dialog based on a java.awt.Frame

After some failures figuring out how to reach my goal here is the best result I got.

protected void changeContent(Container parent) 
{
    for (Component comp : parent.getComponents()) 
    {
        changeFont(comp);
        comp.setMaximumSize( new java.awt.Dimension(comp.getWidth()*2, comp.getHeight()*2) );
        comp.setPreferredSize( new java.awt.Dimension(comp.getWidth()*2, comp.getHeight()*2) );
        comp.setSize( comp.getWidth()*2, comp.getHeight()*2);
        if (comp instanceof Container) 
        {
            changeContent((Container)comp);
        }
    }
}
protected void changeFont(Component comp) 
{
    AffineTransform at = AffineTransform.getScaleInstance(2, 2);
    if (comp.getFont() != null)
    {
        Font font = comp.getFont().deriveFont(at);
        comp.setFont(font);
    }
}

 

In my case, this little methods solved my problems and made my form grow like a charm! I hope this is usefull for someone else…