The Imaginative Universal

Studies in Virtual Phenomenology -- @jamesashley

Kinect Application Project Template

June 18
by James Ashley 18. June 2013 13:21

Over the past year, every time I start a new Kinect for Windows project, I’ve basically just copied the infrastructure code from a previous project.  The starting point was the code my friend Jarrett Webb and I wrote for our book Beginning Kinect Programming with the Microsoft Kinect SDK, but I’ve made incremental improvements to this code as needed and based on pointers I’ve found in various places.  I finally realized that I’d made enough changes and it was time to just turn this base code into a project template for myself and my colleagues at work.  Realizing that there wasn’t a Kinect Application project template available yet on the visual studio gallery, I uploaded it there, also.

The cool thing about templates uploaded to the gallery is that anyone with visual studio can now install it from the IDE.  If you select Tools | Extension Manager … and then search for “Kinect” under the Online Gallery, you should see something like this.  From here you can install the Kinect Application project template to your computer.

Kinect Application Project Template

If you then create a new project and look under C# | Windows, you will be able to build a Kinect WPF application with a bit of a headstart.  Here are some key features:

1. Initialization Code

All the initialization code and Kinect stream event handlers are stubbed out in the InitSensor method.  All you need to do is uncomment the streams you want to use.  Additionally, the event handler code is also stubbed out with the proper pattern for opening and disposing of frame objects.  Whatever you need to do with the image, depth and skeleton frames can be done inside those using statements.  This code also uses the latest agreed upon best practices for efficiently managing streamed data as of the 1.7 SDK.

void sensor_ColorFrameReady(object sender
    , ColorImageFrameReadyEventArgs e)
{
    using (ColorImageFrame frame = e.OpenColorImageFrame())
    {
        if (frame == null)
            return;

        if (_colorBits == null) _colorBits = 
            new byte[frame.PixelDataLength];
        frame.CopyPixelDataTo(_colorBits);

        throw new NotImplementedException();
    }
}

2. Disposal Code

Whatever you enable in the InitSensor method you will need to disable and dispose of in the DeInitSensor method.  Again, this just requires uncommenting the appropriate lines.  The DeInitSensor also implements a disposal pattern that is somewhat popular now.  The sensor is actually shut down on a background thread rather than on the main thread.  I’m not sure if this is a best practice as such, but it resolves a problem many C# developers were running into in shutting down their Kinect-enabled applications.

3. Status Changed Code

The Kinect can actually be disconnected in mid-process or simply not be on when you first run an application.  It is also surprisingly common to forget to plug the Kinect’s power supply in.  Generally, your application will just crash in such situations.  If you properly handle the KinectSensors.StatusChanged event, however, your application will just start up again when you get the sensor plugged back in.  A pattern for doing this was first introduced in the KinectChooser component in the Developer Toolkit.  A lightweight version of this pattern is included in the Kinect Application Project Template.

void KinectSensors_StatusChanged(object sender
    , StatusChangedEventArgs e)
{
    if (e.Status == KinectStatus.Disconnected)
    {
        if (_sensor != null)
        {
            DeInitSensor(_sensor);
        }
    }

    if (e.Status == KinectStatus.Connected)
    {
        _sensor = e.Sensor;
        InitSensor(_sensor);
    }
}
 

4. Extension Methods

While most people were working on controls for the Kinect 4 Windows SDK, Clint Rutkas and the Coding4Fun guys brilliantly came up with the idea of developing extension methods for handling the various Kinect streams.

The extension methods included with this template provide lots of conversions from bitmaps byte arrays to BitmapSource types (useful for WPF image controls) and vice-versa.  This allows you to do something easy like display a color stream which otherwise can be rather hairy.  The snippet below assumes there is an image control in the MainWindow named canvas.

using (ColorImageFrame frame = e.OpenColorImageFrame())
{
    if (frame == null)
        return;

    if (_colorBits == null) _colorBits = 
        new byte[frame.PixelDataLength];
    frame.CopyPixelDataTo(_colorBits);

    // new line
    this.canvas.Source = 
        _colorBits.ToBitmapSource(PixelFormats.Bgr32, 640, 480);
}

More in line with the original Coding4Fun Toolkit, the extension methods also make some very difficult scenarios trivial – for instance background subtraction (also known as green screening), skeleton drawing, player masking.  These methods should make it easier for quickly mock up a demo or even show off the power of the Kinect in the middle of a presentation using just a few lines of code.

private void InitSensor(KinectSensor sensor)
{
    if (sensor == null)
        return;

    sensor.ColorStream.Enable();
    sensor.DepthStream.Enable();
    sensor.SkeletonStream.Enable();
    sensor.Start();
    this.canvas.Source = sensor.RenderActivePlayer();
}

Again, this code assumes there is an image control in MainWindow named canvas.  You’ll want to put the following code in the InitSensor method to ensure that the code is called again if your Kinect sensor accidentally gets dislodged.  To create a simple background subtraction image, enable the color, depth and skeleton streams and then call the RenderActivePlayer extension method.  By stacking another image beneath the canvas image, I create an effect like this:

me on tatooine

Here are some overloads of the RenderActivePlayer method and the effects they create.  I’ve removed Tatooine from the background in the following samples.

 

canvas.Source = sensor.RenderActivePlayer(System.Drawing.Color.Blue);
blue_man

 

canvas.Source = sensor.RenderActivePlayer(System.Drawing.Color.Blue
                , System.Drawing.Color.Fuchsia);

blue_fuschia

 

canvas.Source = sensor.RenderActivePlayer(System.Drawing.Color.Transparent
                , System.Drawing.Color.Fuchsia);

trasnparent_fuschia

 

And so on.  There’s also this one:

canvas.Source = sensor.RenderPredatorView();

predator

 

… as well as this oldie but goodie:

canvas.Source = sensor.RenderPlayerSkeleton();
skeleton_view

The base method uses the colors (and quite honestly most of the code) from the Kinect Toolkit that goes with the SDK.  As with the RenderActivePlayer extension method, however, there are lots of overrides so you can change all the colors if you wish to.

canvas.Source = sensor.RenderPlayerSkeleton(System.Drawing.Color.Turquoise
    , System.Drawing.Color.Indigo
    , System.Drawing.Color.IndianRed
    , trackedBoneThickness: 1
    , jointThickness: 10);

balls

 

Finally, you can also layer all these different effects:

canvas.Source = sensor.RenderActivePlayer();
canvas2.Source = sensor.RenderPlayerSkeleton(System.Drawing.Color.Transparent);
everything

Tags: ,

Kinect

Comments

8/10/2013 1:56:50 AM #

Farzin

hi
i try to write this code but something is wrong in my code that i can not run the program that i write (i do not know how to active RenderActivePlayer). i need code that can show me Skeleton and RGB picture together, like your last picture. could you plaese send this project that you write here to me.
thanks you in advance
Farzin

Farzin United States

8/17/2013 4:09:35 PM #

erez

Hi,
I put this code in Init:

canvas.Source = sensor.RenderActivePlayer(System.Drawing.Color.Blue)

But nothing happaned .

This is my code : did i miss something ?


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using System.Drawing;
using Microsoft.Kinect;
using ImaginativeUniversal;
using System.ComponentModel;

namespace KinectApp1
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {

        #region private members

        private KinectSensor _sensor;
        byte[] _colorBits;
        DepthImagePixel[] _depthBits;
        Skeleton[] _skeletons;

        public MainWindow()
        {
            this.Closing += new System.ComponentModel.CancelEventHandler(MainWindow_Closing);
            this.Loaded += new RoutedEventHandler(MainWindow_Loaded);
            InitializeComponent();
        }

        #endregion

        #region kinect setup

        private void InitSensor(KinectSensor sensor)
        {
            if (sensor == null)
                return;

            sensor.ColorFrameReady += new EventHandler<ColorImageFrameReadyEventArgs>(sensor_ColorFrameReady);
            sensor.DepthFrameReady += new EventHandler<DepthImageFrameReadyEventArgs>(sensor_DepthFrameReady);
            sensor.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(sensor_SkeletonFrameReady);
            sensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(sensor_AllFramesReady);

            sensor.ColorStream.Enable();
            sensor.DepthStream.Enable();
            sensor.SkeletonStream.Enable();
            sensor.Start();
            this.canvas.Source = sensor.RenderActivePlayer(System.Drawing.Color.Blue);
       }

        private void DeInitSensor(KinectSensor sensor)
        {
            if (sensor == null)
                return;

            sensor.AllFramesReady -= sensor_AllFramesReady;
           sensor.SkeletonFrameReady -= sensor_SkeletonFrameReady;
            sensor.DepthFrameReady -= sensor_DepthFrameReady;
            sensor.ColorFrameReady -= sensor_ColorFrameReady;

            BackgroundWorker bw = new BackgroundWorker();
            bw.DoWork += (a, b) =>
            {
                sensor.Stop();

               sensor.SkeletonStream.Disable();
                sensor.DepthStream.Disable();
                sensor.ColorStream.Disable();

                sensor = null;
            };
            bw.RunWorkerAsync();

        }

        void MainWindow_Loaded(object sender, RoutedEventArgs e)
        {
            KinectSensor.KinectSensors.StatusChanged += new EventHandler<StatusChangedEventArgs>(KinectSensors_StatusChanged);
            _sensor = KinectSensor.KinectSensors.FirstOrDefault();
            InitSensor(_sensor);
        }

        void MainWindow_Closing(object sender, System.ComponentModel.CancelEventArgs e)
        {
            DeInitSensor(_sensor);
        }


        void KinectSensors_StatusChanged(object sender, StatusChangedEventArgs e)
        {
            if (e.Status == KinectStatus.Disconnected)
            {
                if (_sensor != null)
                {
                    DeInitSensor(_sensor);
                }
            }

            if (e.Status == KinectStatus.Connected)
            {
                _sensor = e.Sensor;
                InitSensor(_sensor);
            }
        }

        #endregion

        #region kinect stream handlers

        void sensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
        {
      
            using (ColorImageFrame frame = e.OpenColorImageFrame())
            {

                if (frame == null)
                    return;

                if (_colorBits == null) _colorBits =
                    new byte[frame.PixelDataLength];
                frame.CopyPixelDataTo(_colorBits);

                // new line
                this.canvas.Source =
                    _colorBits.ToBitmapSource(PixelFormats.Bgr32, 640, 480);                
              
              
            }
        }

        void sensor_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
        {
            using (DepthImageFrame frame = e.OpenDepthImageFrame())
            {
                if (frame == null)
                    return;

                if (_depthBits == null) _depthBits = new DepthImagePixel[frame.PixelDataLength];
                
                frame.CopyDepthImagePixelDataTo(_depthBits);
            
                //throw new NotImplementedException();
            }
        }

        void sensor_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
        {
            using (SkeletonFrame frame = e.OpenSkeletonFrame())
            {
                if (frame == null)
                    return;

                if (_skeletons == null) _skeletons = new Skeleton[frame.SkeletonArrayLength];
                frame.CopySkeletonDataTo(_skeletons);

                //throw new NotImplementedException();
            }
        }


        void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
        {
           /// throw new NotImplementedException();
        }

        #endregion
    }
}

erez Israel

8/17/2013 4:11:10 PM #

erez

I mean i able to see the picture , but the subtraction  is not working...

erez Israel

Comments are closed