Kinect.ReactiveV2 – Rx-ing the Kinect for Windows SDK

A few weeks ago I was finally able to get my hands on to the new Kinect for Windows V2 SDK. There are a few API changes compared to V1. So I started to port Kinect.Reactive to the new Kinect for Windows Dev Preview SDK and Kinect.ReactiveV2 was born.

Kinect.ReactiveV2 is, as it’s older brother, a project that contains a bunch of extension methods that should ease the development with the Kinect for Windows SDK. The project uses the ReactiveExtensions (an open source framework built by Microsoft) to transform the various Kinect reader events into IObservable<T> sequences. This transformation enables you to use Linq style query operators on those events.

Here is an example of how to use the BodyIndexFrame data as an observable sequence.

using System.Linq;
using System.Reactive;
using Microsoft.Kinect;
using Kinect.ReactiveV2;

var sensor = KinectSensor.Default;

var bodyIndexFrameDescription = sensor.BodyIndexFrameSource.FrameDescription;
var bodyIndexData = new byte[bodyIndexFrameDescription.LengthInPixels];

      .Subscribe(data => someBitmap.WritePixels(rect, data, stride, 0));

You’ll also get an extension method called SceneChanges() on every KinectSensor instance which notifies all it’s subscribers whenever a person entered or left a scene.

using System;
using System.Linq;
using System.Reactive;
using Microsoft.Kinect;
using Kinect.ReactiveV2;

var sensor = KinectSensor.Default;

      .Subscribe(_ =>
            if (_.SceneChangedType is PersonEnteredScene)
                  Console.WriteLine("Person {0} entered scene", _.SceneChangedType.TrackingId);
            else if (_.SceneChangedType is PersonLeftScene)
                  Console.WriteLine("Person {0} left scene", _.SceneChangedType.TrackingId);

Until now there are extension methods included for the BodyFrame, BodyIndexFrame, ColorFrame, DepthFrame, InfraredFrame and MultiSourceFrame.

The source code is available here.
Download the nuget package from here, or directly typing Install-Package Kinect.ReactiveV2 into the package manager console.

Please be aware that “This is preliminary software and/or hardware and APIs are preliminary and subject to change”.

ContinousGrippedState in Kinect.Reactive

For a while now I was wondering why the Kinect’s InteractionStream sends only one InteractionHandEventType.Grip when the user closes its hand. While the user still holds its hand in a closed state the SDK will fire events that have a HandEventType of None. This confused me from the very beginning. Compared to mouse events you’ll get continous mousedown events when the user does not release the mouse button.

So I thought about a way to get the same functionality when using the Kinect for Windows SDKs 1.x InteractionStream.
This extension method solved my problem and is now part of Kinect.Reactive:

/// <summary>
/// Returns a sequence with continuous GrippedState HandEventType until GripRelease.
/// </summary>
/// <param name="source">The source observable.</param>
/// <returns>The observable.</returns>
public static IObservable<UserInfo[]> ContinousGrippedState(this IObservable<UserInfo[]> source)
  if (source == null) throw new ArgumentNullException("source");

  var memory = new Dictionary<Tuple<int, InteractionHandType>, object>();
  var propInfo = typeof(InteractionHandPointer).GetProperty("HandEventType");
  var handEventTypeSetter = new Action<InteractionHandPointer>(o => propInfo.SetValue(o, InteractionHandEventType.Grip));

  return source.Select(_ =>
    _.ForEach(u => u.HandPointers.ForEach(h =>
     if (h.HandEventType == InteractionHandEventType.Grip)
        memory.Add(Tuple.Create(u.SkeletonTrackingId, h.HandType), null);
     else if (h.HandEventType == InteractionHandEventType.GripRelease)
        memory.Remove(Tuple.Create(u.SkeletonTrackingId, h.HandType));
     else if (memory.ContainsKey(Tuple.Create(u.SkeletonTrackingId, h.HandType)))

   return _;

Use the extension method this way and you’ll get continously events with e.HandEventType == InteractionHandEventType.Grip until you’ll release your hand.

IDisposable subscription = null;

KinectConnector.GetKinect().ContinueWith(k =>
  var disp = k.Result.KickStart(true)
              .GetUserInfoObservable(new InteractionClientConsole())
              .SelectMany(_ => _.Select(__ => __.HandPointers.Where(CheckForRightGripAndGripRelease)))
              .Subscribe(_ => _.ForEach(__ => Console.WriteLine(String.Format("Active: {0}, HandEventType: {1}", __.HandType, __.HandEventType))));
  subscription = disp;


Azure (WebSite-) deployment additional dependencies

If you want to publish your (Website-) project from within Visual Studio via WebDeploy you need to watch out that all files that you actually need on the server are included in the WebDeploy package. You’ll get problems if you dynamically load Assemblies via Assembly.LoadFile or Assembly.LoadFrom. If you haven’t referenced those Dlls or VS projects directly these assemblies won’t be included in the WebDeploy package and the WebSite won’t work in Azure.

To include files that are not directly referenced in your project edit the (WebProject) *.csproj-File in the following way:

<!-- To modify your build process, add your task inside one of the targets below and uncomment it.
       Other similar extension points exist, see Microsoft.Common.targets. -->
  <Target Name="BeforeBuild">
  <Target Name="AfterBuild">
      <Content Include="[path to your dll]" /> <!-- the dll that should be included in the WebDeploy-package -->

Drag & Drop with Kinect for Windows

With the inclusion of the InteractionStream and the ability to detect a Grip gesture in the Kinect for Windows SDK Update 1.7 it’s now possible to grab UI elements on a screen and move them around. This blog post shows a possible implementation in a WPF application. Please notice that I’m using the following nuGet packages

Code-Behind: MainWindow.cs

// this code can be called after initialization of the MainWindow

// Get a kinect instance with started SkeletonStream and DepthStream
var kinect = await KinectConnector.GetKinect();

// instantiate an object that implements IInteractionClient
var interactionClient = new InteractionClient();

// GetUserInfoObservable() method is available through Kinect.Reactive
      .SelectMany(_ => _.Select(__ => __.HandPointers.Where(___ => ___.IsActive)))
      .Where(_ => _.FirstOrDefault() != null)
      .Select(_ => _.First())
      .Subscribe(_ =>
            var region = this.kinectRegion;
            var p = new Point(_.X * region.ActualWidth, _.Y * region.ActualHeight);
            if (_.HandEventType == InteractionHandEventType.Grip)
                  var elem = this.kinectRegion.InputHitTest(p) as TextBlock));
                  if (elem != null)
                        this.lastTouched = elem;
            else if(_.HandEventType == InteractionHandEventType.GripRelease)
                  this.lastTouched = null;
                  if (this.lastTouched == null) return;
                  Canvas.SetLeft(this.lastTouched, p.X - this.lastTouched.ActualWidth / 2);
                  Canvas.SetTop(this.lastTouched, p.Y - this.lastTouched.ActualHeight / 2);

XAML: MainWindow.xaml

<Window x:Class="DragAndDrop.MainWindow"
        Title="MainWindow" WindowState="Maximized">
        <Style TargetType="TextBlock">
            <Setter Property="Height" Value="200" />
            <Setter Property="Width" Value="200" />
            <Setter Property="Foreground" Value="White" />
            <Setter Property="FontWeight" Value="ExtraBold" />
            <Setter Property="FontSize" Value="35" />
            <Setter Property="Text" Value="Drag Me" />
            <Setter Property="TextAlignment" Value="Center" />
            <Setter Property="Background" Value="Black" />
    <k:KinectRegion x:Name="kinectRegion" KinectSensor="{Binding Kinect}">
                <RowDefinition Height="100" />
                <RowDefinition Height="*" />
            <Grid Grid.Row="0">
                <k:KinectUserViewer x:Name="userViewer" />
            <Canvas Grid.Row="1">
                <TextBlock Canvas.Left="50" Canvas.Top="50" />
                <TextBlock Canvas.Left="260" Canvas.Top="50" />
                <TextBlock Canvas.Left="470" Canvas.Top="50" />
                <TextBlock Canvas.Left="680" Canvas.Top="50" />
                <TextBlock Canvas.Left="890" Canvas.Top="50" />

Subscribing to the InteractionStream the Rx way

The most exciting feature in the Kinect for Windows SDK Update 1.7 was probably the InteractionStream. The InteractionStream is based on the SkeletonStream and the DepthStream and it enables you to detect basic interactions like a grip gesture or a button push in a Kinect for Windows application.
Since the InteractionStream needs Skeleton- and DepthData for its calculations you have to provide the InteractionStream with depth and skeleton data whenever new frames are available.
A straight forward approach to do that could be the following:

var skeletonData = // initialize array
var depthData = // initialize array
var kinect = // somehow get a Kinect sensor instance
IInteractionClient interactionClient = // a class that implements IInteractionClient

var interactionStream = new InteractionStream(kinect, interactionClient);

kinect.AllFramesReady += (s, e) =>
    long skeletonTimestamp = 0;
    long depthTimestamp = 0;
    var accelerometerReading = kinect.AccelerometerGetCurrentReading();

    using (var depthImageFrame = e.OpenDepthImageFrame())
    using (var skeletonFrame = e.OpenSkeletonFrame())
      if (depthImageFrame == null || skeletonFrame == null) return;

      skeletonTimestamp = skeletonFrame.Timestamp;
      depthData = depthImageFrame.GetRawPixelData();
      depthTimestamp = depthImageFrame.Timestamp;

    interactionStream.ProcessDepth(depthData, depthTimestamp);
    interactionStream.ProcessSkeleton(skeletonData, accelerometerReading, skeletonTimestamp);

interactionStream.InteractionFrameReady += OnInteractionFrameReady;

// The method that handles the InteractionFrameReady events
private void InteractionFrameReady(object sender, InteractionFrameReadyEventArgs e)
    UserInfo[] userInfos = // initialize array
    using (var interactionFrame = e.OpenInteractionFrame())
      if (interactionFrame != null)

    // do something with the UserInfos array

Since I am a huge fan of the ReactiveExtensions framework I have tried to find a solution to encapsulate this code in one method and produce an IObservable. I wanted to be able to subscribe to the InteractionStream in the same ‘Rx way’ as I am used to to subscribe to the SkeletonStream for example. So the goal was to have something like this:

kinect.InteractionStreamObservable().Subscribe(userInfos => 
    // do something useful with the userInfos

This is the solution I have found and it is already included in the Kinect.Reactive NuGet package.

public static IObservable<UserInfo[]> GetUserInfoObservable(this KinectSensor kinectSensor, IInteractionClient interactionClient)
    // null checks and checks if streams are enabled

    return Observable.Create<UserInfo[]>(obs =>
      var stream = new InteractionStream(kinectSensor, interactionClient);
      var allFramesSub = 
                    .SelectStreams((_, __) => Tuple.Create(_.Timestamp, __.Timestamp))
	                .Subscribe(_ =>
                          var accelerometer = kinectSensor.AccelerometerGetCurrentReading();
                          stream.ProcessSkeleton(_.Item3, accelerometer, _.Item4.Item1);
                          stream.ProcessDepth(_.Item2, _.Item4.Item2);

	        .Subscribe(_ => obs.OnNext(_));

      return new Action(() =>

Subscribing to the InteractionStream is now very easy including the benefits of the Rx-Framework.

kinect.GetUserInfoObservable(new InteractionClient ())
      .SelectMany(_ => _.Where(userInfo => userInfo.SkeletonTrackingId == 1))
      .SelectMany(_ => _.HandPointers.Where(handPointer => handPointer.HandType == InteractionHandType.Right))
      .// and so on…

await GetKinect()

In the first blog post about FluentKinect I’ve mentioned that I’m not very happy with the actual process of getting a KinectSensor instance from the KinectSensors collection.

FluentKinect has now been updated and the KinectConnector’s static method GetKinect is now awaitable.

If you now call KinectConnector.GetKinect() and no KinectSensor is connected to your PC a new Task is started that listens for StatusChanged events of the KinectSensor collection. If you later plugin a Kinect controller the Task returns the connected KinectSensor instance.

KinectConnector’s GetKinect method:

public static Task<KinectSensor> GetKinect()
	return Task.Factory.StartNew<KinectSensor>(() =>
		if (kinectSensor != null) return kinectSensor;

		var kinect = KinectSensor.KinectSensors
							.FirstOrDefault(_ => _.Status == KinectStatus.Connected);
		if (kinect != null)
			kinectSensor = kinect;
			return kinectSensor;

		using (var signal = new ManualResetEventSlim())
			KinectSensor.KinectSensors.StatusChanged += (s, e) =>
				if (e.Status == KinectStatus.Connected)
					kinectSensor = e.Sensor;
					coordinateMapper = new CoordinateMapper(kinectSensor);


		return kinectSensor;

How to use it:

var kinect = await KinectConnector.GetKinect();

It’s not thread safe at the moment but there are a few improvements now in my opinion.

    • You can start and debug your actual program without a Kinect controller connected because no more exception is thrown. This helps with Kinect programming on a plane for example. ;-)
    • Your programm starts faster because GetKinect returns immediately
    • Since GetKinect returns a Task you get the ability to await the result

The code was pushed to GitHub and the nuget package FluentKinect has been updated as well.

Looking forward to new improvements!