Drag & Drop with Kinect for Windows

With the inclusion of the InteractionStream and the ability to detect a Grip gesture in the Kinect for Windows SDK Update 1.7 it’s now possible to grab UI elements on a screen and move them around. This blog post shows a possible implementation in a WPF application. Please notice that I’m using the following nuGet packages

Code-Behind: MainWindow.cs

// this code can be called after initialization of the MainWindow

// Get a kinect instance with started SkeletonStream and DepthStream
var kinect = await KinectConnector.GetKinect();

// instantiate an object that implements IInteractionClient
var interactionClient = new InteractionClient();

// GetUserInfoObservable() method is available through Kinect.Reactive
      .SelectMany(_ => _.Select(__ => __.HandPointers.Where(___ => ___.IsActive)))
      .Where(_ => _.FirstOrDefault() != null)
      .Select(_ => _.First())
      .Subscribe(_ =>
            var region = this.kinectRegion;
            var p = new Point(_.X * region.ActualWidth, _.Y * region.ActualHeight);
            if (_.HandEventType == InteractionHandEventType.Grip)
                  var elem = this.kinectRegion.InputHitTest(p) as TextBlock));
                  if (elem != null)
                        this.lastTouched = elem;
            else if(_.HandEventType == InteractionHandEventType.GripRelease)
                  this.lastTouched = null;
                  if (this.lastTouched == null) return;
                  Canvas.SetLeft(this.lastTouched, p.X - this.lastTouched.ActualWidth / 2);
                  Canvas.SetTop(this.lastTouched, p.Y - this.lastTouched.ActualHeight / 2);

XAML: MainWindow.xaml

<Window x:Class="DragAndDrop.MainWindow"
        Title="MainWindow" WindowState="Maximized">
        <Style TargetType="TextBlock">
            <Setter Property="Height" Value="200" />
            <Setter Property="Width" Value="200" />
            <Setter Property="Foreground" Value="White" />
            <Setter Property="FontWeight" Value="ExtraBold" />
            <Setter Property="FontSize" Value="35" />
            <Setter Property="Text" Value="Drag Me" />
            <Setter Property="TextAlignment" Value="Center" />
            <Setter Property="Background" Value="Black" />
    <k:KinectRegion x:Name="kinectRegion" KinectSensor="{Binding Kinect}">
                <RowDefinition Height="100" />
                <RowDefinition Height="*" />
            <Grid Grid.Row="0">
                <k:KinectUserViewer x:Name="userViewer" />
            <Canvas Grid.Row="1">
                <TextBlock Canvas.Left="50" Canvas.Top="50" />
                <TextBlock Canvas.Left="260" Canvas.Top="50" />
                <TextBlock Canvas.Left="470" Canvas.Top="50" />
                <TextBlock Canvas.Left="680" Canvas.Top="50" />
                <TextBlock Canvas.Left="890" Canvas.Top="50" />


17 thoughts on “Drag & Drop with Kinect for Windows

  1. Hi,
    I have tried tour code with the three packages I have installed with package manager console.

    I have two properties unavailable.

    First one HandPointers in the skeleton and lastTouched in the MainWindow.

    Any idea? My version of the toolkit is 1.8.

    Thank you!


    • Hi,

      please make sure that you’ve added all necessary using statement in MainWindow.cs

      using FluentKinect;
      using Kinect.Reactive;
      using Microsoft.Kinect;
      using Microsoft.Kinect.Toolkit.Interaction;
      using System;
      using System.ComponentModel;
      using System.Linq;
      using System.Reactive.Linq;
      using System.Threading.Tasks;
      using System.Windows;
      using System.Windows.Controls;
      using System.Windows.Input;

      The missing lastTouched object is just a private field of type TextBlock in MainWindow.cs

      See this gist for further reference: https://gist.github.com/MarcusKohnert/51dc6e457f24db19771f

      • Hi,
        thanks a lot for the feedback.

        Sorry to bother you again.

        Now, I’ve got an error with ObserveOnDispatcher.

        It’s seem the property is not available. I’ve looking on the web , they said I can find out in System.Reactive.Windows.Threading but Reactive.windows doesn’t exists in my version.

        I’m lost, I really need to use the drag and drop with Kinect.

        Thanks a lot for your help.


  2. Hello Marcus.
    I’m sorry for that I didn’t say hello for you first.
    May I ask that do they have the relationship between “Drag & Drop” and “Hover Button”?
    I’m thinking that if the function of Hover Button including Drag & Drop, can it works?

    Thank you very much.
    I’m a student. My name is Joyce.

  3. Thanks a lot, your “Basta 2013” code works.

    but ı want ask a question how can ı enable or disable hands. for example only active right hant or left hand

    • You should be able to filter events by hand type. Probably in MainWindows.cs line number 12. Don’t have the source at hat but something similar to this:
      __.HandPointers.Where(___ => ___.IsActive && __.IsLeft)

  4. Thanks a lot, your “Basta 2013” code works.

    ı change some where. but I want ask q question. can we detect finger grip.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s