Drag & Drop with Kinect for Windows

With the inclusion of the InteractionStream and the ability to detect a Grip gesture in the Kinect for Windows SDK Update 1.7 it’s now possible to grab UI elements on a screen and move them around. This blog post shows a possible implementation in a WPF application. Please notice that I’m using the following nuGet packages

Code-Behind: MainWindow.cs

// this code can be called after initialization of the MainWindow

// Get a kinect instance with started SkeletonStream and DepthStream
var kinect = await KinectConnector.GetKinect();
kinect.KickStart();

// instantiate an object that implements IInteractionClient
var interactionClient = new InteractionClient();

// GetUserInfoObservable() method is available through Kinect.Reactive
kinect.GetUserInfoObservable(interactionClient)
      .SelectMany(_ => _.Select(__ => __.HandPointers.Where(___ => ___.IsActive)))
      .Where(_ => _.FirstOrDefault() != null)
      .Select(_ => _.First())
      .ObserveOnDispatcher()
      .Subscribe(_ =>
      {
            var region = this.kinectRegion;
            var p = new Point(_.X * region.ActualWidth, _.Y * region.ActualHeight);
            if (_.HandEventType == InteractionHandEventType.Grip)
            {
                  var elem = this.kinectRegion.InputHitTest(p) as TextBlock));
                  if (elem != null)
                  {
                        this.lastTouched = elem;
                  }									  
            }
            else if(_.HandEventType == InteractionHandEventType.GripRelease)
            {
                  this.lastTouched = null;
            }
            else
            {
                  if (this.lastTouched == null) return;
                  
                  Canvas.SetLeft(this.lastTouched, p.X - this.lastTouched.ActualWidth / 2);
                  Canvas.SetTop(this.lastTouched, p.Y - this.lastTouched.ActualHeight / 2);
            }
	});

XAML: MainWindow.xaml


<Window x:Class="DragAndDrop.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:dd="clr-namespace:DragAndDrop"
        xmlns:k="clr-namespace:Microsoft.Kinect.Toolkit.Controls;assembly=Microsoft.Kinect.Toolkit.Controls"
        Title="MainWindow" WindowState="Maximized">
    <Window.Resources>
        <Style TargetType="TextBlock">
            <Setter Property="Height" Value="200" />
            <Setter Property="Width" Value="200" />
            <Setter Property="Foreground" Value="White" />
            <Setter Property="FontWeight" Value="ExtraBold" />
            <Setter Property="FontSize" Value="35" />
            <Setter Property="Text" Value="Drag Me" />
            <Setter Property="TextAlignment" Value="Center" />
            <Setter Property="Background" Value="Black" />
        </Style>
    </Window.Resources>
    <k:KinectRegion x:Name="kinectRegion" KinectSensor="{Binding Kinect}">
        <Grid>
            <Grid.RowDefinitions>
                <RowDefinition Height="100" />
                <RowDefinition Height="*" />
            </Grid.RowDefinitions>
            <Grid Grid.Row="0">
                <k:KinectUserViewer x:Name="userViewer" />
            </Grid>
            <Canvas Grid.Row="1">
                <TextBlock Canvas.Left="50" Canvas.Top="50" />
                <TextBlock Canvas.Left="260" Canvas.Top="50" />
                <TextBlock Canvas.Left="470" Canvas.Top="50" />
                <TextBlock Canvas.Left="680" Canvas.Top="50" />
                <TextBlock Canvas.Left="890" Canvas.Top="50" />
            </Canvas>
        </Grid>
    </k:KinectRegion>
</Window>

Advertisements

13 thoughts on “Drag & Drop with Kinect for Windows

  1. Hi,
    I have tried tour code with the three packages I have installed with package manager console.

    I have two properties unavailable.

    First one HandPointers in the skeleton and lastTouched in the MainWindow.

    Any idea? My version of the toolkit is 1.8.

    Thank you!

    JSB

    • Hi,

      please make sure that you’ve added all necessary using statement in MainWindow.cs

      using FluentKinect;
      using Kinect.Reactive;
      using Microsoft.Kinect;
      using Microsoft.Kinect.Toolkit.Interaction;
      using System;
      using System.ComponentModel;
      using System.Linq;
      using System.Reactive.Linq;
      using System.Threading.Tasks;
      using System.Windows;
      using System.Windows.Controls;
      using System.Windows.Input;

      The missing lastTouched object is just a private field of type TextBlock in MainWindow.cs

      See this gist for further reference: https://gist.github.com/MarcusKohnert/51dc6e457f24db19771f

      • Hi,
        thanks a lot for the feedback.

        Sorry to bother you again.

        Now, I’ve got an error with ObserveOnDispatcher.

        It’s seem the property is not available. I’ve looking on the web , they said I can find out in System.Reactive.Windows.Threading but Reactive.windows doesn’t exists in my version.

        I’m lost, I really need to use the drag and drop with Kinect.

        Thanks a lot for your help.

        Regards,
        JSB

  2. Hello Marcus.
    I’m sorry for that I didn’t say hello for you first.
    May I ask that do they have the relationship between “Drag & Drop” and “Hover Button”?
    I’m thinking that if the function of Hover Button including Drag & Drop, can it works?

    Thank you very much.
    I’m a student. My name is Joyce.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s