Drag & Drop with Kinect for Windows

With the inclusion of the InteractionStream and the ability to detect a Grip gesture in the Kinect for Windows SDK Update 1.7 it’s now possible to grab UI elements on a screen and move them around. This blog post shows a possible implementation in a WPF application. Please notice that I’m using the following nuGet packages

• FluentKinect to initialize the Kinect sensor
• Rx-Main to have the LINQ operators available on events
• Rx-PlatformServices includes ObserveOnDispatcher()
• Kinect.Reactive to subscribe to the InteractionStream

Code-Behind: MainWindow.cs

// this code can be called after initialization of the MainWindow

// Get a kinect instance with started SkeletonStream and DepthStream
var kinect = await KinectConnector.GetKinect();
kinect.KickStart();

// instantiate an object that implements IInteractionClient
var interactionClient = new InteractionClient();

// GetUserInfoObservable() method is available through Kinect.Reactive
kinect.GetUserInfoObservable(interactionClient)
      .SelectMany(_ => _.Select(__ => __.HandPointers.Where(___ => ___.IsActive)))
      .Where(_ => _.FirstOrDefault() != null)
      .Select(_ => _.First())
      .ObserveOnDispatcher()
      .Subscribe(_ =>
      {
            var region = this.kinectRegion;
            var p = new Point(_.X * region.ActualWidth, _.Y * region.ActualHeight);
            if (_.HandEventType == InteractionHandEventType.Grip)
            {
                  var elem = this.kinectRegion.InputHitTest(p) as TextBlock));
                  if (elem != null)
                  {
                        this.lastTouched = elem;
                  }									  
            }
            else if(_.HandEventType == InteractionHandEventType.GripRelease)
            {
                  this.lastTouched = null;
            }
            else
            {
                  if (this.lastTouched == null) return;
                  
                  Canvas.SetLeft(this.lastTouched, p.X - this.lastTouched.ActualWidth / 2);
                  Canvas.SetTop(this.lastTouched, p.Y - this.lastTouched.ActualHeight / 2);
            }
	});

XAML: MainWindow.xaml

<Window x:Class="DragAndDrop.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:dd="clr-namespace:DragAndDrop"
        xmlns:k="clr-namespace:Microsoft.Kinect.Toolkit.Controls;assembly=Microsoft.Kinect.Toolkit.Controls"
        Title="MainWindow" WindowState="Maximized">
    <Window.Resources>
        <Style TargetType="TextBlock">
            <Setter Property="Height" Value="200" />
            <Setter Property="Width" Value="200" />
            <Setter Property="Foreground" Value="White" />
            <Setter Property="FontWeight" Value="ExtraBold" />
            <Setter Property="FontSize" Value="35" />
            <Setter Property="Text" Value="Drag Me" />
            <Setter Property="TextAlignment" Value="Center" />
            <Setter Property="Background" Value="Black" />
        </Style>
    </Window.Resources>
    <k:KinectRegion x:Name="kinectRegion" KinectSensor="{Binding Kinect}">
        <Grid>
            <Grid.RowDefinitions>
                <RowDefinition Height="100" />
                <RowDefinition Height="*" />
            </Grid.RowDefinitions>
            <Grid Grid.Row="0">
                <k:KinectUserViewer x:Name="userViewer" />
            </Grid>
            <Canvas Grid.Row="1">
                <TextBlock Canvas.Left="50" Canvas.Top="50" />
                <TextBlock Canvas.Left="260" Canvas.Top="50" />
                <TextBlock Canvas.Left="470" Canvas.Top="50" />
                <TextBlock Canvas.Left="680" Canvas.Top="50" />
                <TextBlock Canvas.Left="890" Canvas.Top="50" />
            </Canvas>
        </Grid>
    </k:KinectRegion>
</Window>