Custom IKinectController – Drag and Drop

Drag and Drop sample with IKinectManipulateableController

This sample shows how you can implement a custom KinectControl, for example to move things around in a Canvas.

To hook into the whole KinectRegion magic you can implement your own UserControl that implements IKinectControl.
When you move your hand KinectRegion keeps track of the movements and will constantly check if there is some Kinect enabled control at the current hand pointer’s position.
To determine such a control the KinectRegion looks for IKinectControls.

The IKinectControl interface forces you to implement the method IKinectController CreateController(IInput inputModel, KinectRegion kinectRegion).
The IKinectControl is also required to implement IsManipulatable and IsPressable.
This way you are specifying on what gestures your control will react to.
Because this sample shows how to move controls around the Draggable control’s property IsManipulatable returns true.

UserControl Draggable.cs implements IKinectControl

public sealed partial class Draggable : UserControl, IKinectControl
{
    public Draggable()
    {
        this.InitializeComponent();
    }

    public IKinectController CreateController(IInputModel inputModel, KinectRegion kinectRegion)
    {
        // Only one controller is instantiated for one Control
        var model = new ManipulatableModel(inputModel.GestureRecognizer.GestureSettings, this);
        return new DragAndDropController(this, model, kinectRegion);
    }

    public bool IsManipulatable { get { return true; } }

    public bool IsPressable { get { return false; } }
}

You’ll find two interfaces in the SDK *.Controls namespace that implement IKinectController and of course they are closely related to the other properties you have to implement.

Those interfaces are:
– IKinectPressableController
– IKinectManipulateableController

Since IsManipulatable returns true CreateController should return an instance of IKinectManipulateableController.
For an example how to implement IKinectPressableController see here.

Controller class DragAndDropController

public class DragAndDropController : IKinectManipulatableController, IDisposable
{
}

The DragAndDropController‘s constructor gets a reference to the manipulatable control, an IInputModel and a reference to the KinectRegion.

public DragAndDropController(FrameworkElement element, ManipulatableModel model, KinectRegion kinectRegion)
{
    this.element = new WeakReference(element);
    this.kinectRegion = kinectRegion;
    this.inputModel = model;

    if (this.inputModel == null)
        return;
...

The ManipulatableModel provides four events you can subscribe to in order to react to user input.
This sample uses the nuget package Kinect.ReactiveV2.Input that provides specific Rx-extension methods to subscribe to these events.

...
    this.eventSubscriptions = new CompositeDisposable 
    {
        this.inputModel.ManipulationStartedObservable()
                       .Subscribe(_ => VisualStateManager.GoToState(this.Control, "Focused", true)),

        this.inputModel.ManipulationInertiaStartingObservable()
                       .Subscribe(_ => Debug.WriteLine(string.Format("ManipulationInertiaStarting: {0}, ", DateTime.Now))),

        this.inputModel.ManipulationUpdatedObservable()
                       .Subscribe(_ => OnManipulationUpdated(_)),

        this.inputModel.ManipulationCompletedObservable()
                       .Subscribe(_ => VisualStateManager.GoToState(this.Control, "Unfocused", true)),
    };
}

All subscriptions are composed into one CompositeDisposable that is disposed in the controllers’s Dispose() method.

  • ManipulationStartedObservable –> is fired when the user closes it’s hand
  • ManipulationInertiaStartingObservable –> ???
  • ManipulationUpdatedObservable –> is fired when the user moves it’s hand while keeping it closed
  • ManipulationCompletedObservable –> is fired when the user releases it’s hand

For this sample the most interesting observable is ManipulationUpdatedObservable. Everytime this event is fired the method OnManipulationUpdated is called.

private void OnManipulationUpdated(KinectManipulationUpdatedEventArgs args)
{
    var dragableElement = this.Element;
    if (!(dragableElement.Parent is Canvas)) return;

    var delta = args.Delta.Translation;
    var translationPoint = new Point(delta.X, delta.Y);
    var translatedPoint = InputPointerManager.TransformInputPointerCoordinatesToWindowCoordinates(translationPoint, this.kinectRegion.Bounds);

    var offsetY = Canvas.GetTop(dragableElement);
    var offsetX = Canvas.GetLeft(dragableElement);

    if (double.IsNaN(offsetY)) offsetY = 0;
    if (double.IsNaN(offsetX)) offsetX = 0;

    Canvas.SetTop(dragableElement, offsetY + translatedPoint.Y);
    Canvas.SetLeft(dragableElement, offsetX + translatedPoint.X);
}

If this IKinectControl is placed inside a Canvas it’s position is updated relative to the users hand.

The complete code is found here.
This article in markdown is found here.

Advertisements

21 thoughts on “Custom IKinectController – Drag and Drop

  1. Thanks for sharing.
    How would you get control of the Drag Content at runtime.
    Let’s say I put an image control in drag content then at runtime if I drag it right it should increase width/height and on drag left it should decrease

    How would you go about doing that?

    • In the constructor of DragAndDropController you’ll get a reference to the Draggable.
      Save this reference in a field and you can access it in OnManipulationUpdated.

      To increase and decrease the width/heigth you’ll have to evaluate KinectManipulationUpdatedEventArgs’ Delta.Translation property or even better the Delta.Scale property for z-axis changes.
      Applying a ScaleTransform (in WPF) accordingly to the content should do the trick.

      The K4W SDK Browser includes a sample that shows how to do ImageZooming. It’s in the ‘Controls Basic-WPF’ or ‘Controls Basic-XAML’ sample, tile with caption ‘Zoomable Photo’.

  2. I am trying the demo in XAML / Windows store

    ManipulationStarted, Completed events are not raised. This is what I’m doing

    public DragAndDropController(Draggable element, ManipulatableModel model, KinectRegion kinectRegion)
    {
    ….
    draggableElement = element;

    draggableElement.ManipulationStarted += draggableElement_ManipulationStarted;
    draggableElement.ManipulationCompleted += draggableElement_ManipulationCompleted;
    }

    • The events are fired on the object of type ManipulatableModel (in this case ‘model’) not Draggable.

      This should work for your sample:
      model.ManipulationStarted += draggableElement_ManipulationStarted;
      model.ManipulationCompleted += draggableElement_ManipulationCompleted;

      • Sorry for not making myself clear, I already got this working with one hand, what I want to do is Zoom the dragContent ‘using both hands’
        Now I know KinectRegion only supports One hand but is there a way a IKinectControl can raise an event for two hands?

        I am trying this from the Controls basics sample which works fine as far as drawing a Ellipse for the second hand is concerned
        But I still can’t grab any IKinectControl/Draggable controls with it due to the KinectRegion limitation,

        What would be a way to ‘detect’ if I am over a Draggable content with my second hand?

        Any work around that?

        KinectCoreWindow kinectCoreWindow = KinectCoreWindow.GetForCurrentThread();
        kinectCoreWindow.PointerMoved += kinectCoreWindow_PointerMoved;

        private void kinectCoreWindow_PointerMoved(KinectCoreWindow sender, KinectPointerEventArgs args)
        {
        KinectPointerPoint kinectPointerPoint = args.CurrentPoint;

        var dragableElement = this.Element;

        var mainScreen = dragableElement.Parent as Canvas;
        if (lastTime == 0 || lastTime != kinectPointerPoint.Timestamp)
        {
        lastTime = kinectPointerPoint.Timestamp;
        if (mainScreen != null)
        {
        foreach (var child in mainScreen.Children)
        {
        if (child is StackPanel || child is Ellipse) mainScreen.Children.Remove(child);
        }
        }
        }

        RenderPointer(args.CurrentPoint.Properties.IsPrimary, kinectPointerPoint.Properties.IsEngaged,
        kinectPointerPoint.Position,
        kinectPointerPoint.Properties.UnclampedPosition,
        kinectPointerPoint.Properties.HandReachExtent,
        kinectPointerPoint.Properties.BodyTimeCounter,
        kinectPointerPoint.Properties.BodyTrackingId,
        kinectPointerPoint.Properties.HandType);
        }

  3. Could you please provide a link to download a zip file for this project. Github is not showing to download zip file for this.

  4. I have a question. I am following this tutorial, you made folder with names Common, Properties and Themes and put different types of files in these folders. What functionality are they performing and why are these important ?

    • Those folders are included in the visual studio project template I’ve used. The files in there are not necessarily required for the sample. But be aware that Common\HandPointerStyles.xaml gives you the Kinect UI Element styles that the Kinect for Windows SDK provides.

  5. I have another question. In my xaml page I am using draggable user control and inside that user control I am using another user control which is an image. But when I am calling second user control is CS file it’s giving nullpointerexception.
    Here is the piece of code.

    Rocket is null.
    Please help me to get rid of this problem.

  6. i have placed image inside dragelement.if i drag image outside canvas then its goes way.How would i restrict boundry using kinect.

    • You could check if the translatedPoint is still in the Canvas e.g. with VisualTreeHelper. If so call Canvas.SetTop and Canvas.SetLeft, if not do nothing or show a warning or something alike.

  7. Hey Marcus,

    Thank you for the awesome tutorial. I want to implement Drag and Drop operation with Kinect V2 for copy & paste.

    I have three UserControls (Left, Middle, Right) which are hosted in one Main UserControl(UC). I have three images in Left UC, so if a user grabs one image from these three and drags this image to the right UC and drops it, i want to show this image in the right UC and also save its path in a file. The images in the left UC should remain the same, i.e., three and in the same order(the image which i dragged and dropped has been copied to right UC). I am trying my luck for quite a time now but not able to implement this.

    Can you please guide me how can i implement this functionality with kinect. I am new to Kinect programming so i appreciate your help in this regard. Thank you.

    • As far as I understand correctly you’ll need to implement the functionality to copy the image. This should be done in the ManipulationStartedObservable subscription. But keep in mind that you can’t drag the copied image but only the original one.

      Implement a copy method on the image element and call this method in the ManipulationStartedObservable subscription.
      In the ManipulationCompletedObservable subscription you’ll save the path in a file.

  8. Thank you for helping. I wanted the drag and drop functionality also for my image so instead of copying i created a duplicate drag drop element so its working well for my requirements.

    In my main.xaml i have three images (image 1,2,3) with drag drop functionality. If user taps or presses image 2 then only image 2 should be active for drag and drop. The other two images even if the user wants to drag and drop should not be able to do. They should be inactive for the user. To summarize this only the image that is taped or pressed by the user should be active for manipulation(drag and drop). I have implemented tap or press by wrapping Image in a Button.

    I also tried to implement both pressable and manipulatable on the same user control element but if IsManipulatable is true then i get null reference for pressable. i am refering to the same problem which you have posted here.

    https://social.msdn.microsoft.com/Forums/en-US/4ca9dcdc-50bd-4893-8b39-697fc2477478/ikinectcontrol-that-is-both-pressable-and-manipulatable?forum=kinectv2sdk

    Have you found a workaround for this? Thanks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s