ContinousGrippedState in Kinect.Reactive

For a while now I was wondering why the Kinect’s InteractionStream sends only one InteractionHandEventType.Grip when the user closes its hand. While the user still holds its hand in a closed state the SDK will fire events that have a HandEventType of None. This confused me from the very beginning. Compared to mouse events you’ll get continous mousedown events when the user does not release the mouse button.

So I thought about a way to get the same functionality when using the Kinect for Windows SDKs 1.x InteractionStream.
This extension method solved my problem and is now part of Kinect.Reactive:

/// <summary>
/// Returns a sequence with continuous GrippedState HandEventType until GripRelease.
/// </summary>
/// <param name="source">The source observable.</param>
/// <returns>The observable.</returns>
public static IObservable<UserInfo[]> ContinousGrippedState(this IObservable<UserInfo[]> source)
{
  if (source == null) throw new ArgumentNullException("source");

  var memory = new Dictionary<Tuple<int, InteractionHandType>, object>();
  var propInfo = typeof(InteractionHandPointer).GetProperty("HandEventType");
  var handEventTypeSetter = new Action<InteractionHandPointer>(o => propInfo.SetValue(o, InteractionHandEventType.Grip));

  return source.Select(_ =>
  {
    _.ForEach(u => u.HandPointers.ForEach(h =>
   {
     if (h.HandEventType == InteractionHandEventType.Grip)
     {
        memory.Add(Tuple.Create(u.SkeletonTrackingId, h.HandType), null);
     }
     else if (h.HandEventType == InteractionHandEventType.GripRelease)
     {
        memory.Remove(Tuple.Create(u.SkeletonTrackingId, h.HandType));
     }
     else if (memory.ContainsKey(Tuple.Create(u.SkeletonTrackingId, h.HandType)))
     {
        handEventTypeSetter(h);
     }
   }));

   return _;
  });
}

Use the extension method this way and you’ll get continously events with e.HandEventType == InteractionHandEventType.Grip until you’ll release your hand.

IDisposable subscription = null;

KinectConnector.GetKinect().ContinueWith(k =>
{
  var disp = k.Result.KickStart(true)
              .GetUserInfoObservable(new InteractionClientConsole())
              .ContinousGrippedState()
              .SelectMany(_ => _.Select(__ => __.HandPointers.Where(CheckForRightGripAndGripRelease)))
              .Subscribe(_ => _.ForEach(__ => Console.WriteLine(String.Format("Active: {0}, HandEventType: {1}", __.HandType, __.HandEventType))));
                
  subscription = disp;
});

Console.WriteLine("Waiting...");
Console.ReadLine();
Advertisements

2 thoughts on “ContinousGrippedState in Kinect.Reactive

  1. Thanks for providing this information.
    Would be great incase you provide a simple running POC as I am unable to find a way to write the second code snipped i.e. between IDisposable subsc…………………….Console.ReadLine();

    Regards

    • You mean you’re missing the CheckForRightGripAndGripRelease method in line 8?
      I don’t have the sample at hand but it should be something like this:

      private bool CheckForRightGripAndGripRelease(InteractionHandPointer pointer)
      {
      return pointer.HandType == InteractionHandType.Right && pointer.HandEventType != InteractionHandEventType.None;
      }

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s