Subscribing to the InteractionStream the Rx way

The most exciting feature in the Kinect for Windows SDK Update 1.7 was probably the InteractionStream. The InteractionStream is based on the SkeletonStream and the DepthStream and it enables you to detect basic interactions like a grip gesture or a button push in a Kinect for Windows application.
Since the InteractionStream needs Skeleton- and DepthData for its calculations you have to provide the InteractionStream with depth and skeleton data whenever new frames are available.
A straight forward approach to do that could be the following:

var skeletonData = // initialize array
var depthData = // initialize array
var kinect = // somehow get a Kinect sensor instance
IInteractionClient interactionClient = // a class that implements IInteractionClient

var interactionStream = new InteractionStream(kinect, interactionClient);

kinect.AllFramesReady += (s, e) =>
{
    long skeletonTimestamp = 0;
    long depthTimestamp = 0;
    var accelerometerReading = kinect.AccelerometerGetCurrentReading();

    using (var depthImageFrame = e.OpenDepthImageFrame())
    using (var skeletonFrame = e.OpenSkeletonFrame())
    {
      if (depthImageFrame == null || skeletonFrame == null) return;

      skeletonFrame.CopySkeletonDataTo(skeletonData);
      skeletonTimestamp = skeletonFrame.Timestamp;
      depthData = depthImageFrame.GetRawPixelData();
      depthTimestamp = depthImageFrame.Timestamp;
    }

    interactionStream.ProcessDepth(depthData, depthTimestamp);
    interactionStream.ProcessSkeleton(skeletonData, accelerometerReading, skeletonTimestamp);
};

interactionStream.InteractionFrameReady += OnInteractionFrameReady;

// The method that handles the InteractionFrameReady events
private void InteractionFrameReady(object sender, InteractionFrameReadyEventArgs e)
{
    UserInfo[] userInfos = // initialize array
    using (var interactionFrame = e.OpenInteractionFrame())
    {
      if (interactionFrame != null)
        interactionFrame.CopyInteractionDataTo(userInfos);
    }

    // do something with the UserInfos array
}

Since I am a huge fan of the ReactiveExtensions framework I have tried to find a solution to encapsulate this code in one method and produce an IObservable. I wanted to be able to subscribe to the InteractionStream in the same ‘Rx way’ as I am used to to subscribe to the SkeletonStream for example. So the goal was to have something like this:

kinect.InteractionStreamObservable().Subscribe(userInfos => 
{
    // do something useful with the userInfos
});

This is the solution I have found and it is already included in the Kinect.Reactive NuGet package.

public static IObservable<UserInfo[]> GetUserInfoObservable(this KinectSensor kinectSensor, IInteractionClient interactionClient)
{
    // null checks and checks if streams are enabled

    return Observable.Create<UserInfo[]>(obs =>
    {
      var stream = new InteractionStream(kinectSensor, interactionClient);
      var allFramesSub = 
        kinectSensor.GetAllFramesReadyObservable()
                    .SelectStreams((_, __) => Tuple.Create(_.Timestamp, __.Timestamp))
	                .Subscribe(_ =>
	                {
                          var accelerometer = kinectSensor.AccelerometerGetCurrentReading();
                          stream.ProcessSkeleton(_.Item3, accelerometer, _.Item4.Item1);
                          stream.ProcessDepth(_.Item2, _.Item4.Item2);
	                });

      stream.GetInteractionFrameReadyObservable()
	        .SelectUserInfo()
	        .Subscribe(_ => obs.OnNext(_));

      return new Action(() =>
      {
        allFramesSub.Dispose();
        stream.Dispose();
      });
  });
}

Subscribing to the InteractionStream is now very easy including the benefits of the Rx-Framework.

kinect.GetUserInfoObservable(new InteractionClient ())
      .SelectMany(_ => _.Where(userInfo => userInfo.SkeletonTrackingId == 1))
      .SelectMany(_ => _.HandPointers.Where(handPointer => handPointer.HandType == InteractionHandType.Right))
      .// and so on…
 
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s