Kinect SDK: Minimum Bounding Box

Introduction

A standard problem in computer vision is deriving the minimum bounding box (MBB) for a user, and is often the first step in many computer vision applications. A MBB can be defined as the smallest rectangle completely enclosing a set of points. The task of identifying a MBB for a user becomes trivial with the Kinect SDK, and is accomplished by processing the depth frame data returned from the sensor.

Implementation

The XAML for the UI of the application is shown below. The code highlighted in yellow is responsible for displaying the video stream and overlaying it with the MBB. The MBB is a Rectangle that binds to a number of properties to control it’s location and size. It uses a converter (not shown) to control when the MBB is visible.

<Window x:Class="KinectDemo.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:conv="clr-namespace:KinectDemo.Converters"
        Title="Minimum Bounding Box" ResizeMode="NoResize" SizeToContent="WidthAndHeight"
        Loaded="Window_Loaded" Closed="Window_Closed">
    <Grid>
        <Grid.Resources>
            <conv:BooleanToVisibilityConverter x:Key="boolVis" />
        </Grid.Resources>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="Auto" />
            <ColumnDefinition Width="200" />
        </Grid.ColumnDefinitions>
        <StackPanel Grid.Column="0">
            <TextBlock Margin="0,10,0,10" 
                       HorizontalAlignment="Center"
                       Text="Video Stream" />
            <Viewbox Margin="10,0,10,10">
                <Grid Height="240"
                      Width="320">
                    <Image Source="{Binding ColourBitmap}" />
                    <Rectangle Height="{Binding Box.Height}"
                               HorizontalAlignment="Left"
                               RadiusX="5"
                               RadiusY="5"
                               Stroke="Red"
                               StrokeThickness="2"
                               VerticalAlignment="Top"
                               Visibility="{Binding Path=IsUserDetected, 
                                            Converter={StaticResource boolVis}}"
                               Width="{Binding Box.Width}">
                        <Rectangle.RenderTransform>
                            <TranslateTransform X="{Binding Box.X}"
                                                Y="{Binding Box.Y}" />
                        </Rectangle.RenderTransform>
                    </Rectangle>
                </Grid>
            </Viewbox>
        </StackPanel>
        <StackPanel Grid.Column="1">
            <GroupBox Header="Motor Control"
                      Height="100"
                      VerticalAlignment="Top"
                      Width="190">
                <StackPanel HorizontalAlignment="Center" 
                            Margin="10"
                            Orientation="Horizontal">
                    <Button x:Name="motorUp"
                            Click="motorUp_Click"
                            Content="Up"
                            Height="30"
                            Width="70" />
                    <Button x:Name="motorDown"
                            Click="motorDown_Click"
                            Content="Down"
                            Height="30"
                            Margin="10,0,0,0"
                            Width="70" />
                </StackPanel>
            </GroupBox>
            <GroupBox Header="Information"
                      Height="100"
                      VerticalAlignment="Top"
                      Width="190">
                <StackPanel Orientation="Horizontal" Margin="10">
                    <TextBlock Text="Frame rate: " />
                    <TextBlock Text="{Binding FramesPerSecond}"
                               VerticalAlignment="Top"
                               Width="50" />
                </StackPanel>
            </GroupBox>
        </StackPanel>
    </Grid>
</Window>

The constructor initializes the StreamManager class (contained in my KinectManager library), which handles the stream processing. The DataContext of MainWindow is set to kinectStream for binding purposes, and the UseBoundingBox property of the StreamManager class is set to true so that bounding box processing occurs.

        public MainWindow()
        {
            InitializeComponent();
            this.kinectStream = new StreamManager();
            this.DataContext = this.kinectStream;
            this.kinectStream.UseBoundingBox = true;
        }

The Window_Loaded event handler initializes the three required subsystems of the Kinect pipeline, and initialises the smoothing and filtering parameters that will be applied to the skeleton tracking data returned from the sensor. Finally, event handlers are registered for when a depth frame is available, and when a video frame is available. Each event handler simply invokes a method in the StreamManager class to process the received data.

        private void Window_Loaded(object sender, RoutedEventArgs e)
        {
            this.runtime = new Runtime();
            try
            {
                this.runtime.Initialize(
                    RuntimeOptions.UseDepthAndPlayerIndex | 
                    RuntimeOptions.UseSkeletalTracking | 
                    RuntimeOptions.UseColor);
                this.cam = runtime.NuiCamera;
            }
            catch (InvalidOperationException)
            {
                MessageBox.Show
                    ("Runtime initialization failed. Ensure Kinect is plugged in");
                return;
            }
            try
            {
                this.runtime.DepthStream.Open(ImageStreamType.Depth, 2, 
                    ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex);
                this.runtime.VideoStream.Open(ImageStreamType.Video, 2, 
                    ImageResolution.Resolution640x480, ImageType.Color);
            }
            catch (InvalidOperationException)
            {
                MessageBox.Show
                    ("Failed to open stream. Specify a supported image type/resolution.");
                return;
            }
            this.kinectStream.LastTime = DateTime.Now;
            this.runtime.SkeletonEngine.TransformSmooth = true;
            var parameters = new TransformSmoothParameters
            {
                Smoothing = 0.75f,
                Correction = 0.0f,
                Prediction = 0.0f,
                JitterRadius = 0.05f,
                MaxDeviationRadius = 0.04f
            };
            this.runtime.SkeletonEngine.SmoothParameters = parameters;
            this.runtime.DepthFrameReady += 
                new EventHandler<ImageFrameReadyEventArgs>(runtime_DepthFrameReady);
            this.runtime.VideoFrameReady += 
                new EventHandler<ImageFrameReadyEventArgs>(runtime_VideoFrameReady);
        }

Three important properties from the StreamManager class are shown below. UseBoundingBox is set to true if the MBB should be derived when processing depth frame data. Box is an instance of my MinimumBoundingBox type (not shown here). While Int32Rect seems like a natural type to use to represent a MBB, there is no property change notification built in, hence creating the MinimumBoundingBox type. IsUserDetected is set to true by ConvertDepthFrame if a user is detected from the depth frame data, and is bound to from the Visibility property of the Rectangle in the UI (thus only making the MBB visible when a user is detected).

        public bool UseBoundingBox { get; set; }
        public MinimumBoundingBox Box { get; private set; }    
        public bool IsUserDetected { get; private set; }

GetDepthStream is a method in the StreamManager class. This method takes the depth data, and converts it by invoking the ConvertDepthFrame method.

        public void GetDepthStream(ImageFrameReadyEventArgs e)
        {
            if (depthFrame32 == null)
            {
                depthFrame32 = new byte[e.ImageFrame.Image.Width * 
                    e.ImageFrame.Image.Height * 4];
            }
            PlanarImage image = e.ImageFrame.Image;
            byte[] depthFrame = ConvertDepthFrame(image.Bits);
            this.DepthBitmap = BitmapSource.Create(image.Width, image.Height, 
                96, 96, PixelFormats.Bgr32, null, depthFrame, image.Width * 4);
            this.OnPropertyChanged("DepthBitmap");
        }

ConvertDepthFrame converts a 16-bit grey scale depth frame into a 32-bit frame. It does this by transforming the 13-bit depth information into an 8-bit intensity value. If the UseBoundingBox property is set to true, it also derives the MBB values and sets the Box properties to the derived values.

        private byte[] ConvertDepthFrame(byte[] depthFrame16)
        {
            int topLeftX = int.MaxValue;
            int topLeftY = int.MaxValue;
            int bottomRightX = int.MinValue;
            int bottomRightY = int.MinValue;
            bool isUserDetected = false;
            for (int i16 = 0, i32 = 0; 
                i16 < depthFrame16.Length && i32 < depthFrame32.Length; 
                i16 += 2, i32 += 4)
            {
                int user = depthFrame16[i16] & 0x07;
                int realDepth = (depthFrame16[i16 + 1] << 5) | (depthFrame16[i16] >> 3);
                byte intensity = (byte)(255 - (255 * realDepth / 0x0fff));
                if (this.UseBoundingBox)
                {
                    if (user != 0)
                    {
                        int y = (i32 / 4) / this.DepthBitmap.PixelWidth;
                        int x = (i32 / 4) - this.DepthBitmap.PixelWidth * y;
                        if (x < topLeftX)
                            topLeftX = x;
                        if (x > bottomRightX)
                            bottomRightX = x;
                        if (y < topLeftY)
                            topLeftY = y;
                        if (y > bottomRightY)
                            bottomRightY = y;
                        isUserDetected = true;
                    }
                }
                depthFrame32[i32 + RED_IDX] = (byte)(intensity / 2);
                depthFrame32[i32 + GREEN_IDX] = (byte)(intensity / 2);
                depthFrame32[i32 + BLUE_IDX] = (byte)(intensity / 2);
            }
            if (this.UseBoundingBox)
            {
                if (topLeftX != int.MaxValue)
                {
                    this.Box.X = topLeftX;
                }
                if (bottomRightX != int.MinValue)
                {
                    this.Box.Width = bottomRightX - topLeftX;
                }
                if (topLeftY != int.MaxValue)
                {
                    this.Box.Y = topLeftY;
                }
                if (bottomRightY != int.MinValue)
                {
                    this.Box.Height = bottomRightY - topLeftY;
                }
                if (isUserDetected != IsUserDetected)
                {
                    this.IsUserDetected = isUserDetected;
                    this.OnPropertyChanged("IsUserDetected");
                }
            }
            return this.depthFrame32;
        }

The application is shown below. Once a user is identified from the depth stream returned from the Kinect sensor, the MBB is derived and the Rectangle representing the MBB is made visible over the video stream. As the user moves, the MBB changes it’s size and location.

Conclusion

The Kinect for Windows SDK beta from Microsoft Research is a starter kit for application developers. It allows access to the Kinect sensor, and experimentation with its features. The sensor provides depth data to the application which can be used to accurately derive a minimum bounding box for a user. Deriving a minimum bounding box for a user is the first step in developing a security application that is capable of identifying users based upon their skeletal data.

Kinect SDK: ‘Make me into a Cyberman’ Part II

Introduction

A previous post documented an application that ‘upgraded’ a person into a cyberman. The application used speech recognition to recognise the word upgrade, then tracked the skeleton of the user in order to identify their head, before superimposing a cyber helmet over their head. The cyber helmet continued to be superimposed while the user was tracked. However, an omission in this application was that regardless of the distance from the Kinect sensor, the cyber helmet remained a fixed width and height that covered the user’s head only for a set distance away. As the user approached the sensor their head would show around the cyber helmet. In this blog post I will describe how I’ve incorporated depth data into the application, to scale the width and height of the cyber helmet based upon the distance the user is from the sensor.

Implementation

Previously the Image control that defined the cyber head was of a fixed size. Here, it’s changed to use a MaxHeight and MaxWidth of 150. The rest of the XAML is as in the previous post.

                <Image x:Name="head"
                       MaxHeight="150"
                       MaxWidth="150"
                       Source="Images/cyberman.png"
                       Visibility="Collapsed" />

The Kinect sensor does not have sufficient resolution to ensure consistent accuracy of the skeleton tracking data, over time. This problem manifests itself as the data seeming to vibrate around their positions. However, the Kinect SDK provides an algorithm for filtering and smoothing incoming data from the sensor, which can be seen highlighted in the Window_Loaded event handler below. The parameters can be manipulated to provide the required level of filtering and smoothing for your desired user experience.

        private void Window_Loaded(object sender, RoutedEventArgs e)
        {
            this.runtime = new Runtime();
            try
            {
                this.runtime.Initialize(RuntimeOptions.UseColor | 
                    RuntimeOptions.UseSkeletalTracking);
                this.cam = runtime.NuiCamera;
            }
            catch (InvalidOperationException)
            {
                MessageBox.Show
                    ("Runtime initialization failed. Ensure Kinect is plugged in");
                return;
            }
            try
            {
                this.runtime.VideoStream.Open(ImageStreamType.Video, 2, 
                    ImageResolution.Resolution640x480, ImageType.Color);
            }
            catch (InvalidOperationException)
            {
                MessageBox.Show
                    ("Failed to open stream. Specify a supported image type/resolution.");
                return;
            }
            this.runtime.SkeletonEngine.TransformSmooth = true;
            var parameters = new TransformSmoothParameters
            {
                Smoothing = 1.0f,
                Correction = 0.1f,
                Prediction = 0.1f,
                JitterRadius = 0.05f,
                MaxDeviationRadius = 0.05f
            };
            this.runtime.SkeletonEngine.SmoothParameters = parameters;
            
            this.kinectStream.LastTime = DateTime.Now;
            this.runtime.VideoFrameReady += 
                new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
            this.runtime.SkeletonFrameReady += 
                new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);
        }

The nui_SkeletonFrameReady event handler retrieves a frame of skeleton data. e.SkeletonFrame.Skeletons is an array of SkeletonData structures, each of which contains the data for a single skeleton. If the TrackingState field of the SkeletonData structure indicates that the skeleton is being tracked, a loop enumerates all of the joints in the SkeletonData structure. If the joint ID is equal to the head, the cyber helmet is resized based upon the user’s distance from the Kinect sensor. Then getDisplayPosition is called in order to convert coordinates in skeleton space to image space. The image of the cyber helmet is then drawn on the Canvas at the coordinates returned by getDisplayPosition.

        private void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
        {
            if (this.Upgrade == true)
            {
                foreach (SkeletonData data in e.SkeletonFrame.Skeletons)
                {
                    if (SkeletonTrackingState.Tracked == data.TrackingState)
                    {
                        foreach (Joint joint in data.Joints)
                        {
                            if (joint.Position.W < 0.6f)
                            {
                                return;
                            }
                            if (joint.ID == JointID.Head)
                            {
                                this.head.Height = this.head.MaxHeight / joint.Position.Z;
                                this.head.Width = this.head.MaxWidth / joint.Position.Z;
                                var point = this.getDisplayPosition(joint, 
                                    (int)this.head.Width / 2, (int)this.head.Height / 2);
                                Canvas.SetLeft(this.head, point.X);
                                Canvas.SetTop(this.head, point.Y);
                            }
                        }
                    }
                }
            }
        }

For an explanation of how the getDisplayPosition method works, see this previous post. The only slight change here is that the method has been modified to accept a width offset and a height offset as parameters, in order to handle the changing size of the cyber helmet based upon the distance of the user from the Kinect sensor.

        private Point getDisplayPosition(Joint joint, int widthOffset, int heightOffset)
        {
            int colourX, colourY;
            float depthX, depthY;
            this.runtime.SkeletonEngine.SkeletonToDepthImage(joint.Position, 
                out depthX, out depthY);
            depthX = Math.Max(0, Math.Min(depthX * 320, 320));
            depthY = Math.Max(0, Math.Min(depthY * 240, 240));
            ImageViewArea imageView = new ImageViewArea();
            this.runtime.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(
                ImageResolution.Resolution640x480, imageView, (int)depthX, 
                (int)depthY, 0, out colourX, out colourY);
            return new Point((int)(this.canvas.Width * colourX / 640) - widthOffset, 
                (int)(this.canvas.Height * colourY / 480) - heightOffset);
        }

Although not shown here, I’ve also refactored the stream access code into a library called KinectManager, which uses bindings to update the data on the UI.

The application is shown below. When the user says ‘upgrade’, and provided that the skeleton tracking engine has identified a person, a cyber helmet is placed over the identified user’s head, with the helmet still covering the user’s head as they are tracked. Furthermore, the cyber helmet is resized to cover the user’s head based upon their distance from the Kinect sensor.

Conclusion

The Kinect for Windows SDK beta from Microsoft Research is a starter kit for application developers. It allows access to the Kinect sensor, and experimentation with its features. The sensor provides skeleton tracking data to the application, that can be used to accurately overlay an image on a specific part of a person in real time. Furthermore, depth data can be utilised to ensure that overlaid images are scaled correctly based upon the user’s distance from the sensor.