Content Master Technology Blog

Deploying Databases with Object-Level Permissions

2011-12-20T17:49:00.000Z

Cross-posted from Jason Lee's Blog

As I mentioned in my last post, we’re currently creating some guidance on deploying enterprise-scale applications. As we go along, I plan to blog about a few of the things that I find particularly tricky to figure out.

This time I want to look at database deployment. When you build a web application project in Visual Studio 2010, the Web Publishing Pipeline features allow you to hook into the IIS Web Deployment Tool (commonly known as “Web Deploy”) to package and optionally deploy your web application. As part of this process you can also deploy local databases to a target server environment. This is all nice and easy to configure through the project property pages in Visual Studio 2010, as shown below.

I don’t want to describe this process in any detail, you can find that elsewhere on the web (for example here).

Deploying databases in this way has advantages and disadvantages. On the plus side:

It’s easy.
It’s UI-driven.
It figures out most of the settings for you.

On the downside:

There’s no support for differential updates. In other words, the destination database is destroyed and recreated every time you deploy, so you’ll lose any data.
Some of the default database deployment settings are unsuitable for many real-world scenarios – this is the issue I want to focus on here.

In many cases, you’ll want to avoid the Web Deploy approach altogether and use VSDBCMD.exe to deploy and update your databases, but that’s a conversation for another day. In this post I want to focus on how you can change some of the default behaviours for database deployment using Web Deploy. In particular, Web Deploy omits object-level permissions by default. This causes problems if your database (a) contains stored procedures and (b) grants execute permissions on the stored procedures to database roles.

Suppose you’re using the Visual Studio 2010/Web Deploy approach to deploy an ASP.NET membership database from a local development machine to a destination server environment. (NB you’d typically only deploy a membership database if you’ve modified the schema, otherwise it’s easier just to run ASPNET_REGSQL.exe and create the database from scratch on the destination server). You opt for a full deployment, including schema and data from the source database. On the source database, you can see that various database roles are granted execute permissions on stored procedures:

However, when the database is recreated on the destination database server, these permissions are missing:

This can be mystifying at first—essentially, you add users to the built-in database roles such as aspnet_Membership_BasicAccess and aspnet_Membership_FullAccess, but membership of these roles has no effect. They’re basically just empty roles that aren’t mapped to any permissions.

The problem is that stored procedures are “objects” in database terms (don’t ask me, I’m not a DBA), and by default Web Deploy does not include object-level permissions when it scripts the database. To change this behaviour, you need to modify the project file for your web application project.

1. In the Solution Explorer window, right-click your web application project node, and then click Unload Project.

2. Right-click the project node again, and click Edit [project file].

3. Locate the PropertyGroup element that corresponds to your build configuration (for example Release|AnyCPU).

<PropertyGroup

Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">

3. Within this element, locate the PreSource element and add a Permissions=”True” attribute, as shown below. The Permissions attribute indicates that the database script should include all permissions, including object-level permissions, which are defined in the source database.

<PreSource Path="[Source Database Connection String]"

ScriptSchema="True"

ScriptData="True"

Permissions="True"

CopyAllFullTextCatalogs="False"

DriDefaults="True" />

</Object>

</ObjectGroup>

</Objects>

</PublishDatabaseSettings>

4. Save and close the project file.

There’s a whole host of settings you can add to the PreSource element to configure how your database is deployed. For example, if you want to deploy a database that already exists on the destination server, you need to add a ScriptDropsFirst=”True” attribute to the PreSource element – otherwise Web Deploy will complain that you’re trying to create objects that already exist. The full list of properties that you can set as PreSource attibutes can be tricky to track down unless you know how the database deployment process works:

Web Deploy uses the dbFullSql provider to deploy databases (the link includes some properties you can use as PreSource attributes).
Under the covers, the dbFullSql provider uses SQL Server Management Objects (SMO) to generate database scripts. The ScriptingOptions Properties page describes some SMO properties you can specify as PreSource attributes.

Alternatively, to get a full list of properties, you can run the following Web Deploy command:

msdeploy.exe –verb:sync –source:dbFullSql /?

There’s much more to database deployment than I can cover in a quick blog post, and we’ll cover these kinds of issues in much more detail when we publish to MSDN. For now though I hope this helps to shed some light on the intricacies of database deployment.

Deploying Web Packages as a Non-Administrator User

2011-12-20T15:03:00.000Z

Cross-posted from Jason Lee's Blog

Regular readers (all six of you ;-)) will have noticed that I haven’t posted about SharePoint for a while. For the last couple of months I’ve been working with the Developer Guidance team at Microsoft to write some MSDN content on enterprise-scale web deployment and application lifecycle management. I’ll let you know when the content is available, and I don’t plan to duplicate it here. What I want to do is just to draw attention to a couple of areas that I found particularly tricky to figure out.

The first area involves the IIS Web Deployment Tool (commonly known as “Web Deploy”), and a gotcha around deploying web packages as a non-administrator user. For brevity I’ll have to assume that you’re broadly familiar with:

Creating web packages for web application projects in Visual Studio 2010 (for example, see ASP.NET Web Application Project Deployment Overview).
The various different approaches you can use to deploy web packages (for example, see Using Web Deploy Remotely).

One of the big advantages of Web Deploy 2.0, on IIS 7 or later, is that non-administrator users can deploy web packages to specific IIS web sites. This is generally useful in two scenarios:

Hosted environments, where tenants need control over specific sites but do not have server-level administrator privileges.
Enterprise environments, where members of a development team may need to deploy specific sites but do not typically have server-level administrator privileges.

If you want to enable non-administrator users to deploy web packages, you need to configure the Web Deploy Handler on the target IIS web server. The other deployment approaches (the remote agent and the temp agent) don’t allow users who aren’t server administrators to deploy packages. I’ll assume that you’ve configured the Web Deployment Handler to allow a non-administrator user (FABRIKAM\User) to deploy content to a specific IIS website, as described here.

By default, the Web Deploy Handler exposes an HTTPS endpoint at the following address:

https://[server name]:8172/MSDeploy.axd

For example:

https://TESTWEB1:8172/MSDeploy.axd

However, when a non-administrator user deploys a web package to the Web Deploy Handler, they need to add the IIS website name to the endpoint address as a query string:

https://[server name]:8172/MSDeploy.axd?site=[site name]

For example:

https://TESTWEB1:8172/MSDeploy.axd?site=DemoSite

Why the difference? In a word, authorization. Your non-administrator user doesn’t have server-level access to IIS, they only have access to specific IIS websites. If they attempt to connect to the server-level endpoint, Web Deploy will an ERROR_USER_UNAUTHORIZED error. The event log on the destination server will show an IISWMSVC_AUTHORIZATION_SERVER_NOT_ALLOWED error like this:

So you’ve got to use the site query string. Now for the gotcha.

Due to an open bug in the current version of Web Deploy (2.1), you can’t specify a query string in the endpoint address if you use the .deploy.cmd file generated by Visual Studio to deploy your web package. In other words, this won’t work:

DemoProject.deploy.cmd /Y /M:https://TESTWEB1/MSDeploy.axd?site=DemoSite /U:FABRIKAM\User /P:Pa$$w0rd A/:Basic -allowUntrusted

I’ve seen some fairly bizarre “workarounds” for this—for example, drop the query string and use an administrator account—this works, but it kind of defeats the object when the whole point of the exercise was to use a non-administrator user to deploy the web package. What you need to do is to use Web Deploy (MSDeploy.exe) directly rather than running the .deploy.cmd file.

All the .deploy.cmd file contains is a bunch of parameterized Web Deploy commands. This is put together by the build process to take some of the work out of the deployment. For example, you don’t need to specify the location of the web package, the Web Deploy providers to use for the source and destination, the Web Deploy verb, or the location of the .SetParameters.xml file, because the .deploy.cmd file knows this already. However, there’s nothing to stop you using the raw Web Deploy commands directly. The easiest way to do this is to look at the output when you run the .deploy.cmd file – you’ll see the actual MSDeploy.exe commands written to the console window. You should see something like this (ignore the line breaks):

msdeploy.exe

-source:package='…\DemoProject.zip'

-dest:auto,

computerName='https://TESTWEB1:8172/MSDeploy.axd?site=DemoSite',

userName='FABRIKAM\User',

password='Pa$$w0rd',

authtype='Basic'

-verb:sync

-setParamFile:"…\DemoProject.SetParameters.xml"

-allowUntrusted

Run this command directly from the command line, using your non-administrator user credentials, and the deployment should succeed.

Packaging and deploying web applications is a fairly broad and complex topic, and I’ve had to gloss over many of the details in this blog post. The content we’re developing for MSDN will cover these kinds of issues in much more detail, and I’ll link to the content as soon as it’s available.

Thanks to Tom Dykstra at Microsoft for helping me troubleshoot the issue and pointing out the bug.

Installing Windows Intune Endpoint Protection over existing anti-malware software

2011-11-08T10:36:00.005Z

How does Windows Intune install Window Intune Endpoint Protection when there is existing anti-malware software on the client computer?

Over the past year Content Master has been involved in training many Microsoft employees, Partners, and IT Pros on the technical ins and outs of Windows Intune. A question we have been asked many times is exactly how the Windows Intune installation process behaves when installed on computers with existing anti-malware software. It’s important to recognise that when the Windows Intune client is installed the overriding principle is to ensure that all client computers are protected from malware. This means that if you install Windows Intune on a computer with no existing malware protection the Windows Intune Endpoint Protection (WIEP) agents will be installed. This is exactly what you would expect, but things get a little more complicated when an existing anti-malware solution is in place; so that is the focus of this blog.

Before you deploy the Windows Intune client, we strongly recommend that you create default Windows Intune policies to set a baseline for deploying updates, software, and especially WIEP. By determining the WIEP settings you require and configuring the appropriate policy settings before deploying the client to any computer, you can take control of the deployment process and save time and effort later on.

Now let’s examine the options available in the policy setting of the Windows Intune Agent Settings policy and see how they change the installation behaviour. Then we will then look at the behaviour when installing the Windows Intune client.

WIEP Settings in the Windows Intune Agent Settings policy

The Windows Intune Agent Settings policy includes settings that control the behaviour of Endpoint Protection, updates and network bandwidth.

Figure 1 shows the Endpoint Protection setting that controls the installation behaviour of WIEP:

Figure 1: Endpoint Protection Settings

Yes enables Windows Intune Endpoint Protection on client computers. If Windows Intune Endpoint Protection is not already installed on a client computer, it will be installed.

No disables Windows Intune Endpoint Protection on client computers. If Windows Intune Endpoint Protection is not already installed on a client computer, it will be installed only if another endpoint protection application is not already installed on the computer. When Windows Intune Endpoint Protection is installed in this case, it is installed and configured as disabled.

Only on computers that are unprotected when Endpoint Protection is installed enables Windows Intune Endpoint Protection on client computers that do not already have another endpoint protection application installed. If another endpoint protection application is already installed on a client computer, this policy setting causes Endpoint Protection to not be installed on that computer. This is the default and recommended option.

You can find more information on the Endpoint Protection Policy Settings and other Windows Intune Agent Policy Settings in Windows Intune Help.

Installation Behaviour:

The default behaviour of the WIEP installation is to ensure that the client computer is protected. If you create a Windows Intune Agent Settings policy and configure the Endpoint Protection policy settings, you can control the WIEP installation behaviour. This is described in the flowchart below:

Figure 2: WIEP Installation Behaviour Flowchart

There are five paths through this flowchart:

Path 1 (1-2-3): This path describes the simplest situation; where there is no endpoint protection software installed on the client computer the Windows Intune client installation will install and enable WIEP.

Path 2 (1-2-4-5): This path describes an environment where Microsoft Security Essentials (MSE) or Forefront Endpoint Protection (FEP) is installed on the client computer. In this case, the existing endpoint protection software is automatically upgraded to WIEP.

Path 3 (1-2-4-6-7): This path describes an environment where MSE or FEP is not installed on the client computer, and there is no Endpoint Protection policy defined in the Windows Intune Agent Settings policy. In this case, WIEP is not installed and the client computer continues to use the existing endpoint protection software.

Path 4 (1-2-4-6-8-9): This path describes an environment where MSE or FEP is not installed on the client computer, there is an Endpoint Protection policy defined in the Windows Intune Agent Settings policy, and the existing endpoint protection software is recognised.[1] In this case, the Windows Intune client installation removes the existing endpoint protection software and installs WIEP.

Path 5 (1-2-4-6-8-10): This path describes an environment where MSE or FEP is not installed on the client computer, there is an Endpoint Protection policy defined in the Windows Intune Agent Settings policy, and the existing endpoint protection software is not recognised. In this case, the Windows Intune client installation installs WIEP in a disabled state and the client computer continues to use the existing anti-malware software.

Windows Intune will always install the Windows Intune Endpoint Protection agent, even in Paths 3 and 4 where the Windows Intune Endpoint Protection engine is not installed. This is illustrated in figures 3 and 4 below:

Figure 3: Windows Intune Endpoint Protection Agent installed

Figure 4: Windows Intune Endpoint Protection Agent and Windows Intune Endpoint Protection engine installed

Recommendations

The most important thing is to determine whether you want to use Windows Intune to manage your endpoint protection, with all the benefits that brings. Windows Intune Endpoint Protection helps enhance the security of computers in your organization by providing real-time protection against potential threats; keeping malware definitions up-to-date; and automatically running scans that you can control with policies. All of this is integrated into the Windows Intune administrator console to provide a centralised point of management, along with customisable reports that help you to keep track of malware issues and their resolution.

If you decide to use WIEP, you should plan and implement the appropriate Endpoint Protection policy settings in the Windows Intune Agent Settings policy before you deploy the Windows Intune client software. In this way, you can control the WIEP installation as the client joins the Windows Intune account.

As always, it is wise to experiment with different methods of installing WIEP in a test lab, so that you can determine which method will work best in your organisation. For example, you may want to verify that the Windows Intune client installation can successfully remove existing anti-malware software from your client computers. You may also want to verify that WIEP can co-exist with your existing endpoint protection application if it is not one of the recognised packages.

Conclusion

Windows Intune includes a fully featured and cloud enabled endpoint protection solution based on the Forefront Endpoint Protection 2010 product. If you plan for the deployment of WIEP and configure the appropriate Windows Intune Agent Settings policy before you install the Windows Intune client on computers throughout your organisation you will have more control of the endpoint protection status as soon as they are managed by Windows Intune.

[1] A list of anti-malware software that Windows Intune recognises can be found at: http://onlinehelp.microsoft.com/windowsintune/hh127706.aspx

Blogger: User Profile: Rose Malcolm

Building Windows Phone Apps: A Developer’s Guide

2011-09-22T16:37:00.001+01:00

For anyone looking for a great book on developing Windows Phone 7 applications, take a look here.

“This book combines useful, technical information about the platform, written by those who’ve been there and done it, with some stories about the people and the apps they’d written. The book will act as a handy reference that gives you a ‘puddy up’ into developing on the platform and, importantly, helps you to avoid making the same mistakes others have made.”

The best news is that the digital download is completely free, although you can also get a hard copy for a mere £3.86!

Working with the Documents Tab on the SharePoint Ribbon

2011-08-29T19:07:00.000+01:00

Cross-posted from Jason Lee's Blog

This week I've been taking a look at using ribbon controls with the SharePoint JavaScript client object model to drive some custom functionality. Ribbon customizations for SharePoint 2010 are fairly well documented. However, when you work with contextual tab groups—and the Documents tab in particular—there are a few nuances and idiosyncrasies that it's worth being aware of up front.

In this case, I want to add a ribbon button that enables the user to perform some additional actions when they select a file in a document library. There are countless scenarios in which you might want to do this – for example, you might add a "Request a copy of this document in large print/audio format/Welsh" control to the ribbon and use the document metadata to prepopulate an InfoPath form. To start with, however, I want to keep it simple:

When the user selects a document in a document library, display a button on the ribbon.
When the user clicks the button, display some information about the selected document as a client-side notification.

The logical place to put this button is on the Documents tab. This is part of the Library Tools contextual tab group – it's contextual because it's only displayed when the context is relevant, i.e. when the user browses to a document library. The Documents tab is selected automatically when the user selects one or more documents in the library's list view web part:

Let's take it one bit at a time for now, and I'll provide a full code listing at the bottom. Firstly, like all declarative ribbon customizations, we start with a CustomAction feature element:

<CustomAction Id="Jason.SP.GSD"
              Location="CommandUI.Ribbon"
              Sequence="11"
              RegistrationType="List"
              RegistrationId="101">

The key point of note here is that if you plan to add controls to a contextual tab, you must use the RegistrationType and RegistrationId attributes to target your ribbon customizations to an appropriate list type. If you're deploying controls to a standard ribbon tab, you can get away with omitting these attributes. It didn't initially occur to me that it should be any different in this case – I'm adding controls to the Documents tab, the Documents tab only shows up when I'm looking at a document library, I shouldn't have to worry about scope, right? But no – if you don't set these attributes, your controls simply won't show up on the tab. In this case, a RegistrationType of "List" and a RegistrationId of "101" scopes our ribbon customization to the document library base type.

Next, we define the controls we want to add to the ribbon. This process is identical regardless of whether you're adding to a contextual tab or a regular tab. In this case, we want to add a new group named "Jason's Actions" to the Documents tab. Within this group, we want to create a single button labelled "Get Selection Details". To accomplish this we need to create two CommandUIDefinition elements – one to define the maximum size of my group element, and one to define the group itself. Creating tabs, groups, and controls has been covered comprehensively elsewhere, so I don't want to go into too much detail – if you're looking for more information in this area, Chris O'Brien's blog post series is an excellent place to start.

In this case, we'll use the absolute minimum markup required to add a new button in its own group - a Group element to define the group and the controls within it, and a MaxSize element that defines how the group should be rendered on the ribbon. You can specify many more elements if you want – for example you can add a Scale element to specify how your group should render at different sizes, and you can define your own GroupTemplate to specify precisely how controls within your group should be arranged. However, each group must have a matching MaxSize element – otherwise it won't appear on the tab. The easiest approach to creating ribbon controls is to pick out existing controls that resemble what you're looking for and take a look at how they're defined. Let's say we want our group and button to look like the Share & Track group shown here – a large, simple layout with a single control:

To replicate this group, the first step is to take a look at the group definition. Ribbon controls are defined in the 14\TEMPLATE\GLOBAL\XML\CMDUI.XML file. To find specific elements in this file, unless you know the ID of the element you're looking for, it's best to start with the top-level elements and narrow down your search – start by finding the right tab group (Id="Ribbon.LibraryContextualGroup"), then locate the correct tab (Id="Ribbon.Document"), then identify the group you're looking for. In this case, the group we want to borrow from has an ID of "Ribbon.Documents.Share":

<Group Id="Ribbon.Documents.Share"
       Sequence="40"
       Command="ShareGroup"
       Description=""
       Title="$Resources:core,cui_GrpShare;"
       Image32by32Popup=".../formatmap32x32.png" 
       Image32by32PopupTop="-128" 
       Image32by32PopupLeft="-64"
       Template="Ribbon.Templates.Flexible2">

By examining the definition of this group, we can figure out the properties we need:

The Share & Track group has a Template attribute of Ribbon.Templates.Flexible2. This identifies the group template that gets applied to the group (also defined in CMDUI.XML if you want to take a closer look). We'll use this value to apply the same layout to our own group.
The Share & Track group has a Sequence attribute of 40. We'll use a value of 41 to place our group immediately to the right of the Share & Track group.

Next, we can take a look at how controls are defined within the group. For example, the following markup defines the E-mail a Link button you saw in the previous image:

<Button Id="Ribbon.Documents.Share.EmailItemLink"
        Sequence="10"
        Command="EmailLink"
        Image16by16=".../formatmap16x16.png" 
        Image16by16Top="-16" 
        Image16by16Left="-88"
        Image32by32=".../formatmap32x32.png" 
        Image32by32Top="-128" 
        Image32by32Left="-448"
        LabelText="$Resources:core,cui_ButEmailLink;"
        ToolTipTitle="$Resources:core,cui_ButEmailLink;"
        ToolTipDescription="...,cui_STT_ButEmailLinkDocument;"
        TemplateAlias="o1"
/>

In this case, the TemplateAlias attribute is the value that interests us. Every group template contains one or more placeholders, represented by ControlRef elements, in which you can place your controls. In this case, the E-mail a link button specifies that it should be added to the o1 placeholder in the Flexible2 group template. If we use the same value in our own button, we should get the same result.

Finally, we can also take a look at the matching MaxSize element for the Share & Track group. Remember that these elements are always paired – a Group element always has a corresponding MaxSize element defined within the same tab. Within each MaxSize element, the GroupId attribute identifies the corresponding group:

<MaxSize Id="Ribbon.Documents.Scaling.Share.MaxSize"
         Sequence="40"
         GroupId="Ribbon.Documents.Share" 
         Size="LargeLarge" 
/>

In this case, all we're interested in is the Size attribute. A group template can define multiple layouts, and this attribute identifies the specific layout in the Flexible2 template that we want to use – in this case, the LargeLarge layout.

We can now use all this information we've collected to define our group and button:

<CommandUIExtension>
  <CommandUIDefinitions>
    <CommandUIDefinition Location="Ribbon.Documents.Scaling._children"> 
      <MaxSize Id="Jason.SP.GSD.JasonsActions.MaxSize"
        Sequence="11"
        GroupId="Jason.SP.GSD.JasonsActions"
        Size="LargeLarge" /> 
    </CommandUIDefinition>
    <CommandUIDefinition Location="Ribbon.Documents.Groups._children">
      <Group Id="Jason.SP.GSD.JasonsActions"
        Sequence="41"
        Title="Jason's Actions"
        Description="Contains custom document actions"
        Template="Ribbon.Templates.Flexible2"> 
          <Controls Id="Jason.SP.GSD.JasonsActions.Controls">
            <Button Id="Jason.SP.GSD.JasonsActions.GetButton"
              Sequence="1"
              Image32by32=".../ThumbsUp.PNG"
              LabelText="Get Selection Details"
              Description="Gets the details of the selected document"
              TemplateAlias="o1"
              Command="Jason.SP.GSD.GetCmd" />
          </Controls>
        </Group>
      </CommandUIDefinition>
    </CommandUIDefinitions>

There are a few additional points worth mentioning at this stage:

You need a CommandUIDefinition element for each block of XML you want to add to the ribbon.
When setting the Location attribute, imagine you're slotting the XML directly into the CMDUI.XML file. Look up the ID of the parent element you want to add to, and append "._children" to get your Location value. For example, we want to add our group to the Groups element with an ID of "Ribbon.Documents.Groups", so our Location attribute is "Ribbon.Document.Groups._children".

Note that the button has a Command attribute value of "JL.GetSelectionDetails". This ties the button to a CommandUIHandler element in which we can define the JavaScript that should run when the user clicks the button, as shown below:

    <CommandUIHandlers>
      <CommandUIHandler  
        Command="Jason.SP.GSD.GetCmd"
        EnabledScript="javascript:
          SP.ListOperation.Selection.getSelectedItems().length == 1;"
        CommandAction="javascript: 
          var selectedItems = 
            SP.ListOperation.Selection.getSelectedItems();
          var item = selectedItems[0];
          var itemID = item['id'];
          if (item['fsObjType'] == 0) {
            SP.UI.Notify.addNotification(String.format(
              'Document selected: ID={0}', itemID));
          }
          else {
            SP.UI.Notify.addNotification(String.format(
              'Folder selected: ID={0}', itemID));
          }" 
      /> 
    </CommandUIHandlers>
  </CommandUIExtension>
</CustomAction>

The first point of interest is the EnabledScript attribute. When you add a control to the Documents tab it is disabled by default – you must use this attribute to specify the conditions under which the control should be enabled. The EnabledScript attribute should specify (or call) a JavaScript function that returns a Boolean value – true to enable the control, false to disable it. In this case, we want the button to be enabled when the user has selected a single document in the document library list view. The JavaScript client-side object model for SharePoint includes a class named SP.ListOperation.Selection for just this kind of eventuality. We can use the getSelectedItems method to return a collection of the items selected in the list view, then check that the length of the collection is equal to 1.

Note: In this example I've added all my JavaScript logic directly to the CommandUIHandler element. As your JavaScript logic grows larger and more complex, a better option would be to deploy a standalone JavaScript file. Yaroslav Pentarskyy describes this approach in this blog post.

Next, the CommandAction attribute specifies the JavaScript function we want to call when our button is clicked. The getSelectedItems method returns a Dictionary of key-value pairs. The value of each dictionary entry is an object with two attributes – id and fsObjType. The id attribute represents the integer ID of the list item, while the fsObjType attribute represents the type of list item object – 0 for a document or a list item, 1 for a folder. While this doesn't give us a great deal of information about the selected item, the integer ID gives us enough information to submit a query for additional document metadata, should we so wish. In this case, as a proof of concept, we simply display a notification containing the document ID when the user clicks our button.

Here's the button in its default disabled state:

When we select a document, the button is enabled:

When we click the button, a notification displays the integer ID of the selected document:

And that concludes today's task. Next time I plan to cover how to extend this to do something useful with the selected document. The contents of the feature element are shown below in their entirety for reference.

<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
  <CustomAction Id="Jason.SP.GSD" 
                Location="CommandUI.Ribbon" 
                Sequence="11" 
                RegistrationType="List" 
                RegistrationId="101">
    <CommandUIExtension>
      <CommandUIDefinitions>
        <CommandUIDefinition Location="Ribbon.Documents.Scaling._children">
          <MaxSize Id=" Jason.SP.GSD.JasonsActions.MaxSize" 
                   Sequence="11" 
                   GroupId="Jason.SP.GSD.JasonsActions" 
                   Size="LargeLarge" />
        </CommandUIDefinition>
        <CommandUIDefinition Location="Ribbon.Documents.Groups._children">
          <Group Id=" Jason.SP.GSD.JasonsActions" 
                 Sequence="41" 
                 Title="Jason's Actions" 
                 Description="Contains custom document actions" 
                 Template="Ribbon.Templates.Flexible2">
            <Controls Id="Jason.SP.GSD.JasonsActions.Controls">
              <Button Id=" Jason.SP.GSD.JasonsActions.GetButton" 
                      Sequence="1" 
                      Image32by32="/SiteCollectionImages/RibbonIcons/ThumbsUp.PNG" 
                      LabelText="Get Selection Details" 
                      Description="Gets the details of the selected document" 
                      TemplateAlias="o1" 
                      Command=" Jason.SP.GSD.GetCmd" />
            </Controls>
          </Group>
        </CommandUIDefinition>
      </CommandUIDefinitions>
      <CommandUIHandlers>
        <CommandUIHandler Command="Jason.SP.GSD.GetCmd" 
                          EnabledScript="javascript:
                            SP.ListOperation.Selection.getSelectedItems().length == 1;" 
                          CommandAction="javascript:
                            var selectedItems = 
                              SP.ListOperation.Selection.getSelectedItems();
                            var item = selectedItems[0];
                            var itemID = item['id'];
                            if (item['fsObjType'] == 0) {
                              SP.UI.Notify.addNotification(String.format(
                                'Document selected: ID={0}', itemID));
                            }
                            else {
                              SP.UI.Notify.addNotification(String.format(
                                'Folder selected: ID={0}', itemID));
                            } 
        "/>
      </CommandUIHandlers>
    </CommandUIExtension>
  </CustomAction>
</Elements>

Kinect SDK: Gesture Recognition Pt III

2011-08-26T12:30:00.000+01:00

Introduction

In my previous blog post I discussed the development of a robust and extensible PostureRecognizer class. The class is used to recognize the start of a gesture. As a reminder, my high-level approach to the gesture recognition process is as follows:

Detect whether the user is moving or stationary.
Detect the start of a gesture (a posture).
Capture the gesture.
Detect the end of a gesture (a posture).
Identify the gesture.

This blog post will focus on capturing the gesture that occurs once the starting posture has been identified.

Implementation

The UI is unchanged from my previous post, with the exception of another Canvas being added to draw the captured gesture on.

The StreamManager constructor creates an instance of the PostureRecognizer class and registers an event handler for the PostureDetected event. It then registers the path where the gesture data will be saved, via a stream. An instance of the GestureRecognizer class is then created, and the name of the canvas to draw the captured gesture on is passed to the DrawGesture method.

        public StreamManager(Canvas canvas)
        {
            this.PostureRecognizer = new PostureRecognizer();
            this.PostureRecognizer.PostureDetected += 
               new Action<Posture>(OnPostureDetected);
            
            this.gestureCanvas = canvas;
            this.path = Path.Combine(Environment.CurrentDirectory, @"data\gestures.dat");
            this.stream = File.Open(this.path, FileMode.OpenOrCreate);
            this.gestureDetector = new GestureRecognizer(this.stream);
            this.gestureDetector.DrawGesture(this.gestureCanvas, Colors.Red);
        }

The StartGestureCapture and EndGestureCapture methods are shown below. The StartGestureCapture method invokes the StartGestureCapture method in the GestureRecognizer class, and if a gesture is already being captured, stops the capture. The EndGestureCapture method invokes the EndGestureCapture method in the GestureRecognizer class, and invokes the SaveGesture method in the GestureRecognizer class, provided that a gesture is being captured.

        private void StartGestureCapture()
        {
            this.gestureDetector.GestureName = this.GestureName;
            if (this.gestureDetector.IsRecordingGesture)
            {
                this.gestureDetector.EndGestureCapture();
                return;
            }
            this.gestureDetector.StartGestureCapture();
        }
        private void EndGestureCapture()
        {
            if (this.gestureDetector.IsRecordingGesture)
            {
                this.gestureDetector.EndGestureCapture();
                this.gestureDetector.SaveGesture(stream);
            }
        }

The GetSkeletonStream method is as in the previous post, and invokes the TrackPostures method in the PostureRecognizer class. If a posture is identified, the RaisePostureDetected method is invoked which sets the CurrentPosture property to the identified posture, and invokes the PostureDetected event. The handler for the event is the OnPostureDetected method in the StreamManager class and was registered in the StreamManager constructor.

        public event Action<Posture> PostureDetected;
        private void RaisePostureDetected(Posture posture)
        {
            if (this.currentPosture != posture)
            {
                this.CurrentPosture = posture;
            }
            if (this.PostureDetected != null)
            {
                this.PostureDetected(posture);
            }
        }

The OnPostureDetected method is shown below. If a gesture is not being captured, it invokes the StartGestureCapture method, and stores the identified posture in the startPosture variable. If a gesture is being captured and the posture is different to the posture stored in the startPosture variable, the EndGestureCapture method is invoked. The detected Joints are then enumerated, and the Add method in the GestureRecognizer class is invoked if the gesture involves the right hand.

        private void OnPostureDetected(Posture posture)
        {
            if (this.gestureDetector.IsRecordingGesture == true)
            {
                if ((posture != Posture.None) &&
                    (posture != startPosture))
                {
                    this.EndGestureCapture();
                }
            }
            else
            {
                this.StartGestureCapture();
                this.startPosture = posture;
            }
            foreach (Joint joint in this.skeleton.Joints)
            {
                if (joint.Position.W < 0.9f || 
                    joint.TrackingState != JointTrackingState.Tracked)
                {
                    continue;
                }
                if (joint.ID == JointID.HandRight)
                {
                    this.gestureDetector.Add(joint.Position, 
                                             this.KinectRuntime.SkeletonEngine);
                }
            }
        }

The GestureData class is used to model the gesture data. It contains the a Position and the Time the Position was captured.

    public class GestureData
    {
        public Vector Position { get; set; }
        public DateTime Time { get; set; }
    }

The GestureRecognizer class is shown below. Gestures are stored in a collection named gestures, of type List<GestureData>. An instance of the TemplateRecognizer class is also created, which will use template matching to perform gesture identification. The bulk of the work in this class is performed in the Add method. It stores the passed in Vector as a GestureData object, and adds that to the gestures collection. It then draws the position as a small ellipse on the Canvas contained in the displayCanvas variable. Then, if there is more data in the gestures collection than the value of WindowSize, data is removed from both the start of the collection and the displayCanvas. The position data is then added to an instance of the Gesture class. The StartGestureCapture, EndGestureCapture, and SaveGesture methods largely invoke methods in the TemplateRecognizer class.

    public class GestureRecognizer
    {
        private Gesture gesture;
        private string gestureName; 
        private readonly List<GestureData> gestures = new List<GestureData>();
        private readonly TemplateRecognizer.TemplateRecognizer templateRecognizer;
        private readonly int windowSize;
        private DateTime lastGestureDate = DateTime.Now;
        private Canvas displayCanvas;
        private Colour displayColour;
        protected List<GestureData> Gestures
        {
            get { return this.gestures; }
        }
        public string GestureName
        {
            get { return this.gestureName; }
            set { this.gestureName = value; }
        }
        public bool IsRecordingGesture
        {
            get { return this.gesture != null; }
        }
        public TemplateRecognizer.TemplateRecognizer TemplateRecognizer
        {
            get { return this.templateRecognizer; }
        }
        public int WindowSize
        {
            get { return this.windowSize; }
        }
        public GestureRecognizer(Stream stream, int windowSize = 100)
        {
            this.templateRecognizer = new TemplateRecognizer.TemplateRecognizer(stream);
            this.windowSize = windowSize;
        }
        public GestureRecognizer(string gestureName, Stream stream, int windowSize = 100)
        {
            this.gestureName = gestureName;
            this.templateRecognizer = new TemplateRecognizer.TemplateRecognizer(stream);
            this.windowSize = windowSize;
        }
        public void Add(Vector position, SkeletonEngine engine)
        {
            GestureData newGesture = new GestureData
            {
                Position = position,
                Time = DateTime.Now
            };
            this.gestures.Add(newGesture);
            if (this.displayCanvas != null)
            {
                Ellipse ellipse = new Ellipse
                {
                    HorizontalAlignment = HorizontalAlignment.Left,
                    VerticalAlignment = VerticalAlignment.Top,
                    Height = 8,
                    Width = 8,
                    StrokeThickness = 8,
                    Stroke = new SolidColorBrush(this.displayColour),
                    StrokeLineJoin = PenLineJoin.Round
                };
                float x, y;
                engine.SkeletonToDepthImage(position, out x, out y);
                x = (float)(x * this.displayCanvas.ActualWidth);
                y = (float)(y * this.displayCanvas.ActualHeight);
                Canvas.SetLeft(ellipse, x - ellipse.Width / 2);
                Canvas.SetTop(ellipse, y - ellipse.Height / 2);
                this.displayCanvas.Children.Add(ellipse);
            }
            if (this.gestures.Count > this.WindowSize)
            {
                GestureData gestureToRemove = this.gestures[0];
                if (this.displayCanvas != null)
                {
                    this.displayCanvas.Children.RemoveAt(0);
                }
                this.gestures.Remove(gestureToRemove);
            }
            if (this.gesture != null)
            {
                Vector2 vector = new Vector2
                {
                    X = position.X,
                    Y = position.Y
                };
                this.gesture.Points.Add(vector);
            }
        }
        public void DrawGesture(Canvas canvas, Color colour)
        {
            this.displayCanvas = canvas;
            this.displayColour = colour;
        }
        public void StartGestureCapture()
        {
            this.gesture = new Gesture(this.WindowSize);
        }
        public void EndGestureCapture()
        {
            this.templateRecognizer.Add(gesture);
            this.gesture = null;
        }
        public void SaveGesture(Stream stream)
        {
            this.templateRecognizer.Save(stream);
        }
    }

The Gesture class stores all the positions of the captured gesture in a collection called points, of type List<Vector>.

    [Serializable]
    public class Gesture
    {
        private List<Vector> points;
        private readonly int samplesCount;
        public List<Vector> Points
        {
            get { return this.points; }
            set { this.points = value; }
        }
        public Gesture(int samplesCount)
        {
            this.samplesCount = samplesCount;
            this.points = new List<Vector>();
        }
    }

The TemplateRecognizer class will perform the bulk of the processing required for gesture identification, which will be covered in a future blog post. At the moment the class simply handles serialization and deserialization of the gesture data, by using a BinaryFormatter. The constructor deserializes the gesture data if it exists, while the Save method serializes the gesture data to a file.

    public class TemplateRecognizer
    {
        private readonly List<Gesture> gestures;
        public TemplateRecognizer(Stream stream)
        {
            if (stream == null || stream.Length == 0)
            {
                this.gestures = new List<Gesture>();
                return;
            }
            BinaryFormatter formatter = new BinaryFormatter();
            this.gestures = (List<Gesture>)formatter.Deserialize(stream);
        }
        public void Add(Gesture gesture)
        {
            this.gestures.Add(gesture);
        }
        public void Save(Stream stream)
        {
            BinaryFormatter formatter = new BinaryFormatter();
            formatter.Serialize(stream, this.gestures);
        }
    }

The application is shown below. It determines whether the user is moving or stationary, and recognizes the start of a gesture – in this case, a posture. Once a posture is identified (currently hard coded to RightHello) the gesture points are stored in a collection and drawn on the canvas. When a different posture is identified the gesture capture stops.

<

Conclusion

The Kinect for Windows SDK beta from Microsoft Research is a starter kit for application developers. It allows access to the Kinect sensor, and experimentation with its features. My gesture recognition process now determines whether the user is moving or stationary, recognizes a posture and then captures the gesture that follows, before saving it to a file. The next step of the process will be to scale the gesture data to a reference, before saving it to a file.

Kinect SDK: Gesture Recognition Pt II

2011-08-11T14:05:00.001+01:00

Introduction

In my previous blog post I outlined the approach I’d be taking to gesture recognition, and discussed how I’d detect whether the user is moving or stationary. As a reminder, my high-level approach to the gesture recognition process is as follows:

Detect whether the user is moving or stationary.
Detect the start of a gesture (a posture).
Capture the gesture.
Detect the end of a gesture (a posture).
Identify the gesture.

This blog post will focus on detecting the start of a gesture – in this case, a posture. I’d previously written a PoseRecognition class which I’ve since refactored to be more robust and extensible. The refactored PostureRecognizer class recognises three poses – the left hand waving hello, the right hand waving hello, and both hands being clasped together. New poses can easily be added to the class.

Implementation

The XAML for the UI is shown below. An Image shows the video stream from the sensor, with a Canvas being used to overlay the skeleton of the user on the video stream. A Slider controls the elevation of the sensor via bindings to a Camera class, thus eliminating some of the references to the UI from the code-behind file. TextBlocks are bound to IsSkeletonTracking, IsStable, and PostureRecognizer.CurrentPosture, to show if the skeleton is being tracked, if it is stable, and the current posture, respectively.

<Window x:Class="KinectDemo.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:conv="clr-namespace:KinectDemo.Converters"
        Title="Gesture Recognition" ResizeMode="NoResize" SizeToContent="WidthAndHeight"
        Loaded="Window_Loaded" Closed="Window_Closed">
    <Grid>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="Auto" />
            <ColumnDefinition Width="300" />
        </Grid.ColumnDefinitions>
        <StackPanel Grid.Column="0">
            <TextBlock HorizontalAlignment="Center"
                       Text="Tracking..." />
            <Viewbox>
                <Grid ClipToBounds="True">
                    <Image Height="300"
                           Margin="10,0,10,10"
                           Source="{Binding ColourBitmap}"
                           Width="400" />
                    <Canvas x:Name="skeletonCanvas" />
                </Grid>
            </Viewbox>
        </StackPanel>
        <StackPanel Grid.Column="1">
            <GroupBox Header="Motor Control"
                      Height="100"
                      VerticalAlignment="Top"
                      Width="290">
                    <Slider x:Name="elevation"
                            AutoToolTipPlacement="BottomRight"
                            IsSnapToTickEnabled="True"
                            LargeChange="10"
                            Maximum="{Binding ElevationMaximum}"
                            Minimum="{Binding ElevationMinimum}"
                            HorizontalAlignment="Center"
                            Orientation="Vertical"
                            SmallChange="3"
                            TickFrequency="3"
                            Value="{Binding Path=ElevationAngle, Mode=TwoWay}" />
            </GroupBox>
            <GroupBox Header="Information"
                      Height="150"
                      VerticalAlignment="Top"
                      Width="290">
                <GroupBox.Resources>
                    <conv:BooleanToStringConverter x:Key="boolStr" />
                    <conv:PostureToStringConverter x:Key="postureStr" />
                </GroupBox.Resources>
                <StackPanel>
                    <StackPanel Orientation="Horizontal" 
                                Margin="10">
                        <TextBlock Text="Frame rate: " />
                        <TextBlock Text="{Binding FramesPerSecond}"
                                   VerticalAlignment="Top"
                                   Width="50" />
                    </StackPanel>
                    <StackPanel Margin="10,0,0,0" 
                                Orientation="Horizontal">
                        <TextBlock Text="Tracking skeleton: " />
                        <TextBlock Text="{Binding IsSkeletonTracking, 
                                          Converter={StaticResource boolStr}}"
                                   VerticalAlignment="Top"
                                   Width="30" />
                    </StackPanel>
                    <StackPanel Orientation="Horizontal"
                                Margin="10">
                        <TextBlock Text="Stable: " />
                        <TextBlock Text="{Binding Path=IsStable, 
                                                  Converter={StaticResource boolStr}}"
                                   VerticalAlignment="Top"
                                   Width="30" />
                    </StackPanel>
                    <StackPanel Orientation="Horizontal" 
                                Margin="10,0,0,0">
                        <TextBlock Text="Current posture: " />
                        <TextBlock Text="{Binding Path=PostureRecognizer.CurrentPosture, 
                                                  Converter={StaticResource postureStr}}"
                                   VerticalAlignment="Top"
                                   Width="160" />
                    </StackPanel>
                </StackPanel>
            </GroupBox>
        </StackPanel>
    </Grid>
</Window>

The StreamManager class (inside my KinectManager library) contains a property to access to the PostureRecognizer class, which is initialized in the StreamManager constructor.

        public PostureRecognizer PostureRecognizer { get; private set; }

The SkeletonFrameReady event handler hooks into a method in the StreamManager class called GetSkeletonStream. It retrieves a frame of skeleton data and processes it as follows: e.SkeletonFrame.Skeletons is an array of SkeletonData structures, each of which contains the data for a single skeleton. If the TrackingState field of the SkeletonData structure indicates that the skeleton is being tracked, the IsSkeletonTracking property is set to true, and the position of the skeleton is added to a collection in the BaryCenter class. Then the IsStable method of the BaryCenter class is invoked to determine if the user is stable or not. If the user is stable, the TrackPostures method from the PostureRecognizer class is invoked. Finally, the IsStable property is updated.

        public void GetSkeletonStream(SkeletonFrameReadyEventArgs e)
        {
            bool stable = false;
            foreach (var skeleton in e.SkeletonFrame.Skeletons)
            {
                if (skeleton.TrackingState == SkeletonTrackingState.Tracked)
                {
                    this.IsSkeletonTracking = true;
                    this.baryCenter.Add(skeleton.Position, skeleton.TrackingID);
                    stable = this.baryCenter.IsStable(skeleton.TrackingID) ? true : false;
                    if (stable)
                    {
                        this.PostureRecognizer.TrackPostures(skeleton);
                    }
                }
            }
            this.IsStable = stable;
        }

The Posture enumeration, used by the PostureRecognizer class, is shown below.

    public enum Posture
    {
        None,
        HandsClasped,
        LeftHello,
        RightHello
    }

The PostureRecognizer class defines a property called CurrentPosture, along with a backing store variable, and two constants.

        private const float Epsilon = 0.1f;
        private const float MaxRange = 0.25f;
        private Posture currentPosture = Posture.None;
        public Posture CurrentPosture
        {
            get { return this.currentPosture; }
            set
            {
                this.currentPosture = value;
                this.OnPropertyChanged("CurrentPosture");
            }     
        }

The PostureRecognizer.TrackPostures method is shown below. It enumerates the Joints in the skeleton and gets the positions of the head, left and right hands and stores them as Vectors. It then calls two helper methods to determine if the user’s hands are clapsed or if the user is waving hello with their left or right hand. If a posture is identified, the RaisePostureDetected method is invoked. If a posture is not identified, the CurrentPosture property is set to Posture.None.

        public void TrackPostures(SkeletonData skeleton)
        {
            if (skeleton.TrackingState != SkeletonTrackingState.Tracked)
            {
                return;
            }
            Vector? headPosition = null;
            Vector? leftHandPosition = null;
            Vector? rightHandPosition = null;
            foreach (Joint joint in skeleton.Joints)
            {
                if (joint.Position.W < 0.8f || 
                    joint.TrackingState != JointTrackingState.Tracked)
                {
                    continue;
                }
                switch (joint.ID)
                {
                    case JointID.Head:
                        headPosition = joint.Position;
                        break;
                    case JointID.HandLeft:
                        leftHandPosition = joint.Position;
                        break;
                    case JointID.HandRight:
                        rightHandPosition = joint.Position;
                        break;
                }
            }
            if (this.AreHandsClasped(rightHandPosition, leftHandPosition))
            {
                this.RaisePostureDetected(Posture.HandsClasped);
                return;
            }
            if (this.IsHello(headPosition, leftHandPosition))
            {
                this.RaisePostureDetected(Posture.LeftHello);
                return;
            }
            if (this.IsHello(headPosition, rightHandPosition))
            {
                this.RaisePostureDetected(Posture.RightHello);
                return;
            }
            this.CurrentPosture = Posture.None;
        }

The IsHello and AreHandsClasped helper methods are shown below. They both examine the Vector parameters to see if they have values, and if they do, perform simple arithmetic to determine whether the hello posture has been made, or whether the user’s hands are clasped together. In particular, the AreHandsClasped method uses the Vector extension methods I developed in my previous blog post.

        private bool IsHello(Vector? headPosition, Vector? handPosition)
        {
            if (!headPosition.HasValue || !handPosition.HasValue)
                return false;
            if (Math.Abs(handPosition.Value.X - headPosition.Value.X) < MaxRange)
                return false;
            if (Math.Abs(handPosition.Value.Y - headPosition.Value.Y) > MaxRange)
                return false;
            if (Math.Abs(handPosition.Value.Z - headPosition.Value.Z) > MaxRange)
                return false;
            return true;
        }
        private bool AreHandsClasped(Vector? leftHandPosition, Vector? rightHandPosition)
        {
            if (!leftHandPosition.HasValue || !rightHandPosition.HasValue)
                return false;
            Vector result = leftHandPosition.Value.Subtract(rightHandPosition.Value);
            float distance = result.Length();
            if (distance > Epsilon)
                return false;
            return true;
        }

The RaisePostureDetected method is shown below. It simply updates the CurrentPosture property to the identified posture, providing that the two are not identical. The method name could be considered curious, as the method is not raising an event. However, the point of recognizing a posture is that it will be the start or end point of a gesture. Therefore, this method will eventually fire a PostureDetected event that will capture a gesture if a start posture has been identified, or stop the gesture capture if an end posture has been identified. I will return to this topic in my next blog post.

        private void RaisePostureDetected(Posture posture)
        {
            if (this.currentPosture != posture)
            {
                this.CurrentPosture = posture;
            }
        }

The UI uses the PostureToStringConverter class (code not shown) to convert the Posture enumeration to it’s string representation for display on the UI.

The application is shown below. It uses skeleton tracking to determine whether the user is moving or stationary, and identifies the user’s pose if they are stationary.

Conclusion

The Kinect for Windows SDK beta from Microsoft Research is a starter kit for application developers. It allows access to the Kinect sensor, and experimentation with its features. My gesture recognition process now determines whether the user is moving or stationary, and recognizes the start of a gesture – in this case, a posture. The next step in the process will be to capture the gesture that occurs between the starting posture and the ending posture.

Surface 2.0 Hands On Labs

2011-08-08T21:19:00.007+01:00

When Microsoft Surface 2.0 was released, Microsoft also included a set of hands on labs to help developers get started developing applications for the Surface platform. The labs are meant to augment Designing and Developing Microsoft Surface Applications online course.

There are 11 labs available when you download the Microsoft Surface 2.0 Hands on Labs.zip file. Each lab covers a distinct element of developing for Microsoft Surface 2.0:

Introduction Lab

ScatterView lab

Vision Based Object Recognition lab

Using the Microsoft Surface Input Simulator lab

Drag and Drop APIs lab

Using the Library Container Controls lab

Windows 7 Touch Enabled Applications lab

Creating a 360 Degree User Interface lab

Stylizing a Surface Application lab

Developing for Direct Touch lab

Developing for Multiple Users and Objects lab

Each lab contains a Lab document and lab solution files. Some, but not all of the labs also contain lab starter files. You can download the hands on labs here.

I ran through each of these labs with an eye to call out anything exceptional in them that can provide you pointers in how to develop Surface 2.0 applications. The rest of today’s post will speak to the Introduction Lab, which I’ll simply summarize.

This lab walks you through how to create a Surface application solution in Visual Studio 2010, then has you review the C# code that is generated by the Surface 2.0 Visual Studio templates. The third exercise shows you how to edit the XAML and C# files in the solution, and the fourth exercise introduces some of the commonly used Surface controls.

Though you can use controls as you see fit in your applications, the Surface design guidance recommends that you only use controls as necessary. You should never use controls in place of people directly touching your applications’ content. For example, if you include images in your application, do not require people to touch a SurfaceButton control instead of directly touching an image displayed by the application.

In my next post I’ll go into Lab 2, which is about the ScatterView control.

Kinect SDK: Gesture Recognition Pt 1

2011-07-29T14:04:00.001+01:00

Introduction

Previous blog posts have looked at a number of topics including simple gesture recognition, skeleton tracking, pose recognition, and smoothing skeleton data. It’s now time to link these topics together in order to produce a robust and extensible gesture recognizer, that can be used in different NUI (natural user interface) applications.

My high-level approach to the gesture recognition process will be as follows:

Detect whether the user is moving or stationary.
Detect the start of a gesture (a posture).
Capture the gesture.
Detect the end of a gesture (a posture).
Identify the gesture.

A gesture can be thought of as a sequence of points. The coordinates of these points will define a distance to the sensor and a gesture recognizer will have to recognize a gesture regardless of the distance to the sensor. Therefore, it will be necessary to scale gestures to a common reference. It will then be possible to robustly identify gestures captured at any distance from the sensor.

There are a large number of algorithmic solutions for gesture recognition, and I will write more about this in a future blog post. The focus of this post is on detecting whether the user is moving or stationary. This can be undertaken by examining the center of mass of the user.

The center of mass is the mean location of all the mass in a system, and is also known as the barycenter. The common definition of barycenter comes from astrophysics, where it is the center of mass where two or more celestial bodies orbit each other. The barycenter of a shape is the intersection of all straight lines that divide the shape into two parts of equal moment about the line. Therefore, it can be thought of as the mean of all points of the shape.

The application documented here uses a barycenter in order to determine if the user is moving (stable) or stationary (not stable).

Implementation

The XAML for the UI is shown below. An Image shows the video stream from the sensor, with a Canvas being used to overlay the skeleton of the user on the video stream. A Slider now controls the elevation of the sensor via bindings to a Camera class, thus eliminating some of the references to the UI from the code-behind file. TextBlocks are bound to IsSkeletonTracking and IsStable, to show if the skeleton is being tracked, and if it is stable, respectively.

<Window x:Class="KinectDemo.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:conv="clr-namespace:KinectDemo.Converters"
        Title="Gesture Recognition" ResizeMode="NoResize" SizeToContent="WidthAndHeight"
        Loaded="Window_Loaded" Closed="Window_Closed">
    <Grid>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="Auto" />
            <ColumnDefinition Width="300" />
        </Grid.ColumnDefinitions>
        <StackPanel Grid.Column="0">
            <TextBlock HorizontalAlignment="Center"
                       Text="Tracking..." />
            <Viewbox>
                <Grid ClipToBounds="True">
                    <Image Height="300"
                           Margin="10,0,10,10"
                           Source="{Binding ColourBitmap}"
                           Width="400" />
                    <Canvas x:Name="skeletonCanvas" />
                </Grid>
            </Viewbox>
        </StackPanel>
        <StackPanel Grid.Column="1">
            <GroupBox Header="Motor Control"
                      Height="100"
                      VerticalAlignment="Top"
                      Width="290">
                    <Slider x:Name="elevation"
                            AutoToolTipPlacement="BottomRight"
                            IsSnapToTickEnabled="True"
                            LargeChange="10"
                            Maximum="{Binding ElevationMaximum}"
                            Minimum="{Binding ElevationMinimum}"
                            HorizontalAlignment="Center"
                            Orientation="Vertical"
                            SmallChange="3"
                            TickFrequency="3"
                            Value="{Binding Path=ElevationAngle, Mode=TwoWay}" />
            </GroupBox>
            <GroupBox Header="Information"
                      Height="200"
                      VerticalAlignment="Top"
                      Width="290">
                <GroupBox.Resources>
                    <conv:BooleanToStringConverter x:Key="boolStr" />
                </GroupBox.Resources>
                <StackPanel>
                    <StackPanel Orientation="Horizontal" 
                                Margin="10">
                        <TextBlock Text="Frame rate: " />
                        <TextBlock Text="{Binding FramesPerSecond}"
                                   VerticalAlignment="Top"
                                   Width="50" />
                    </StackPanel>
                    <StackPanel Margin="10,0,0,0" 
                                Orientation="Horizontal">
                        <TextBlock Text="Tracking skeleton: " />
                        <TextBlock Text="{Binding IsSkeletonTracking, 
                                          Converter={StaticResource boolStr}}"
                                   VerticalAlignment="Top"
                                   Width="30" />
                    </StackPanel>
                    <StackPanel Orientation="Horizontal"
                                Margin="10">
                        <TextBlock Text="Stable: " />
                        <TextBlock Text="{Binding Path=IsStable, 
                                          Converter={StaticResource boolStr}}"
                                   VerticalAlignment="Top"
                                   Width="30" />
                    </StackPanel>
                </StackPanel>
            </GroupBox>
        </StackPanel>
    </Grid>
</Window>

The StreamManager class (inside my KinectManager library) contains properties and backing store for IsSkeletonTracking and IsStable. It also contains an instance of the BaryCenter class.

        private bool? isStable = null;
        private bool isSkeletonTracking;
        private readonly BaryCenter baryCenter = new BaryCenter();
        public bool IsSkeletonTracking
        {
            get { return this.isSkeletonTracking; }
            private set
            {
                this.isSkeletonTracking = value;
                this.OnPropertyChanged("IsSkeletonTracking");
                if (this.isSkeletonTracking == false)
                {
                    this.IsStable = null;
                }
            }
        }
        
        public bool? IsStable
        {
            get { return this.isStable; }
            private set
            {
                this.isStable = value;
                this.OnPropertyChanged("IsStable");
            }
        }

        public void GetSkeletonStream(SkeletonFrameReadyEventArgs e)
        {
            bool stable = false;
            foreach (var skeleton in e.SkeletonFrame.Skeletons)
            {
                if (skeleton.TrackingState == SkeletonTrackingState.Tracked)
                {
                    this.IsSkeletonTracking = true;
                    this.baryCenter.Add(skeleton.Position, skeleton.TrackingID);
                    stable = this.baryCenter.IsStable(skeleton.TrackingID) ? true : false;
                }
            }
            this.IsStable = stable;
        }

The Vector class in the Microsoft.Research.Kinect.Nui namespace is currently lacking many useful vector operations, including basic vector arithmetic. Therefore I defined extension methods to calculate the length of a vector, and to subtract one vector from another. The extension methods are currently defined in a Helper class. Extension methods enable you to “add” methods to existing types without creating a new derived type, or otherwise modifying the original type. There is no difference between calling an extension method and the methods that are actually defined in a type.

        public static float Length(this Vector vector)
        {
            return (float)Math.Sqrt(vector.X * vector.X + 
                                    vector.Y * vector.Y + 
                                    vector.Z * vector.Z);
        }
        public static Vector Subtract(this Vector left, Vector right)
        {
            return new Vector
            {
                X = left.X - right.X,
                Y = left.Y - right.Y,
                Z = left.Z - right.Z,
                W = left.W - right.W
            };
        }

The Add method and IsStable method of the BaryCenter class are shown below. The Add method stores skeleton positions in a circular queue of type Dictionary, where each entry in the Dictionary is an int and a List of Vectors. The IsStable method returns a boolean value indicating whether the user is stable or not. If there are not enough skeleton positions stored in the Dictionary it returns false. Otherwise it subtracts each of the stored skeleton position vectors from the latest skeleton position vector, and gets the vector length of the resulting subtraction. If the vector length is greater than a threshold it returns false. Otherwise it returns true.

        public void Add(Vector position, int trackingID)
        {
            if (!this.positions.ContainsKey(trackingID))
            {
                this.positions.Add(trackingID, new List<Vector>());
            }
            this.positions[trackingID].Add(position);
            if (this.positions[trackingID].Count > this.windowSize)
            {
                this.positions[trackingID].RemoveAt(0);
            }
        }
        public bool IsStable(int trackingID)
        {
            List<Vector> currentPositions = positions[trackingID];
            if (currentPositions.Count != this.windowSize)
            {
                return false;
            }
            Vector current = currentPositions[currentPositions.Count - 1];
            for (int i = 0; i < currentPositions.Count - 2; i++)
            {
                Vector result = currentPositions[i].Subtract(current);
                float length = result.Length();
                
                if (length > this.Threshold)
                {
                    return false;
                }
            }
            return true;
        }

The application is shown below. It uses skeleton tracking to derive whether the user is moving or stationary, and indicates this on the UI.

Conclusion

The Kinect for Windows SDK beta from Microsoft Research is a starter kit for application developers. It allows access to the Kinect sensor, and experimentation with its features. The first part of my gesture recognition process is to determine whether the user is moving or stationary, and is performed with the BaryCenter class. Coupled with pose recognition, this will produce a robust indication of whether a gesture is being made or not.

Enterprise Library and Azure – What do you want to see?

2011-07-25T18:19:00.001+01:00

I'm currently working with the team in Microsoft's patterns & practices group who are in the early stages of a project to create a Windows Azure Integration Pack for Enterprise Library. We're currently looking for feedback about the proposed content (or indeed suggestions for content we haven't thought of), so if you're developing applications for Windows Azure or using Enterprise Library in a current project, go and take a look at what's on the list, and cast your vote for what you'd like to see included here.

Kinect SDK: Smoothing Skeleton Data

2011-07-21T12:59:00.001+01:00

Introduction

As previously noted, the Kinect sensor does not have sufficient resolution to ensure consistent accuracy of the skeleton tracking data over time. This problem manifests itself as the data seeming to vibrate around their positions. However, the Kinect SDK provides an algorithm for filtering and smoothing incoming data from the sensor, the code for which can be seen below. The smoothing algorithm parameters can be manipulated in order to provide the required level of filtering and smoothing for your desired user experience.

            this.runtime.SkeletonEngine.TransformSmooth = true;
            var parameters = new TransformSmoothParameters
            {
                Smoothing = 1.0f,
                Correction = 0.1f,
                Prediction = 0.1f,
                JitterRadius = 0.05f,
                MaxDeviationRadius = 0.05f
            };
            this.runtime.SkeletonEngine.SmoothParameters = parameters;

A common question I’ve seen online is how does the smoothing work, and what do the various parameters do. This post attempts to answer these questions.

Smoothing and Filtering Data

Filtering is the most commonly used signal processing technique. Filters are usually used to remove or attenuate an undesired portion of a signal’s spectrum while enhancing the desired portions of the signal. The filter made available by the Kinect SDK is based on the Holt double exponential smoothing method, commonly used for statistical analysis of economic data. This algorithm provides smoothing with less latency than many other smoothing filter algorithms.

The exponential function is used to model the relationship in which a change in the independent variable, x, gives the same proportional change in the dependent variable, y.

The double exponential function is a constant raised to the power of an exponential function.

The simplest way to smooth a time series (data points measured at uniformly spaced time intervals) is to calculate a simple moving average, with the output being the mean of the last x data points. Exponential smoothing builds on this by assigning exponentially decreasing weights as the data points get older; one or more smoothing parameters determine the weights assigned to the data points. The smoothed data becomes a simple weighted average of the previous data point, and the previous smoothed data point. However, “good smoothing” will not be achieved until several data points have been averaged together. In addition, exponential smoothing is problematic when there is a trend in the data.

Double exponential smoothing can be defined as single exponential smoothing applied to an already exponentially smoothed time series. It performs two smoothing passes. In the first pass the smoothed data is adjusted for the trend of the previous period, by adding it to the last smoothed value. In the second pass the trend is updated, which is expressed as the difference between the last two values. Holt’s double exponential smoothing algorithm simply removes the need for a second pass of the data by smoothing the trend values directly; a number of smoothing parameters determine the weights assigned to the data points.

The algorithm parameters that the Kinect SDK allow you to set are shown in the table below, along with their default values.

Parameter	Description	Default Value	Comments
Smoothing	Specifies the amount of smoothing.	0.5	Higher values correspond to more smoothing and a value of 0 causes the raw data to be returned. Increasing smoothing tends to increase latency. Values must be in the range [0, 1.0].
Correction	Specifies the amount of correction.	0.5	Lower values are slower to correct towards the raw data and appear smoother, while higher values correct toward the raw data more quickly. Values must be in the range [0, 1.0].
Prediction	Specifies the number of predicted frames.	0.5
Jitter Radius	Specifies the jitter-reduction radius, in meters.	0.05	The default value of 0.05 represents 5cm. Any jitter beyond the radius is clamped to the radius.
Maximum Deviation Radius	Specifies the maximum radius that filter positions can deviate from raw data, in meters.	0.04	Filtered values that would exceed the radius from the raw data are clamped at this distance, in the direction of the filtered value.

There is no set of “best” values to use for these parameters. Experimentation is required on an application-by-application basis in order to provide the required level of filtering and smoothing for your desired user experience.

Conclusion

The Kinect for Windows SDK beta from Microsoft Research is a starter kit for application developers. It allows access to the Kinect sensor, and experimentation with its features. The Kinect SDK provides access to Holt’s double exponential smoothing algorithm, which produces smoothing with less latency than many other smoothing filter algorithms. The algorithm parameters can be manipulated in order to provide the required level of filtering and smoothing for your desired user experience.

Kinect SDK: Simple Gesture Recognition

2011-07-20T09:25:00.000+01:00

Introduction

Gesture recognition has long been a research area within computer science, and has seen an increased focus in the last decade due to the development of different devices including smart phones and Microsoft Surface. It’s aim is to allow people to interface with a device and interact naturally without any mechanical devices. Multi-touch devices use gestures to perform various actions; for instance a pinch gesture is commonly used to scale content such as images. In this post I will outline the development of a small application that performs simple gesture recognition in order to scale an image using the user’s hands. The application uses skeleton tracking to recognise the user’s hands, then places an image of the Eye of Sauron in the center of the user’s hands. As the user moves their hands apart the size of the image increases, and as the user closes their hands the size of the image decreases.

Implementation

The XAML for the UI of the application is shown below. A Canvas contains an Image that is used to display the Eye of Sauron. The Image uses a TranslateTransform to control the location of the eye, and a ScaleTransform to control the size of the eye. The properties of both RenderTransforms are set using bindings, as is the Visibility property of the Image.

<Window x:Class="KinectDemo.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="Eye of Sauron" ResizeMode="NoResize" SizeToContent="WidthAndHeight"
        Loaded="Window_Loaded" Closed="Window_Closed">
    <Grid>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="Auto" />
            <ColumnDefinition Width="200" />
        </Grid.ColumnDefinitions>
        <StackPanel Grid.Column="0">
            <TextBlock HorizontalAlignment="Center"
                       Text="Tracking..." />
            <Image Height="300"
                   Margin="10,0,10,10"
                   Source="{Binding ColourBitmap}"
                   Width="400" />
            <Canvas x:Name="canvas" 
                    Height="300"
                    Margin="10,-310,10,10"
                    Width="400">
                <Image x:Name="eye"
                       Height="400"
                       Width="400"
                       Source="Images/SauronEye.png"
                       Visibility="{Binding ImageVisibility}">
                    <Image.RenderTransform>
                        <TransformGroup>
                            <TranslateTransform X="{Binding TranslateX}"
                                                Y="{Binding TranslateY}" />
                            <ScaleTransform CenterX="{Binding CenterX}"
                                            CenterY="{Binding CenterY}" 
                                            ScaleX="{Binding ScaleFactor}"
                                            ScaleY="{Binding ScaleFactor}" />
                        </TransformGroup>
                    </Image.RenderTransform>
                </Image>
            </Canvas>
        </StackPanel>
        <StackPanel Grid.Column="1">
            <GroupBox Header="Motor Control"
                      Height="100"
                      VerticalAlignment="Top"
                      Width="190">
                <StackPanel HorizontalAlignment="Center" 
                            Margin="10"
                            Orientation="Horizontal">
                    <Button x:Name="motorUp"
                            Click="motorUp_Click"
                            Content="Up"
                            Height="30"
                            Width="70" />
                    <Button x:Name="motorDown"
                            Click="motorDown_Click"
                            Content="Down"
                            Height="30"
                            Margin="10,0,0,0"
                            Width="70" />
                </StackPanel>
            </GroupBox>
            <GroupBox Header="Information"
                      Height="100"
                      VerticalAlignment="Top"
                      Width="190">
                <StackPanel Orientation="Horizontal" Margin="10">
                    <TextBlock Text="Frame rate: " />
                    <TextBlock Text="{Binding FramesPerSecond}"
                               VerticalAlignment="Top"
                               Width="50" />
                </StackPanel>
            </GroupBox>
        </StackPanel>
    </Grid>
</Window>

The constructor initializes the StreamManager class (contained in my KinectManager library), which handles the stream processing. The DataContext of MainWindow is set to kinectStream for binding purposes.

        public MainWindow()
        {
            InitializeComponent();
            this.kinectStream = new StreamManager();
            this.DataContext = this.kinectStream;
        }

The Window_Loaded event handler initializes the required subsystems of the Kinect pipeline, and invokes SmoothSkeletonData (which has now moved to the StreamManager class). For an explanation of what the SmoothSkeletonData method does, see this previous post. Finally, event handlers are registered for the subsystems of the Kinect pipeline.

        private void Window_Loaded(object sender, RoutedEventArgs e)
        {
            this.runtime = new Runtime();
            try
            {
                this.runtime.Initialize(
                    RuntimeOptions.UseColor |
                    RuntimeOptions.UseSkeletalTracking);
                this.cam = runtime.NuiCamera;
            }
            catch (InvalidOperationException)
            {
                MessageBox.Show
                    ("Runtime initialization failed. Ensure Kinect is plugged in");
                return;
            }
            try
            {
                this.runtime.VideoStream.Open(ImageStreamType.Video, 2,
                    ImageResolution.Resolution640x480, ImageType.Color);
            }
            catch (InvalidOperationException)
            {
                MessageBox.Show
                    ("Failed to open stream. Specify a supported image type/resolution.");
                return;
            }
            this.kinectStream.KinectRuntime = this.runtime;
            this.kinectStream.LastTime = DateTime.Now;
            this.kinectStream.SmoothSkeletonData();
            this.runtime.VideoFrameReady +=
                new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
            this.runtime.SkeletonFrameReady +=
                new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);
        }

The SkeletonFrameReady event handler invokes the GetSkeletonStream method in the StreamManager class, and passes a number of parameters into it.

        private void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
        {
            this.kinectStream.GetSkeletonStream(
                e, 
                (int)this.eye.Width, 
                (int)this.eye.Height, 
                (int)this.canvas.Width, 
                (int)this.canvas.Height);
        }

Several important properties from the StreamManager class are shown below. The CenterX and CenterY properties are bound to from the ScaleTransform in the UI, and specify the point that is the center of the scale operation. The ImageVisibility property is bound to from the Image control in the UI, and is used to control the visibility of the image of the eye. The KinectRuntime property is used to access the runtime instance from MainWindow.xaml.cs. The ScaleFactor property is bound to from the ScaleTransform in the UI, and is used to resize the image of the eye by the factor specified. The TranslateX and TranslateY properties are bound to from the TranslateTransform in the UI, and are used to specify the location of the image of the eye.

        public float CenterX
        {
            get
            {
                return this.centerX;
            }
            private set
            {
                this.centerX = value;
                this.OnPropertyChanged("CenterX");
            }
        }
        public float CenterY
        {
            get
            {
                return this.centerY;
            }
            private set
            {
                this.centerY = value;
                this.OnPropertyChanged("CenterY");
            }
        }
        public Visibility ImageVisibility
        {
            get
            {
                return this.imageVisibility;
            }
            private set
            {
                this.imageVisibility = value;
                this.OnPropertyChanged("ImageVisibility");
            }
        }
        public Runtime KinectRuntime { get; set; }
        public float ScaleFactor
        {
            get
            {
                return this.scaleFactor;
            }
            private set
            {
                this.scaleFactor = value;
                this.OnPropertyChanged("ScaleFactor");
            }
        }
        public float TranslateX
        {
            get
            {
                return this.translateX;
            }
            private set
            {
                this.translateX = value;
                this.OnPropertyChanged("TranslateX");
            }
        }
        public float TranslateY
        {
            get
            {
                return this.translateY;
            }
            private set
            {
                this.translateY = value;
                this.OnPropertyChanged("TranslateY");
            }
        }

The GetSkeletonStream method, in the StreamManager class, is shown below. It retrieves a frame of skeleton data and processes it as follows: e.SkeletonFrame.Skeletons is an array of SkeletonData structures, each of which contains the data for a single skeleton. If the TrackingState field of the SkeletonData structure indicates that the skeleton is being tracked, the image of the eye is made visible and the handLeft and handRight Vectors are set to the positions of the left and right hands, respectively. The center point between the hands is then calculated as a Vector. The CenterX and CenterY properties are then set to the image space coordinates returned from GetDisplayPosition. The TranslateX and TranslateY properties are then set to the location to display the image of the eye at. Finally, the ScaleFactor property is set to the distance between the right and left hands.

        public void GetSkeletonStream(SkeletonFrameReadyEventArgs e, int imageWidth, 
            int imageHeight, int canvasWidth, int canvasHeight)
        {
            Vector handLeft = new Vector();
            Vector handRight = new Vector();
            JointsCollection joints = null;
            foreach (SkeletonData data in e.SkeletonFrame.Skeletons)
            {
                if (SkeletonTrackingState.Tracked == data.TrackingState)
                {
                    this.ImageVisibility = Visibility.Visible;
                    joints = data.Joints;
                    handLeft = joints[JointID.HandLeft].Position;
                    handRight = joints[JointID.HandRight].Position;
                    break;
                }
                else
                {
                    this.ImageVisibility = Visibility.Collapsed;
                }
            }
            // Find center between hands
            Vector position = new Vector
            {
                X = (handLeft.X + handRight.X) / (float)2.0,
                Y = (handLeft.Y + handRight.Y) / (float)2.0,
                Z = (handLeft.Z + handRight.Z) / (float)2.0
            };
            // Convert depth space co-ordinates to image space
            Vector displayPosition = this.GetDisplayPosition(position, 
                canvasWidth, canvasHeight);
            this.CenterX = displayPosition.X;
            this.CenterY = displayPosition.Y;
            // Position image at center of hands
            displayPosition.X = (float)(displayPosition.X - 
                (imageWidth / 2));
            displayPosition.Y = (float)(displayPosition.Y - 
                (imageHeight / 2));
            this.TranslateX = displayPosition.X;
            this.TranslateY = displayPosition.Y;
            // Derive the scale factor from distance between 
            // the right hand and the left hand
            this.ScaleFactor = handRight.X - handLeft.X;
        }

Skeleton data and image data are based on different coordinate systems. Therefore, it is necessary to convert coordinates in skeleton space to image space, which is what the GetDisplayPosition method does. Skeleton coordinates in the range [-1.0, 1.0] are converted to depth coordinates by calling SkeletonEngine.SkeletonToDepthImage. This method returns x and y coordinates as floating-point numbers in the range [0.0, 1.0]. The floating-point coordinates are then converted to values in the 320x240 depth coordinate space, which is the range that NuiCamera.GetColorPixelCoordinatesFromDepthPixel currently supports. The depth coordinates are then converted to colour image coordinates by calling NuiCamera.GetColorPixelCoordinatesFromDepthPixel. This method returns colour image coordinates as values in the 640x480 colour image space. Finally, the colour image coordinates are scaled to the size of the canvas display in the application, by dividing the x coordinate by 640 and the y coordinate by 480, and multiplying the results by the height or width of the canvas display area, respectively.

        private Vector GetDisplayPosition(Vector position, int width, int height)
        {
            int colourX, colourY;
            float depthX, depthY;
            this.KinectRuntime.SkeletonEngine.SkeletonToDepthImage(position, 
                out depthX, out depthY);
            depthX = Math.Max(0, Math.Min(depthX * 320, 320));
            depthY = Math.Max(0, Math.Min(depthY * 240, 240));
            ImageViewArea imageView = new ImageViewArea();
            this.KinectRuntime.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(
                ImageResolution.Resolution640x480, 
                imageView, 
                (int)depthX, 
                (int)depthY, 
                0, 
                out colourX, 
                out colourY);
            return new Vector
            {
                X = (float)width * colourX / 640,
                Y = (float)height * colourY / 480
            };
        }

The application is shown below. It uses skeleton tracking to recognise the user’s hands, and once they are recognised it makes the Eye of Sauron visible and places it between the user’s hands. As the user moves their hands apart the size of the eye increases, and as the user closes their hands the size of the eye decreases. Therefore, scaling of the image occurs in response to the application recognizing the moving hand gesture.

Conclusion

The Kinect for Windows SDK beta from Microsoft research is a starter kit for application developers. It allows access to the Kinect sensor, and experimentation with its features. The skeleton tracking data returned from the sensor makes it easy to perform simple gesture recognition. The next step will be to generalise and extend the simple gesture recognition into a GestureRecognition class.