DevLost

This article is available also on SilverlightShow:
http://www.silverlightshow.net/items/NESL-Native-Extension-for-Silverlight-or-No-Except-Seven-Limits-for-Silverlight-Experimenting-the-Sensor-API-with-a-Wiimote.aspx

Introduction

The first time I heard about the Native Extension for Silverlight, I wondered: well, in what context can I use this? In what kind of scenario? And overall, is it something really usable? In this article I will try to test one of the features included in NESL, i.e. the capability of interacting with motion sensors. At the end I used the popular wiimote controller since it has a built-in accelerometer. Read the rest of the story to find out the test result.

Introduction to NESL

You as Silverlight developers surely know that from scratch Silverlight was meant as something running in a browser inside a security sandbox. The things started getting a little more articulated when Silverlight 3 introduced the “Out Of Browser” option but this did not change anything in terms of security and/or permissions.

With Silverlight 4 something really changed: the developer could decide to require “elevated trust” for his OOB application. This resulted in a more permissive sandbox. But it was not over. Silverlight 4 also opened a fast lane to the underlying OS, Windows just to be precise, by means of which trusted applications could call COM objects through IDispatch interface. What did it mean? It meant that an OOB trusted Silverlight app could do virtually everything with a suitable COM component already installed on the PC.

However... how many Silverlight compliant (i.e. COM objects exposing IDispatch) COM components are available on the Windows platform? This is what the NESL team wondered before starting the project, I assume. In fact, most Windows Seven platform services do not expose themselves as “friendly” COM components. This is the main reason behind NESL. In short, NESL allow the Silverlight developers to get access to those services which are not reachable if using COM automation. Not all the services are covered of course, but a selected set which is supposed to be growing in time.

To recap, NESL is a weapon that you can use in your Silverlight OOB trusted applications to take advantage from a relevant set of Windows Seven features and exactly when I focused on this, a new reading of the acronym crossed my mind: NESL = No (Except Seven) Limits to Silverlight instead of the officially Native Extensions for Silverlight. In fact, using this approach you can allow Silverlight to do pretty much anything. This possibility was already there when access to COM Automation was added to Silverlight 4, but one thing is to write C++ COM automation components by yourself, and a different thing is if someone else does the drudgery for you.

Start using NESL

At the time I am writing this article, the newest NESL version available is the 2.0. from this link:

http://archive.msdn.microsoft.com/nesl

you can check for the current version and then go to the download section where you can download the binary runtime, the source code and the documentation. The source code includes a set of classes packaged into appropriate Silverlight libraries that act as wrappers on the native components providing the automation. The source code includes also some examples showing how to use these Silverlight libraries. The native components are in the runtime library and its source code is not available.

The libraries included in the version 2.0 of NESL are essentially:

What you need to do in order to get the benefits offered by NESL in your application is:

1. Install the native runtime of NESL (both “exe” and “msi” setup are provided)
2. Enable “Out of Browser” option and “require elevated trust…” sub-option in your Silverlight application
3. Include some of the libraries in the table above depending on the windows 7 platform feature you want to use (the first library in the list is always required)

Having said that, you have to bear in mind that when you deliver your application to the users, they do need to install the NESL runtime as well. This means practically that they have to install and register one or more COM components and this requires admin rights. Fortunately, in NESL there are some helper classes that allow you to automate the process of installation of the runtime. In brief, you can either package the NESL installer in the xap file of your application or you can leave the installer in a website and use the install helpers to download the runtime silently.

A look at the NESL Sensor API

When I started playing with NESL, I was soon captured by the idea of writing something capable to read and display data coming from a sensor. Reading a bit more in depth I realized that the Sensors Silverlight library, provided in the source code package, is a wrapper class for the SLSensorShim.dll, the component of the NESL runtime which exposes automation for the so called Windows 7 Sensor and Location Platform API. This API provides a standard way to integrate sensor and location devices with Windows 7, basically a guide for the manufacturers and a standard programming model for developers who want to use these devices in their applications.

Without going through too many details (there is the NESL developer’s guide for that) the NESL Sensor API includes various categories of sensors, location, environmental, orientation, electrical, mechanical, motion, biometric, light and scanner sensors. The usage of the API is fairly simple: you create an instance of the SensorManager type and use, for instance, the GetSensorsByCategory() method to get a collection of Sensors of a specific category or use the GetSensorsByID() to get the Sensor with the specific ID. Once you have the Sensor, you can handle the sensor events in order to know the state of the Sensor, or when it has been disconnected or even it has new data to report.
Well, I thought, now I just need some kind of sensor to start trying these things out, but.. which sensor? Unfortunately, the biometric device on my notebook was not exposed as a biometric sensor and I started casting around for a suitable device until I remembered that two years earlier I had bought a wiimote controller to make some weird experiment with a bicycle (an attempt to calculate the power of pedalling through acceleration measurements) and then I had just left it in a drawer. So I wondered if I could use it as a motion sensor; a quick googling led me to the CodeProject site where there was an article explaining a “sensor motion driver for the Wiimote”. Well, I thought, here we are.

Testing the NESL Sensor API using a Wiimote controller

Stop and think for a moment: we have a Silverlight application running out of box but still in a sandbox layer, then we have the “Sensor and Location Platform” layer and above the “NESL Sensor” layer providing COM automation. Putting it all together could possibly cause a little overhead? To test it better I decided to use an old 5 or 6-year old notebook equipped with a Mobile Intel Centrino M 760, 2 Gb RAM and a video card with 256 Mb RAM, not exactly a monster of speed but a test lab consistent with a real world scenario.

As I anticipated in the previous paragraph on CodeProject at the following link:

http://www.codeproject.com/KB/system/wiisensor.aspx

you can find a great article entitled “Writing a Sensor Driver for the Wiimote on windows 7”. It is an excellent tutorial on how to write a sensor driver and provides a working sensor motion driver for the wiimote. Furthermore, it guides you step by step through the various phases of installation of the driver.

To make the test application more attractive I looked for a nice gauge control and eventually I found it through some free controls available from Telerik at this link:

http://www.telerik.com/products/free-silverlight-controls.aspx

At this point the recipe is ready:

• An old notebook with Windows 7 (x86)
• A wiimote controller
• A wiimote sensor driver
• A nice gauge control

The additional steps to follow are:

• pairing the wiimote with the notebook
• initializing the Wiimote running the WiimoteTest app
• install the motion sensor driver
• develop the OOB Silverlight application

The first three steps are well explained in the above mentioned article here, and here. Alternatively, you can watch this short video showing how I did with my notebook.

The Silverlight OOB test application

Once the wiimote is connected and enabled as a sensor you are ready to test the NESL Sensors library in your applications. I prepared a video showing how to build the test application capable of interacting with the wiimote in less than 10 minutes. First thing to do is to make your application running out of the browser with elevated trust as shown below:

Then you have to include some references to some NESL libraries and to some Telerik libraries (for the gauge control) :

For the look and feel of the application I inserted three gauge controls, a picture showing the tree axis of the wiimote and the output of a webcam connected to the notebook as below:

I put the video window to see if there is direct feedback between the movements of the wiimote and the needles of the gauges controls.

Just a few words on the Telerik gauge controls; here there is the xaml code I used for the first of them:

 1: <telerik:RadialGauge Height="200"  HorizontalAlignment="Left" Margin="46,8,0,0"

 2:                       Name="radialGaugeX" VerticalAlignment="Top" Width="200" >

 3:  

 4:     <telerik:RadialScale x:Name="radialScale" Min="-10" Max="10" MajorTicks="10"

 5:                     MiddleTicks="2" MinorTicks="1">

 6:  

 7:         <telerik:RadialScale.MajorTick>

 8:             <telerik:MajorTickProperties />

 9:         </telerik:RadialScale.MajorTick>

 10:         <telerik:RadialScale.MiddleTick>

 11:             <telerik:MiddleTickProperties Length="0.05" />

 12:         </telerik:RadialScale.MiddleTick>

 13:         <telerik:RadialScale.MinorTick>

 14:             <telerik:MinorTickProperties Length="0.01" />

 15:         </telerik:RadialScale.MinorTick>

 16:  

 17:         <telerik:RadialScale.Label>

 18:             <telerik:LabelProperties FontSize="10" />

 19:         </telerik:RadialScale.Label>

 20:  

 21:         <telerik:IndicatorList>

 22:             

 23:             <telerik:Needle x:Name="gaugeX_needle" IsAnimated="true" Value="0" />

 24:         </telerik:IndicatorList>

 25:     </telerik:RadialScale>

 26: </telerik:RadialGauge>

I used here the Radial style but you can choose also a Linear option. There is the possibility to define scales, ticks and choose between 5 indicators.

As far as the sensor management is concerned, all is condensed in the following piece of code:

 1: void MainPage_Loaded(object sender, RoutedEventArgs e)

 2:  

 3:    if (Application.Current.IsRunningOutOfBrowser && Application.Current.HasElevatedPermissions)

 4:    {

 5:        try

 6:        {

 7:            SetupWebcam();

 8:  

 9:            sensorManager = new SensorManager();

 10:  

 11:            //get the accelerometer

 12:            WiiAccelerometer = sensorManager.GetSensorsByCategory(

 13:              SensorCategories.SENSOR_CATEGORY_MOTION).FirstOrDefault();

 14:            

 15:            //if one exists

 16:            if (WiiAccelerometer != null)

 17:            {

 18:  

 19:                // WiiAccelerometer.SetProperties(new PropertyValue[] {new PropertyValue(new PropertyKey(SensorProperties.Base, 

 20:                // SensorProperties.SENSOR_PROPERTY_CURRENT_REPORT_INTERVAL), (uint)1000)});

 21:                                        

 22:                //handle the data updated event

 23:                //WiiAccelerometer.SensorDataUpdated += new EventHandler<SensorDataUpdatedEventArgs>(OnSensorDataUpdated);

 24:  

 25:                AccelerometerMovementThreshold = 0.01;

 26:                

 27:                myDispatcherTimer.Interval = new TimeSpan(0, 0, 0, 0, 100); // Milliseconds 

 28:                myDispatcherTimer.Tick += new EventHandler(Each_Tick);

 29:                myDispatcherTimer.Start();

 30:            }

 31:        }

 32:        catch (Exception Ex)

 33:        {

 34:            MessageBox.Show(Ex.Message, "ERROR", MessageBoxButton.OK);

 35:        }

 36:    }

As you see, it’s a matter of creating a SensorManager instance and look for your motion device using the GetSensorsByCategory(…) method (I took the first of the list since I was sure it was the only one present). Here I commented on the suggested way to go in the draft developer’s NESL guide : each motion sensor is required to produce a report of their data within a given interval of time, you can force this time at the value you want by setting this property SENSOR_PROPERTY_CURRENT_REPORT_INTERVAL. Then you can receive a notification of changes in the data adding an event handler to the SensorDataUpdated event.

Unfortunately, this approach does not work in my case, that’s why I commented these lines of code. The application crashes when it tries to set the new interval report. I gave a quick look at the sources of the wiimote sensor driver and I found that the interval report is set to 0. I did not investigate further and I opted for a classic timer approach where I asked for the data from the sensor and I checked if the difference between the current and the previous value exceeds a certain threshold. Here below is the code of the timer function:

 1: public void Each_Tick(object o, EventArgs sender)

 2:  {

 3:      //timestamp

 4:      //DateTime dt = e.NewData.TimeStamp;

 5:      //create the property keys for the data fields to query

 6:      PropertyKey propKey_X_G = new PropertyKey(MotionDataFields.Base,

 7:        MotionDataFields.SENSOR_DATA_TYPE_ACCELERATION_X_G);

 8:      PropertyKey propKey_Y_G = new PropertyKey(MotionDataFields.Base,

 9:        MotionDataFields.SENSOR_DATA_TYPE_ACCELERATION_Y_G);

 10:      PropertyKey propKey_Z_G = new PropertyKey(MotionDataFields.Base,

 11:        MotionDataFields.SENSOR_DATA_TYPE_ACCELERATION_Z_G);

 12:  

 13:      //query the data fields

 14:      List<PropertyValue> accelerationValues =

 15:            WiiAccelerometer.Data.GetSensorValues(new PropertyKey[] { propKey_X_G, propKey_Y_G, propKey_Z_G }).ToList();

 16:      //do the necessary conversions

 17:      double accel_XAxis = Convert.ToDouble(accelerationValues[0].Value); deltaX = lastX == 0.0 ? 0.0 : lastX - accel_XAxis; lastX = accel_XAxis;

 18:      double accel_YAxis = Convert.ToDouble(accelerationValues[1].Value); deltaY = lastY == 0.0 ? 0.0 : lastY - accel_YAxis; lastY = accel_YAxis;

 19:      double accel_ZAxis = Convert.ToDouble(accelerationValues[2].Value); deltaZ = lastZ == 0.0 ? 0.0 : lastZ - accel_ZAxis; lastZ = accel_ZAxis;

 20:  

 21:  

 22:      //do something with the values

 23:      if (deltaX > AccelerometerMovementThreshold || deltaY > AccelerometerMovementThreshold || deltaZ > AccelerometerMovementThreshold)

 24:      {

 25:          gaugeX_needle.Value = accel_XAxis * 10;

 26:          gaugeY_needle.Value = accel_YAxis * 10;

 27:          gaugeZ_needle.Value = accel_ZAxis * 10;

 28:      }

 29:  

 30:  }

Test results

I have to say that the test results are encouraging. Compiling in release mode and using a timer interval of 100 ms the application seems well responsive even with an old and slow notebook. Not the same in debug mode, where the application freezes sometime for a while. I have to add that I was unable to record my video showing the application in action using the Microsoft Expression Encoder screen capture and I had to use an external videocamera. However, honestly I was asking too much of my notebook here.

Summary

In this article we have seen that virtually there are no limits to what Silverlight actually can do; we can even make it speak with a wiimote. The key tool behind that is NESL, a library which provides friendly Silverlight wrapper classes and a runtime which exposes COM automation for most of the features of windows 7 platform. How efficient and practical is this approach? In the article we have tested the NESL Sensors API and the results appear to be encouraging. The question is: what about this library when Silverlight 5 will arrive with, as it seems (see keynote), Platform Invocation (Pinvoke) enabled to in-browser trusted applications ?

Download the source code of the test application