DevLost

In this article we will become familiar with the Sharepoint event receivers, a cool feature which allows the user to be notified about various events occurring inside a Sharepoint environment. An example is the possibility to receive the notification of an item added to a list. Unfortunately, the mechanism works at server side and that means that there is no “out of the box” possibility to use the feature directly in a Silverlight application.

Nevertheless, we will find a way to make the event receivers available in a Silverlight web part in order to give a new dimension of interactivity to a Sharepoint solution.

As an example we will create a small application to invite our colleagues to a coffee break and get a close to real time answer. This will involve the creation of a wcf service with a service contract for a Sharepoint solution and a service contract for a Silverlight web part. Both contracts are duplex service contracts. Now you may be like “all this mess for a silly application to get someone together for a coffee break?” In fact, it may sound useless but it is not. Event receivers are a great feature of Sharepoint and exporting them into a Silverlight web part opens a very interesting perspective.

Let’s suppose we want to create a small application running in a Sharepoint site (a kind of widget) to quickly organize a coffee break with colleagues. We want to be able to select a time and send an invitation to our user’s group. The other users should be notified quite immediately and they should be able to accept or turn down the invitation. Eventually we should be notified about all the confirmations received from our colleagues. All this without the need to refresh any page.

 The image below sums it all up:

Full story on my article on silverlightshow:

http://www.silverlightshow.net/items/Silverlight-and-Sharepoint-working-together-event-receivers-in-Silverlight.aspx

There is a recent Visual Studio extension called “Silverlight Sharepoint Web Parts” which has simplified the work of the “Sharelight” or “Silverpoint” developers. They can now embed a Silverlight application in a site page of Sharepoint in a rapid and efficient manner.

More details in my articles on silverlightshow.net:

Silverlight and Sharepoint working together: the Silverlight SharePoint Web Parts - part 1

Silverlight and Sharepoint working together: the Silverlight SharePoint Web Parts - part 2

This article is available also on SilverlightShow:
http://www.silverlightshow.net/items/The-ABC-of-streaming-in-Silverlight.aspx

Advanced media integration is one of Silverlight’s greatest strengths. In this tutorial we delve into this topic by carrying out some simple exercises. We will learn how to play a movie, how to interact with webcam and microphone and how to create a live streaming solution using Expression Encoder 4 and VLC and capture it in a Silverlight application.

Play a movie

In Silverlight there is a native control called MediaElement which allows you to do a lot of things. The first thing obviously is the possibility to play a video: using its Source property you can define the location of the video you want to see and you can use either a relative or absolute URL. Suppose you have a collection of videos in a subdirectory called “Videos” of your web application and you want to play one of them. The xaml code snippet below is all you will need:

you want to play one of them. The xaml code snippet below is all you will need:

<Grid x:Name="LayoutRoot" >

<MediaElement  x:Name="mediaEl" Source="videos/video1.wmv" />

</Grid>

If you launch the application the movie will start automatically. This happens because the Autostart property of the MediaElement is true by default. To control the video, the MediaElement exposes the canonical Play(), Pause() and Stop() methods. At this point adding a commandbar to control the video is fairly easy ; just add a StackPanel in the xaml as below:

<StackPanel Name="CommandBar" Orientation="Horizontal" Background="Beige" Height="39" Width="783" Margin="5" HorizontalAlignment="Center" Canvas.Left="3">

<Button Name="PlayMovie" Background="AntiqueWhite" Content="Play" FontWeight="Bold" Click="PlayMovie_Click" Margin="50,5,0,5" Width="100"/>

<Button Name="PauseMovie" Background="AntiqueWhite" Content="Pause" FontWeight="Bold" Click="PauseMovie_Click" Margin="50,5,0,5" Width="100"/>

<Button Name="VideoStop" Background="AntiqueWhite" Content="Stop" FontWeight="Bold" Click="VideoStop_Click" Margin="50,5,0,5" Width="100" />

</StackPanel>

And, in the code behind, the following event handlers:

private void PlayMovie_Click(object sender, RoutedEventArgs e)

        {

            if(mediaEl.CurrentState != MediaElementState.Opening ||

                mediaEl.CurrentState == MediaElementState.Stopped ||

                mediaEl.CurrentState == MediaElementState.Paused)

                mediaEl.Play();

        }

        private void PauseMovie_Click(object sender, RoutedEventArgs e)

        {

            if (mediaEl.CurrentState == MediaElementState.Playing)

                mediaEl.Pause();

        }

        private void VideoStop_Click(object sender, RoutedEventArgs e)

        {

            if(mediaEl.CurrentState == MediaElementState.Playing ||

                mediaEl.CurrentState == MediaElementState.Paused)

                mediaEl.Stop();

        }

Let’s focus on the video formats supported. The list below recaps all the formats recognized by Silverlight 4:

Drop & Know…

How do we know if a video is supported? Well, instead of digging into the properties of the file and compare them with the list above, we can be way more pragmatic, i.e. by simply dragging the video from the file system to the Silverlight app and dropping it over the MediaElement. If the MediaElment does not recognize the video, it fires an exception that we can catch the internal error and redirect to the user as a comprehensible message. Let’s see how we can achieve that.

After seeing the list of the properties of the MediaElement you will find that it exposes an AllowDrop property since it derives from UIElement . Yet hooking a drop event here won’t work because the Mediaelement is not a container. No panic! You can embed the MediaElement in a Canvas container and handle the drop event of this one. In the xaml below the Canvas object incorporates the StackPanel above mentioned and the MediaElement:

Canvas Drop="VideoCanvas_Drop" x:Name="VideoCanvas" Width="800" Height="600" AllowDrop="True" Background="RosyBrown" HorizontalAlignment="Center">

    <StackPanel Name="CommandBar" Orientation="Horizontal" Background="Beige" Height="39" Width="783" Margin="5" HorizontalAlignment="Center" Canvas.Left="3">

        <Button Name="PlayMovie" Background="AntiqueWhite" Content="Play" FontWeight="Bold" Click="PlayMovie_Click"

            Margin="50,5,0,5" Width="100"/>

        <Button Name="PauseMovie" Background="AntiqueWhite" Content="Pause" FontWeight="Bold" Click="PauseMovie_Click"

                Margin="50,5,0,5" Width="100"/>

        <Button Name="VideoStop" Background="AntiqueWhite" Content="Stop" FontWeight="Bold" Click="VideoStop_Click"

                Margin="50,5,0,5" Width="100" />

    </StackPanel>

    <MediaElement x:Name="mediaEl" Canvas.Top="50"  Width="800" Height="450" AutoPlay="False" />

</Canvas>

Let’s take a look at the VideoCanvas_Drop event handler:

void VideoCanvas_Drop(object sender, DragEventArgs e)

{

    DragEventArgs dr = e as DragEventArgs;

    if (dr.Data == null)

        return;

    IDataObject dataObject = dr.Data as IDataObject;

    FileInfo[] files = dataObject.GetData(DataFormats.FileDrop) as FileInfo[];

    if(files.Count() > 0)

        mediaEl.SetSource(files[0].OpenRead());

}

The trick here is to use the SetSource property of the MediaElement instead of Source. Since it accepts a Stream as an argument, we can pass it the FileStream related to the dropped file. If the FileStream does not contain a natively supported media source then the MediaElement fires a MediaFailed event that we can catch as below:

public MainPage()

{

    InitializeComponent();

    mediaEl.MediaFailed += new EventHandler<ExceptionRoutedEventArgs>(mediaEl_MediaFailed);

    …

}

void mediaEl_MediaFailed(object sender, ExceptionRoutedEventArgs e)

{

    WarningTxtBlock.Text = "Cannot open this media:\r\n" + e.ErrorException.Message;

}

Using a webcam

Starting From Silverlight 4 it is possible to access to webcam and microphone of the machine. Fortunately the steps required to get a working application are nicely easy. We need substantially two objects: a VideoCaptureDevice and a CaptureSource.

private VideoCaptureDevice myWebCam;
private CaptureSource myCaptureSource;

The first represents the physical webcam and the second one offers some methods to manage the webcam. In this case we won’t use a MediaElement but a VideoBrush object to paint a Rectangle with the video stream coming from the webcam. Let’s create a new Silverlight project and add the following xaml code to the MainPage.xaml:

new Silverlight project and add the following xaml code to the MainPage.xaml:

<Grid x:Name="LayoutRoot" Background="Black" Width="1024" Height="768" >

    <Grid.ColumnDefinitions>

        <ColumnDefinition />

        <ColumnDefinition />

    </Grid.ColumnDefinitions>

    <Grid.RowDefinitions>

        <RowDefinition Height="*"/>

        <RowDefinition Height="70"/>

    </Grid.RowDefinitions>

    <StackPanel Orientation="Horizontal" 

              HorizontalAlignment="Center" 

              Grid.ColumnSpan="2" 

              Grid.Row="1">

        <Button Width="70" Height="30" Margin="8"

             x:Name="StartButton" Click="StartButton_Click"

             Content="Start" />

        <Button Width="70" Height="30" Margin="8"

             x:Name="StopButton" Click="StopButton_Click" 

             Content="Stop" />

        <Button Width="70" Height="30" Margin="8"

             x:Name="SnapshotButton" Click="SnapshotButton_Click" 

             Content="Snapshot" />

    </StackPanel>

    <Border Background="DarkRed" BorderBrush="Beige" BorderThickness="5" Height="380" Margin="10,25,10,10"> 

        <Rectangle x:Name="webcamRectangle" />

    </Border>

    <Border Background="Gray" BorderBrush="Beige" Grid.Row="0" Grid.Column="1" BorderThickness="5" Height="380" Margin="10,25,10,10">

        <Rectangle x:Name="snapshotRectangle" />

    </Border>

</Grid>

As you can see there are two rectangles; the first will be painted with the VideoBrush and the second will be used to show a snapshot from the video. Now in the .cs file we can initialize the two objects mentioned above (myWebCam and myCaptureSource) and prepare everything for the video capture:

private void InitializeWebCam()

{

    myCaptureSource = new CaptureSource();

    //1° step: get the default capture device

    myWebCam = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();

    // 2° step: indicate the video capture device to be used by the CaptureSource 

    myCaptureSource.VideoCaptureDevice = myWebCam;

    // 3° step: initialize a VideoBrush with the CaptureSource just initialized 

    myVideoBrush = new VideoBrush();

    myVideoBrush.SetSource(myCaptureSource);

    // 4° step: fill the Rectangle with the stream provided by the VideoBrush

    webcamRectangle.Fill = myVideoBrush;

As you see, getting the VideoCaptureDevice that represents the webcam available is just a matter of calling the GetDefaultVideoCaptureDevice() of the static CaptureDeviceConfiguration class. Then we have to indicate the myCaptureSource object which is the video capture device from which we want to actually capture. The following step is to initialize a VideoBrush object and set its source to the myCaptureSource. In this way the VideoBrush is ready to paint a target with the stream coming from the capture device. This is what has been done in the fourth step. We then have to handle the click events of the play and stop buttons to control the video:

private void StartButton_Click(object sender, RoutedEventArgs e)

{

    if (CaptureDeviceConfiguration.AllowedDeviceAccess ||

        CaptureDeviceConfiguration.RequestDeviceAccess())

        myCaptureSource.Start();

}

private void StopButton_Click(object sender, RoutedEventArgs e)

{

    if (CaptureDeviceConfiguration.AllowedDeviceAccess ||

        CaptureDeviceConfiguration.RequestDeviceAccess())

        myCaptureSource.Stop();

}

As you may notice before starting or stopping the capture we need to make a formal request in order to use the webcam and/or the microphone. The first time involves opening a dialog box asking for an explicit consent for access to the webcam and microphone as in the image below:

And what about a snapshot taken from the video? Nothing complicated! just modify the previous InitializeWebcam() method by adding this code snippet at the end:

mySnapshotBrush = new ImageBrush();

snapshotRectangle.Fill = mySnapshotBrush;

myCaptureSource.CaptureImageCompleted +=new EventHandler<CaptureImageCompletedEventArgs>(myCaptureSource_CaptureImageCompleted);

In the code above the snapshotRectangle is the Rectangle where we want to place the snapshot and its Fill property is set to an ImageBrush. So when the ImageBrush contains an image it is painted on the rectangle. How do we acquire an image from the video? You can use the CaptureImageAsync() method of the CaptureSource object. Since it is an asynchronous method you need to add an event handler to the CaptureImageCompleted event of the CaptureSource as shown below:

void myCaptureSource_CaptureImageCompleted(object sender, CaptureImageCompletedEventArgs e)

{

    mySnapshotBrush.ImageSource = e.Result;

}

You are almost done;

the only thing to add is calling the async method when the snapshot button is clicked:

private void SnapshotButton_Click(object sender, RoutedEventArgs e)

{

    if (myCaptureSource.VideoCaptureDevice != null &&

        myCaptureSource.State == CaptureState.Started)

    {

        myCaptureSource.CaptureImageAsync();

    }

}

Capturing a live stream

To capture a live stream we will use the MediaElement object again. But first we need a media streaming source. Fortunately there is more than one way to create your own little home streaming service to experiment a little. In the next paragraph we will learn how to broadcast the webcam on our pc live over the network using two different programs, Microsoft Expression Encoder 4 and VLC media player.

Live broadcasting with Expression Encoder

Microsoft Expression Encoder 4 is freely downloadable from here . Unlike the professional version, i.e. Encoder PRO 4, it does not support live Smooth Streaming and does not have support for H.264 but it is more than enough for our purposes. When you start the program you will find three options as in the image below:

Then Expression Encoder offers you a rich UI to create your own streaming project. The first step to do is to add a Live Source. After doing so a yellow framed box appears in the “Live Sources” area on the left:

Focusing on the “Live Source 1” just created you can attach a video device to it by choosing “Video Device” from the drop down list. If you have at least one webcam on board it should be listed in the drop down as below:

Similarly you can add an audio device choosing from the drop down list below. By the way, there is no simple way here to set an IP webcam as a video source since Encoder 4 has no support for RTSP streams. In that case you need a directshow source filter.
Once you have your working live source you have to activate it by clicking on the “Cue” button in the box:

Now let’s focus on the “Presets” section on the right. Starting from the top there is a System tab which collects several predefined encoding profiles. In my experience the best compromise between video quality and latency (I will go back to this later on) is the “Standard Encoding / VC-1 High Speed Broadband” preset.

You can try to change the video parameters, for instance reducing the Buffer Window but probably you will lose quality in the video.

The final step is set your output type. In order to be able to capture the live stream from a Silverlight application we will choose the broadcast - streaming option on port 8080 as below:

Now you are ready to press the start button in the centre:

Live broadcasting with VLC

VLC is known as an efficient and free multimedia player but perhaps not everybody knows that it can be used also as a streaming server. The nice thing is that it is able to stream all that it can read, for instance it can perform a live broadcasting stream even from an IP camera. But let’s stay on track: here as in the Expression Encoder tutorial above, we want to capture the video from the built-in webcam of our PC and stream it over the network. So let’s download the software from here and read the following steps.

The first thing to do is to open the “Media” menu and choose “Open Capture Device…”:

In the dialog box make sure to select your built-in webcam in the “Video device name” section as in the image below:

As for the audio device, you can do the same. Then click on the pick button at the right of the play button to get the other button options as below:

A new stream configuration wizard will appear. On the first page just click on the “next” button to jump directly to the “Destinations” configuration:

In the “Destinations” page there are two things to do: one is configuring the streaming method and the other is choosing the appropriate transcoding. Let’s start with the first one. Open the “New destination” dropdown list and choose the MS-WMSP (MMSH) option as below:

Don’t forget to set the “Display locally” check box in order to be able to capture the live stream from the same PC. Then click to “Add” button on the right of the drop down list. You should see something like that:

The streaming method configuration is completed. Lets’ focus on the transcoding options below now. Expand the Profile dropdown list and choose the “video – VMW + WMA (ASF) option. Then click on the edit button:

In the Edit Trascoding dialog box, click on the “Video codec tab and select the WMV2 codec as below (it could be already automatically set):

Click on “Save” to return to the “Destinations” page. Click on “Next” to go to the final page where there is a useful output string which allows you to launch vlc.exe with the same configuration by adding it to the command line. Click on the “Stream” button and you are done.

The output string is (in case you want to skip all the boring steps above and launch VLC from cmd):

Vlc.exe :sout=#transcode{vcodec=WMV2,vb=800,scale=1,acodec=wma2,ab=64,channels=2,samplerate=22050} :no-sout-rtp-sap :no-sout-standard-sap :ttl=1 :sout-keep

Using the live broadcasting with Silverlight

In the two previous paragraphs we have seen how to create a live broadcasting using two different softwares and you may expect that in Silverlight you have to configure the source of the stream differently in the two cases. Fortunately it is not true: if you use the configurations shown above you will be able to use one or the other option to your liking. The only requirement you need is to set the Source property of the MediaElement as below using the mms moniker:

<MediaElement x:Name="MediaElement" Width="320" Height="240" Source="mms://127.0.0.1:8080"/>

Let’s start a new Silverlight project to create a very basic live TV. Include this xaml in the MainPage.xaml:

<Grid x:Name="LayoutRoot">

  <StackPanel Orientation="Vertical">

      <MediaElement x:Name="MediaElement" Width="320" Height="240" Source="mms://127.0.0.1:8080"/>

      <StackPanel Orientation="Horizontal" HorizontalAlignment="Center">

          <TextBlock x:Name="Status" Text="status" Margin="0,5"/>

          <TextBlock x:Name="Buffer" Text="buffer" Margin="10,5"/>

      </StackPanel>

      <StackPanel Background="Beige" Orientation="Horizontal" HorizontalAlignment="Center">

          <Button x:Name="Play" Content="Play" Click="Play_Click"/>

          <Button x:Name="Pause" Content="Pause" Click="Pause_Click"/>

          <Button x:Name="Stop" Content="Stop" Click="Stop_Click"/>

      </StackPanel>

  </StackPanel>

Grid>

And in the code behind:

public MainPage()

{

    InitializeComponent();

    MediaElement.BufferingTime = new TimeSpan(0,0,0);

    MediaElement.CurrentStateChanged += (sender, e) => {

        Status.Text = MediaElement.CurrentState.ToString(); 

        Buffer.Visibility = MediaElement.CurrentState == MediaElementState.Buffering ? Visibility.Visible : Visibility.Collapsed;

    };

    this.MediaElement.BufferingProgressChanged += (sender, e) => {

        this.Buffer.Text = string.Format("{0:0.0} %", this.MediaElement.BufferingProgress * 100);

    }; 

}

private void Play_Click(object sender, RoutedEventArgs e) 

{

    this.MediaElement.Play();

}

private void Pause_Click(object sender, RoutedEventArgs e) 

{

    this.MediaElement.Pause();

}

private void Stop_Click(object sender, RoutedEventArgs e) 

{

    this.MediaElement.Stop();

} 

As you see, there is a rude panel which contains the basic functionalities (Play, Pause, Stop) to interact with the video. It is interesting that I set to zero the BufferingTime of the MediaElement in order to reduce as much as possible the delay between the real events and what we see through the MediaElement. In fact, in my attempts I measured a latency of around 7 seconds using EE4 and 1,5 seconds using VLC on a private LAN. This latter value is not too bad to me.

Conclusion

Streaming media with Silverlight is easy and funny. The MediaElement object is like a Swiss army knife which allows you to create a complete media player capable of playing a movie, capturing the output of a webcam and even reproducing a live stream. Microsoft Expression encoder 4 and VLC are two great free programs which allow you to create multimedia platforms to play with Silverlight. For more advanced scenarios it could be useful to take a look at the following topics:

• MediaStreamSource: a part of the Silverlight runtime which allows direct access to APIs to manipulate audio and video streams. (An example application here)
• IIS Smooth Streaming: an advanced platform which enables you adaptive streaming of media to Silverlight over HTTP.
• Microsoft Media Platform-Player Framework v.2.5: an open source platform which allows developers to create sophisticated video players for IIS Smooth Streaming delivery. It is available on Codeplex.

This article is available also on SilverlightShow:
http://www.silverlightshow.net/items/NESL-Native-Extension-for-Silverlight-or-No-Except-Seven-Limits-for-Silverlight-Experimenting-the-Sensor-API-with-a-Wiimote.aspx

Introduction

The first time I heard about the Native Extension for Silverlight, I wondered: well, in what context can I use this? In what kind of scenario? And overall, is it something really usable? In this article I will try to test one of the features included in NESL, i.e. the capability of interacting with motion sensors. At the end I used the popular wiimote controller since it has a built-in accelerometer. Read the rest of the story to find out the test result.

Introduction to NESL

You as Silverlight developers surely know that from scratch Silverlight was meant as something running in a browser inside a security sandbox. The things started getting a little more articulated when Silverlight 3 introduced the “Out Of Browser” option but this did not change anything in terms of security and/or permissions.

With Silverlight 4 something really changed: the developer could decide to require “elevated trust” for his OOB application. This resulted in a more permissive sandbox. But it was not over. Silverlight 4 also opened a fast lane to the underlying OS, Windows just to be precise, by means of which trusted applications could call COM objects through IDispatch interface. What did it mean? It meant that an OOB trusted Silverlight app could do virtually everything with a suitable COM component already installed on the PC.

However... how many Silverlight compliant (i.e. COM objects exposing IDispatch) COM components are available on the Windows platform? This is what the NESL team wondered before starting the project, I assume. In fact, most Windows Seven platform services do not expose themselves as “friendly” COM components. This is the main reason behind NESL. In short, NESL allow the Silverlight developers to get access to those services which are not reachable if using COM automation. Not all the services are covered of course, but a selected set which is supposed to be growing in time.

To recap, NESL is a weapon that you can use in your Silverlight OOB trusted applications to take advantage from a relevant set of Windows Seven features and exactly when I focused on this, a new reading of the acronym crossed my mind: NESL = No (Except Seven) Limits to Silverlight instead of the officially Native Extensions for Silverlight. In fact, using this approach you can allow Silverlight to do pretty much anything. This possibility was already there when access to COM Automation was added to Silverlight 4, but one thing is to write C++ COM automation components by yourself, and a different thing is if someone else does the drudgery for you.

Start using NESL

At the time I am writing this article, the newest NESL version available is the 2.0. from this link:

http://archive.msdn.microsoft.com/nesl

you can check for the current version and then go to the download section where you can download the binary runtime, the source code and the documentation. The source code includes a set of classes packaged into appropriate Silverlight libraries that act as wrappers on the native components providing the automation. The source code includes also some examples showing how to use these Silverlight libraries. The native components are in the runtime library and its source code is not available.

The libraries included in the version 2.0 of NESL are essentially:

What you need to do in order to get the benefits offered by NESL in your application is:

1. Install the native runtime of NESL (both “exe” and “msi” setup are provided)
2. Enable “Out of Browser” option and “require elevated trust…” sub-option in your Silverlight application
3. Include some of the libraries in the table above depending on the windows 7 platform feature you want to use (the first library in the list is always required)

Having said that, you have to bear in mind that when you deliver your application to the users, they do need to install the NESL runtime as well. This means practically that they have to install and register one or more COM components and this requires admin rights. Fortunately, in NESL there are some helper classes that allow you to automate the process of installation of the runtime. In brief, you can either package the NESL installer in the xap file of your application or you can leave the installer in a website and use the install helpers to download the runtime silently.

A look at the NESL Sensor API

When I started playing with NESL, I was soon captured by the idea of writing something capable to read and display data coming from a sensor. Reading a bit more in depth I realized that the Sensors Silverlight library, provided in the source code package, is a wrapper class for the SLSensorShim.dll, the component of the NESL runtime which exposes automation for the so called Windows 7 Sensor and Location Platform API. This API provides a standard way to integrate sensor and location devices with Windows 7, basically a guide for the manufacturers and a standard programming model for developers who want to use these devices in their applications.

Without going through too many details (there is the NESL developer’s guide for that) the NESL Sensor API includes various categories of sensors, location, environmental, orientation, electrical, mechanical, motion, biometric, light and scanner sensors. The usage of the API is fairly simple: you create an instance of the SensorManager type and use, for instance, the GetSensorsByCategory() method to get a collection of Sensors of a specific category or use the GetSensorsByID() to get the Sensor with the specific ID. Once you have the Sensor, you can handle the sensor events in order to know the state of the Sensor, or when it has been disconnected or even it has new data to report.
Well, I thought, now I just need some kind of sensor to start trying these things out, but.. which sensor? Unfortunately, the biometric device on my notebook was not exposed as a biometric sensor and I started casting around for a suitable device until I remembered that two years earlier I had bought a wiimote controller to make some weird experiment with a bicycle (an attempt to calculate the power of pedalling through acceleration measurements) and then I had just left it in a drawer. So I wondered if I could use it as a motion sensor; a quick googling led me to the CodeProject site where there was an article explaining a “sensor motion driver for the Wiimote”. Well, I thought, here we are.

Testing the NESL Sensor API using a Wiimote controller

Stop and think for a moment: we have a Silverlight application running out of box but still in a sandbox layer, then we have the “Sensor and Location Platform” layer and above the “NESL Sensor” layer providing COM automation. Putting it all together could possibly cause a little overhead? To test it better I decided to use an old 5 or 6-year old notebook equipped with a Mobile Intel Centrino M 760, 2 Gb RAM and a video card with 256 Mb RAM, not exactly a monster of speed but a test lab consistent with a real world scenario.

As I anticipated in the previous paragraph on CodeProject at the following link:

http://www.codeproject.com/KB/system/wiisensor.aspx

you can find a great article entitled “Writing a Sensor Driver for the Wiimote on windows 7”. It is an excellent tutorial on how to write a sensor driver and provides a working sensor motion driver for the wiimote. Furthermore, it guides you step by step through the various phases of installation of the driver.

To make the test application more attractive I looked for a nice gauge control and eventually I found it through some free controls available from Telerik at this link:

http://www.telerik.com/products/free-silverlight-controls.aspx

At this point the recipe is ready:

• An old notebook with Windows 7 (x86)
• A wiimote controller
• A wiimote sensor driver
• A nice gauge control

The additional steps to follow are:

• pairing the wiimote with the notebook
• initializing the Wiimote running the WiimoteTest app
• install the motion sensor driver
• develop the OOB Silverlight application

The first three steps are well explained in the above mentioned article here, and here. Alternatively, you can watch this short video showing how I did with my notebook.

The Silverlight OOB test application

Once the wiimote is connected and enabled as a sensor you are ready to test the NESL Sensors library in your applications. I prepared a video showing how to build the test application capable of interacting with the wiimote in less than 10 minutes. First thing to do is to make your application running out of the browser with elevated trust as shown below:

Then you have to include some references to some NESL libraries and to some Telerik libraries (for the gauge control) :

For the look and feel of the application I inserted three gauge controls, a picture showing the tree axis of the wiimote and the output of a webcam connected to the notebook as below:

I put the video window to see if there is direct feedback between the movements of the wiimote and the needles of the gauges controls.

Just a few words on the Telerik gauge controls; here there is the xaml code I used for the first of them:

 1: <telerik:RadialGauge Height="200"  HorizontalAlignment="Left" Margin="46,8,0,0"

 2:                       Name="radialGaugeX" VerticalAlignment="Top" Width="200" >

 3:  

 4:     <telerik:RadialScale x:Name="radialScale" Min="-10" Max="10" MajorTicks="10"

 5:                     MiddleTicks="2" MinorTicks="1">

 6:  

 7:         <telerik:RadialScale.MajorTick>

 8:             <telerik:MajorTickProperties />

 9:         </telerik:RadialScale.MajorTick>

 10:         <telerik:RadialScale.MiddleTick>

 11:             <telerik:MiddleTickProperties Length="0.05" />

 12:         </telerik:RadialScale.MiddleTick>

 13:         <telerik:RadialScale.MinorTick>

 14:             <telerik:MinorTickProperties Length="0.01" />

 15:         </telerik:RadialScale.MinorTick>

 16:  

 17:         <telerik:RadialScale.Label>

 18:             <telerik:LabelProperties FontSize="10" />

 19:         </telerik:RadialScale.Label>

 20:  

 21:         <telerik:IndicatorList>

 22:             

 23:             <telerik:Needle x:Name="gaugeX_needle" IsAnimated="true" Value="0" />

 24:         </telerik:IndicatorList>

 25:     </telerik:RadialScale>

 26: </telerik:RadialGauge>

I used here the Radial style but you can choose also a Linear option. There is the possibility to define scales, ticks and choose between 5 indicators.

As far as the sensor management is concerned, all is condensed in the following piece of code:

 1: void MainPage_Loaded(object sender, RoutedEventArgs e)

 2:  

 3:    if (Application.Current.IsRunningOutOfBrowser && Application.Current.HasElevatedPermissions)

 4:    {

 5:        try

 6:        {

 7:            SetupWebcam();

 8:  

 9:            sensorManager = new SensorManager();

 10:  

 11:            //get the accelerometer

 12:            WiiAccelerometer = sensorManager.GetSensorsByCategory(

 13:              SensorCategories.SENSOR_CATEGORY_MOTION).FirstOrDefault();

 14:            

 15:            //if one exists

 16:            if (WiiAccelerometer != null)

 17:            {

 18:  

 19:                // WiiAccelerometer.SetProperties(new PropertyValue[] {new PropertyValue(new PropertyKey(SensorProperties.Base, 

 20:                // SensorProperties.SENSOR_PROPERTY_CURRENT_REPORT_INTERVAL), (uint)1000)});

 21:                                        

 22:                //handle the data updated event

 23:                //WiiAccelerometer.SensorDataUpdated += new EventHandler<SensorDataUpdatedEventArgs>(OnSensorDataUpdated);

 24:  

 25:                AccelerometerMovementThreshold = 0.01;

 26:                

 27:                myDispatcherTimer.Interval = new TimeSpan(0, 0, 0, 0, 100); // Milliseconds 

 28:                myDispatcherTimer.Tick += new EventHandler(Each_Tick);

 29:                myDispatcherTimer.Start();

 30:            }

 31:        }

 32:        catch (Exception Ex)

 33:        {

 34:            MessageBox.Show(Ex.Message, "ERROR", MessageBoxButton.OK);

 35:        }

 36:    }

As you see, it’s a matter of creating a SensorManager instance and look for your motion device using the GetSensorsByCategory(…) method (I took the first of the list since I was sure it was the only one present). Here I commented on the suggested way to go in the draft developer’s NESL guide : each motion sensor is required to produce a report of their data within a given interval of time, you can force this time at the value you want by setting this property SENSOR_PROPERTY_CURRENT_REPORT_INTERVAL. Then you can receive a notification of changes in the data adding an event handler to the SensorDataUpdated event.

Unfortunately, this approach does not work in my case, that’s why I commented these lines of code. The application crashes when it tries to set the new interval report. I gave a quick look at the sources of the wiimote sensor driver and I found that the interval report is set to 0. I did not investigate further and I opted for a classic timer approach where I asked for the data from the sensor and I checked if the difference between the current and the previous value exceeds a certain threshold. Here below is the code of the timer function:

 1: public void Each_Tick(object o, EventArgs sender)

 2:  {

 3:      //timestamp

 4:      //DateTime dt = e.NewData.TimeStamp;

 5:      //create the property keys for the data fields to query

 6:      PropertyKey propKey_X_G = new PropertyKey(MotionDataFields.Base,

 7:        MotionDataFields.SENSOR_DATA_TYPE_ACCELERATION_X_G);

 8:      PropertyKey propKey_Y_G = new PropertyKey(MotionDataFields.Base,

 9:        MotionDataFields.SENSOR_DATA_TYPE_ACCELERATION_Y_G);

 10:      PropertyKey propKey_Z_G = new PropertyKey(MotionDataFields.Base,

 11:        MotionDataFields.SENSOR_DATA_TYPE_ACCELERATION_Z_G);

 12:  

 13:      //query the data fields

 14:      List<PropertyValue> accelerationValues =

 15:            WiiAccelerometer.Data.GetSensorValues(new PropertyKey[] { propKey_X_G, propKey_Y_G, propKey_Z_G }).ToList();

 16:      //do the necessary conversions

 17:      double accel_XAxis = Convert.ToDouble(accelerationValues[0].Value); deltaX = lastX == 0.0 ? 0.0 : lastX - accel_XAxis; lastX = accel_XAxis;

 18:      double accel_YAxis = Convert.ToDouble(accelerationValues[1].Value); deltaY = lastY == 0.0 ? 0.0 : lastY - accel_YAxis; lastY = accel_YAxis;

 19:      double accel_ZAxis = Convert.ToDouble(accelerationValues[2].Value); deltaZ = lastZ == 0.0 ? 0.0 : lastZ - accel_ZAxis; lastZ = accel_ZAxis;

 20:  

 21:  

 22:      //do something with the values

 23:      if (deltaX > AccelerometerMovementThreshold || deltaY > AccelerometerMovementThreshold || deltaZ > AccelerometerMovementThreshold)

 24:      {

 25:          gaugeX_needle.Value = accel_XAxis * 10;

 26:          gaugeY_needle.Value = accel_YAxis * 10;

 27:          gaugeZ_needle.Value = accel_ZAxis * 10;

 28:      }

 29:  

 30:  }

Test results

I have to say that the test results are encouraging. Compiling in release mode and using a timer interval of 100 ms the application seems well responsive even with an old and slow notebook. Not the same in debug mode, where the application freezes sometime for a while. I have to add that I was unable to record my video showing the application in action using the Microsoft Expression Encoder screen capture and I had to use an external videocamera. However, honestly I was asking too much of my notebook here.

Summary

In this article we have seen that virtually there are no limits to what Silverlight actually can do; we can even make it speak with a wiimote. The key tool behind that is NESL, a library which provides friendly Silverlight wrapper classes and a runtime which exposes COM automation for most of the features of windows 7 platform. How efficient and practical is this approach? In the article we have tested the NESL Sensors API and the results appear to be encouraging. The question is: what about this library when Silverlight 5 will arrive with, as it seems (see keynote), Platform Invocation (Pinvoke) enabled to in-browser trusted applications ?

Download the source code of the test application