Sensor Emitter on Windows Phone

Sensor emitter

Learn about your phones sensors

  1. See live measurements, while reading
    how it works and what can be done with it
  2. Connect sensors to your PC wirelessly

A Windows Phone App. Created by Philip Daubmeier 2012.

What is this about?

This project arose from my Master’s Thesis, which I will hopefully finish soon. I will present my results in a few weeks then. While working on the thesis, I realized which great potential lies in sensors and how software could use them. So this site is dedicated to sharing how sensors work, which of them are available in your phone and what can be done with them. A secondary goal is to motivate developers (maybe you?) to use those capabilities to improve the experience of your app or even assemble totally new concepts.

So let’s dive right in:

Learn

Every Windows Phone has quite a few sensors

An ambient light sensor adapts the display brightness to the current environment, while a proximity sensor makes sure you don’t accidentally hang up with your cheek during a phone call. Your phones camera (and the font facing one) is a sensor, too, as it provides an interface to sense the outside world. To find your geographic location, the phone can use the mobile network and the WiFi antennas as ‘triangulation sensors’, as well as satellite positioning systems like GPS or GLONASS.
All this happens magically in the background, without users even knowing most of the time. The more the phone knows about its environment, the more it can use this information to make life easy and convenient. A lot of services are even only possible by having this kind of data from sensors.

Sensing Motion and Attitude

Apart from all these already, there are several additional sensors, that enable the phone to find out how it is aligned and being moved in 3D space:

Accelerometer

Accelerometer

The accelerometer senses accelerations/forces in all 3 axes. This enables your phone to find out two things: The orientation of the phone when it lies still and the motion if it is being moved around. The reason is, there is one force that is always present: the gravitational force. Since you know where this vector points to when using an accelerometer, you can infer the orientation of the device.
In case the device is being moved or shaked, the measured acceleration is a composition of the gravitational force and the rotational and linear motions of the device. In this case, the gravitiy component and therefore the orientation of the phone can only be estimated, based on previous measurements. Unless you intelligently combine it with other sensors (see Compound sensor).

Compass

Compass

Actually, this is a 3-axis magnetometer, measuring magnetic field strengths. However, if no culprits are disturbing the sensor, some intelligent algorithms can calculate the angular offset to the earth’s magnetic north pole, based on the raw magnetic field vector. Other sensors can support this process and make the results more accurate. In the end, you can use it like a conventional compass.

Gyroscope

Gyroscope

Don’t mix this one up with a full-fledged mechanical gyroscope. This sensor, that is built-in in many Windows Phone 7.5 Mango devices, doesn’t measure the actual (absolute) orientation. Just like every gyroscope in smartphones today, it is a miniaturized MEMS-component and can only sense (relative) angular rates, i.e. rotational speed. Those values alone are not particularly valuable. However, in combination with other sensors, the gyroscope can be extremely useful to stabilize measurements and get more accurate results.

Composite

Compound

The compound sensor is no real physical sensor, but a virtual one. It is the logical combination of all three other sensors. At first, the accelerometer lets infer the current attitude of the device. This, however, still leaves one degree of freedom: the yaw axis. This can be gathered from the compass then. The gyroscope, in turn, stabilizes those measurement results in all 3 axes of rotation. The final result is the devices (absolute) orientation in 3D space.
But the compound sensor can do even more: Having a relatively exact measurement of the attitude allows for estimating the gravity vector. Together with the gyroscopes rotation measurements, all irrelevant forces can be substracted from the accelerometer results, leaving nothing but the pure linear acceleration forces as a result.

Where does all this lead to?

The Compound sensor is composed by the Motion API, built into the Windows Phone OS, and enables for a lot of cool stuff. Whether using the phone as a precise game controller, enabling augmented reality applications or for much more other purposes, knowing the true orientation and motion is opening up a whole new world of possibilities.

In theory, if both attitude and linear acceleration is known, it is possible to track the device in 3D space relative to some known starting position. This allows, for example, for indoor navigation, where some starting position outside the building is known from a satellite navigation system. This mechanism is also known as ‘Inertial Navigation System’. Unfortunately, current smartphone sensors are not accurate enough for such applications, and have a huge drift after a short period of time. However, this shows to tell which great new opportunities are possible with even more accurate sensors in the future.

So let’s combine even more sensors together!

This is exactly the intention of this app. As seen above, intelligent combination of measurement data from different sources can lead to great new applications and services. As the number of sensors in our phones is quite large but nonetheless limited to those discribed above, I wanted to make use of sensors attached to my PC to explore new opportunities. Of course, for combining the phones sensors with those of the PC, I need to connect both and transfer a live stream of sensor data. And so, ‘Sensor emitter’ was born.

Connect

Connect with your PC

Connect your phone to the PC

To do this, you just have to follow these simple steps:

  1. Download the Sensor emitter app from the Windows Phone marketplace, if you haven’t done that already.
  2. Download the PC demo application from here.
    (guaranteed to be free of adware/spyware and checked for malware and viruses)
  3. Start the PC application. This will act as the TCP Server for our connection. Make sure you allow it to pass the Windows Firewall, if you are asked. This is important, since the PC is getting an incomming connection from the phone in a second.
  4. Start the phone app, click on Connect Icon ‘connect’ in the appbar. Make sure you are connected to a WiFi network and check if it is the same network your PC is in. Alternatively, you can connect your phone via a USB cable and start the Zune software.
  5. Still in the phone app, enter the hostname or IP together with the port of your PC. Both are shown at the bottom of the demo application.

And you are connected! Now, you can move your phone around and watch the virtual phone model on the PC moving along with it. You will notice that phones without a gyroscope are by far not as accurately followed by the virtual model.

Also, you can play around with magnetic interferences and watch the magnetometer sensor getting in trouble :). Please just don’t use strong magnets, as these could damage your phone in the worst case, but rather pieces of iron.

Was that all there is to it?

Of course not! This was just a simple demo that shows the connection was established and sensor data is arriving at the PC. I am already developing something involving a Kinect sensor that is merged live with the phone sensor data. It is just not complete yet, so stay tuned!

In the meantime, how about creating something yourself on top of the Sensor emitter basement? Jump right on to the next chapter for more info.

Create

Kickstart

To get you started, I’ve packed all you need for a TCP server, ready to receive sensor data, in one easy-to-use c# file:

using System;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using System.Windows;

namespace SensorEmitterServer
{
    public enum Context
    {
        WhileStartingListener,
        WhileReceivingData
    }

    public class ValuesReceivedEventArgs<T> : EventArgs
    {
        public T SensorReading { get; set; }
    }

    public class ExceptionOccuredEventArgs : EventArgs
    {
        public Exception Exception { get; set; }
        public Context Context { get; set; }
    }
 
    /// <summary>
    /// Interface for a sensor reading (classes that hold values of sensor measurements).
    /// </summary>
    public interface ISensorReading
    {
        /// <summary>
        /// Has to return a constant number of values this ISensorReading class expects.
        /// </summary>
        int NumSensorValues { get; }
 
        void SetSensorValues(float[] values);
    }
 
    /// <summary>
    /// A TCP Server, that accepts incoming connections from the Windows Phone
    /// 'Sensor Emitter' App, and packs all received data into SensorReading
    /// objects.
    ///
    /// Usage: Create a SensorServer&lt;SensorEmitterReading&gt; object, connect to both 
    /// ValuesReceived and ExceptionOccured events and start the server with 
    /// the Start method.
    /// </summary>
    /// <typeparam name="T">A class, which has to inherit from ISensorReading 
    /// and supply a parameterless constructor.</typeparam>
    public class SensorServer<T> : IDisposable 
        where T : ISensorReading, new()
    {
        private const int MagicNumber = 0x42fea723;
    
        public const int DefaultTcpPort = 3547;
 
        private volatile bool alive = true;
        private Thread listenThread;
 
        public event EventHandler<ValuesReceivedEventArgs<T>> ValuesReceived;
        public event EventHandler<ExceptionOccuredEventArgs> ExceptionOccured;
 
        private int numSensorValues;
 
        public SensorServer()
        {
            numSensorValues = new T().NumSensorValues;
        }
 
        /// <summary>
        /// Starts listening on the TCP port 3547 and awaits phones that connect to it
        /// via the Windows Phone 'Sensor Emitter' App.
        ///
        /// If any clients connects and sends data, the ValuesReceived event is raised
        /// for each packet that arrives successfully. If any error occurs in the process,
        /// or while starting the TCP server, the ExceptionOccured event is fired.
        /// Make sure to connect to these two events.
        /// </summary>
        public void Start() { Start(SensorServer<T>.DefaultTcpPort); }
 
        /// <summary>
        /// Starts listening on the given TCP port and awaits phones that connect to it
        /// via the Windows Phone 'Sensor Emitter' App.
        ///
        /// If any clients connects and sends data, the ValuesReceived event is raised
        /// for each packet that arrives successfully. If any error occurs in the process,
        /// or while starting the TCP server, the ExceptionOccured event is fired.
        /// Make sure to connect to these two events.
        /// </summary>
        /// <param name="tcpPort">The port to listen to.</param>
        public void Start(int tcpPort)
        {
            listenThread = new Thread(new ParameterizedThreadStart(ListenForClients));
            listenThread.IsBackground = true;
            listenThread.Start(tcpPort);
        }
 
        /// <summary>
        /// Thread, that waits for incoming connections. For each connection, a
        /// seperate thread is started then.
        /// </summary>
        private void ListenForClients(object port)
        {
            var tcpListener = new TcpListener(IPAddress.Any, (int)port);
 
            // Try to start the listener
            try { tcpListener.Start(); }
            catch (SocketException sockEx)
            {
                OnException(sockEx, Context.WhileStartingListener);
            }
 
            while (alive)
            {
                // Blocks until a client has connected to the server
                TcpClient client = tcpListener.AcceptTcpClient();
 
                // Create a thread to handle communication with connected client
                var clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
                clientThread.IsBackground = true;
                clientThread.Start(client);
            }
 
            try { tcpListener.Stop(); }
            catch (SocketException) { }
        }
 
        /// <summary>
        /// Thread, that handles the actual communication after a client connected
        /// </summary>
        private void HandleClientComm(object client)
        {
            TcpClient tcpClient = (TcpClient)client;
            NetworkStream clientStream = tcpClient.GetStream();
 
            // Write magic number as a greeting and to signal we are a compatible 'sensor emitter' server
            clientStream.Write(BitConverter.GetBytes(IPAddress.HostToNetworkOrder(MagicNumber)), 0, 4);
 
            // Read from the incoming TCP stream
            using (BinaryReader br = new BinaryReader(clientStream))
            {
                bool canRead = true;
                while (canRead && alive)
                {
                    try
                    {
                        int length = IPAddress.NetworkToHostOrder(br.ReadInt32());
 
                        if (length < 0 || length != numSensorValues)
                        {
                            if (length < 0)
                                OnException(new ArgumentOutOfRangeException("The format of " + 
                                    "the TCP stream is incorrect. It said 'a negative number " +
                                    "of items is going to follow this', which obviously does " +
                                    "not make sense, as exactly " + numSensorValues +
                                    " values are expected."), Context.WhileReceivingData);
                            else
                                OnException(new ArgumentOutOfRangeException("The format of " + 
                                    "the TCP stream is incorrect. It said '" + length +
                                    " items are going to be in this single measurement " +
                                    "package', which does not match the expected value. " + 
                                    "Valid packages from the SensorEmitter App have " +
                                    "exactly " + numSensorValues + " values."), 
                                    Context.WhileReceivingData);
                            canRead = false;
                        }
                        else
                        {
                            // Read all measurement values
                            float[] values = new float[length];
                            for (int i = 0; i < length; i++)
                                values[i] = ReadNetworkFloat(br);
 
                            // Fire event, that a new pack of values was read
                            OnValuesReceived(values);
                        }
                    }
                    catch (IOException) { canRead = false; }
                }
            }
 
            tcpClient.Close();
        }
 
 
        /// <summary>
        /// Reads a float in network byte order (big endian) with the given binary reader.
        /// This is needed because the IPAddress class doesn't provide an overload
        /// of NetworkToHostOrder for any floating point types. This is equivalent to:
        ///
        ///     int ntohl = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(reader.ReadBytes(4), 0));
        ///     return BitConverter.ToSingle(BitConverter.GetBytes(ntohl), 0);
        ///
        /// but with better performance.
        /// </summary>
        public static float ReadNetworkFloat(BinaryReader reader)
        {
            byte[] bytes = reader.ReadBytes(4);
            if (BitConverter.IsLittleEndian) { Array.Reverse(bytes); }
            return BitConverter.ToSingle(bytes, 0);
        }
 
        /// <summary>
        /// Fires the ExceptionOccured event back to the main thread.
        /// </summary>
        private void OnException(Exception ex, Context context)
        {
            if (ExceptionOccured != null && Application.Current != null)
                Application.Current.Dispatcher.Invoke(ExceptionOccured, this, 
                        new ExceptionOccuredEventArgs() { Exception = ex, Context = context });
        }
 
        /// <summary>
        /// Fires the ValuesReceived event back to the main thread.
        /// </summary>
        private void OnValuesReceived(float[] values)
        {
            if (ValuesReceived != null && Application.Current != null)
            {
                T sensorReading = new T();
                sensorReading.SetSensorValues(values);
 
                Application.Current.Dispatcher.Invoke(ValuesReceived, this, 
                        new ValuesReceivedEventArgs<T>() { SensorReading = sensorReading });
            }
        }
 
        /// <summary>
        /// Stops all associated threads, the TCP Server and clears up ressources of this object.
        /// </summary>
        public void Dispose()
        {
            alive = false;
            listenThread.Abort();
        }
    }
 
    /// <summary>
    /// Holds values that were received from the Windows Phone 'Sensor Emitter' App.
    /// </summary>
    public class SensorEmitterReading : ISensorReading
    {
        public int NumSensorValues { get { return 23; } }
 
        public double QuaternionX { get; set; }
        public double QuaternionY { get; set; }
        public double QuaternionZ { get; set; }
        public double QuaternionW { get; set; }
 
        public double RotationPitch { get; set; }
        public double RotationRoll { get; set; }
        public double RotationYaw { get; set; }
 
        public double RotationRateX { get; set; }
        public double RotationRateY { get; set; }
        public double RotationRateZ { get; set; }
 
        public double RawAccelerationX { get; set; }
        public double RawAccelerationY { get; set; }
        public double RawAccelerationZ { get; set; }
 
        public double LinearAccelerationX { get; set; }
        public double LinearAccelerationY { get; set; }
        public double LinearAccelerationZ { get; set; }
 
        public double GravityX { get; set; }
        public double GravityY { get; set; }
        public double GravityZ { get; set; }
 
        public double MagneticHeading { get; set; }
        public double TrueHeading { get; set; }
        public double HeadingAccuracy { get; set; }
 
        public double MagnetometerX { get; set; }
        public double MagnetometerY { get; set; }
        public double MagnetometerZ { get; set; }
        public bool MagnetometerDataValid { get; set; }
 
        public void SetSensorValues(float[] values)
        {
            if (values == null)
                throw new ArgumentNullException("No array of values given.");
 
            if (values.Length != NumSensorValues)
                throw new ArgumentException("Unexpected length of array. Exactly " + 
                                            NumSensorValues + " items were expected.");
 
            double x = values[0];
            double y = values[1];
            double z = values[2];
            double w = values[3];
 
            RotationPitch = Math.Atan2(2 * (y * z + w * x), w * w - x * x - y * y + z * z);
            RotationRoll = Math.Atan2(2 * (x * y + w * z), w * w + x * x - y * y - z * z);
            RotationYaw = Math.Asin(-2 * (x * z - w * y));
 
            QuaternionX = values[0];
            QuaternionY = values[1];
            QuaternionZ = values[2];
            QuaternionW = values[3];
 
            RotationRateX = values[7];
            RotationRateY = values[8];
            RotationRateZ = values[9];
 
            RawAccelerationX = values[13];
            RawAccelerationY = values[14];
            RawAccelerationZ = values[15];
 
            LinearAccelerationX = values[4];
            LinearAccelerationY = values[5];
            LinearAccelerationZ = values[6];
 
            GravityX = values[10];
            GravityY = values[11];
            GravityZ = values[12];
 
            MagneticHeading = values[16];
            TrueHeading = values[17];
            HeadingAccuracy = values[18];
 
            MagnetometerX = values[19];
            MagnetometerY = values[20];
            MagnetometerZ = values[21];
            MagnetometerDataValid = values[22] == 1d;
        }
    }
}

Just include this file in your c# application, and use it like this:

using SensorEmitterServer;

{
   // put this e.g. at your application start:
   var server = new SensorServer<SensorEmitterReading>();
   server.ExceptionOccured += (s, e) => { /* Handle a possible exception */ };
   server.ValuesReceived += (s, e) => { /* New sensor data received! Do cool stuff here */ };
   server.Start();
}