Sensor emitter
Learn about your phones sensors
- See live measurements, while
reading
how it works and what can be done with it - Connect sensors to your PC wirelessly
A Windows Phone App. Created by Philip Daubmeier 2012.
Learn about your phones sensors
A Windows Phone App. Created by Philip Daubmeier 2012.
What is this about?
This project arose from my Master’s Thesis, which I will hopefully finish soon. I will present my results in a few weeks then. While working on the thesis, I realized which great potential lies in sensors and how software could use them. So this site is dedicated to sharing how sensors work, which of them are available in your phone and what can be done with them. A secondary goal is to motivate developers (maybe you?) to use those capabilities to improve the experience of your app or even assemble totally new concepts.
So let’s dive right in:
Every Windows Phone has quite a few sensors
An ambient light sensor
adapts the display brightness to the current environment, while a
proximity sensor makes sure you don’t accidentally hang up with
your cheek during a phone call. Your phones camera (and the font facing
one) is a sensor, too, as it provides an interface to sense the outside
world. To find your geographic location, the phone can use the mobile
network and the WiFi antennas as ‘triangulation sensors’, as well as satellite
positioning systems like
GPS or
GLONASS.
All this happens magically
in the background, without users even knowing most of the time. The more
the phone knows about its environment, the more it can use this
information to make life easy and convenient. A lot of services are even
only possible by having this kind of data from sensors.
Sensing Motion and Attitude
Apart from all these already, there are several additional sensors, that enable the phone to find out how it is aligned and being moved in 3D space:
Accelerometer
The accelerometer senses accelerations/forces in all 3 axes. This enables
your phone to find out two things: The
orientation of the phone when it lies still and the motion if it is
being moved around. The reason is, there is one force that is always
present: the gravitational force. Since you know where this vector
points to when using an accelerometer, you can infer the orientation of
the device.
In case the device is being moved or shaked, the measured
acceleration is a composition of the gravitational force and the
rotational and linear motions of the device. In this case, the gravitiy
component and therefore the orientation of the phone can only be
estimated, based on previous measurements. Unless you intelligently
combine it with other sensors (see Compound sensor).
Compass
Actually, this is a 3-axis magnetometer, measuring magnetic field strengths. However, if no culprits are disturbing the sensor, some intelligent algorithms can calculate the angular offset to the earth’s magnetic north pole, based on the raw magnetic field vector. Other sensors can support this process and make the results more accurate. In the end, you can use it like a conventional compass.
Gyroscope
Don’t mix this one up with a full-fledged mechanical gyroscope. This sensor, that is built-in in many Windows Phone 7.5 Mango devices, doesn’t measure the actual (absolute) orientation. Just like every gyroscope in smartphones today, it is a miniaturized MEMS-component and can only sense (relative) angular rates, i.e. rotational speed. Those values alone are not particularly valuable. However, in combination with other sensors, the gyroscope can be extremely useful to stabilize measurements and get more accurate results.
Compound
The compound sensor is no real physical sensor, but
a virtual one. It is the logical combination of all three other sensors.
At first, the accelerometer lets infer the current attitude of the
device. This, however, still leaves one degree of freedom: the yaw axis.
This can be gathered from the compass then. The gyroscope, in turn,
stabilizes those measurement results in all 3 axes of rotation. The
final result is the devices (absolute) orientation in 3D space.
But
the compound sensor can do even more: Having a relatively exact
measurement of the attitude allows for estimating the gravity vector.
Together with the gyroscopes rotation measurements, all irrelevant
forces can be substracted from the accelerometer results, leaving
nothing but the pure linear acceleration forces as a result.
Where does all this lead to?
The Compound sensor is composed by the Motion API, built into the Windows Phone OS, and enables for a lot of cool stuff. Whether using the phone as a precise game controller, enabling augmented reality applications or for much more other purposes, knowing the true orientation and motion is opening up a whole new world of possibilities.
In theory, if both attitude and linear acceleration is known, it is possible to track the device in 3D space relative to some known starting position. This allows, for example, for indoor navigation, where some starting position outside the building is known from a satellite navigation system. This mechanism is also known as ‘Inertial Navigation System’. Unfortunately, current smartphone sensors are not accurate enough for such applications, and have a huge drift after a short period of time. However, this shows to tell which great new opportunities are possible with even more accurate sensors in the future.
So let’s combine even more sensors together!
This is exactly the intention of this app. As seen above, intelligent combination of measurement data from different sources can lead to great new applications and services. As the number of sensors in our phones is quite large but nonetheless limited to those discribed above, I wanted to make use of sensors attached to my PC to explore new opportunities. Of course, for combining the phones sensors with those of the PC, I need to connect both and transfer a live stream of sensor data. And so, ‘Sensor emitter’ was born.
Connect your phone to the PC
To do this, you just have to follow these simple steps:
And you are connected! Now, you can move your phone around and watch the virtual phone model on the PC moving along with it. You will notice that phones without a gyroscope are by far not as accurately followed by the virtual model.
Also, you can play around with magnetic interferences and watch the magnetometer sensor getting in trouble :). Please just don’t use strong magnets, as these could damage your phone in the worst case, but rather pieces of iron.
Was that all there is to it?
Of course not! This was just a simple demo that shows the connection was established and sensor data is arriving at the PC. I am already developing something involving a Kinect sensor that is merged live with the phone sensor data. It is just not complete yet, so stay tuned!
In the meantime, how about creating something yourself on top of the Sensor emitter basement? Jump right on to the next chapter for more info.
Kickstart
To get you started, I’ve packed all you need for a TCP server, ready to receive sensor data, in one easy-to-use c# file:
using System; using System.IO; using System.Net; using System.Net.Sockets; using System.Threading; using System.Windows; namespace SensorEmitterServer { public enum Context { WhileStartingListener, WhileReceivingData } public class ValuesReceivedEventArgs<T> : EventArgs { public T SensorReading { get; set; } } public class ExceptionOccuredEventArgs : EventArgs { public Exception Exception { get; set; } public Context Context { get; set; } } /// <summary> /// Interface for a sensor reading (classes that hold values of sensor measurements). /// </summary> public interface ISensorReading { /// <summary> /// Has to return a constant number of values this ISensorReading class expects. /// </summary> int NumSensorValues { get; } void SetSensorValues(float[] values); } /// <summary> /// A TCP Server, that accepts incoming connections from the Windows Phone /// 'Sensor Emitter' App, and packs all received data into SensorReading /// objects. /// /// Usage: Create a SensorServer<SensorEmitterReading> object, connect to both /// ValuesReceived and ExceptionOccured events and start the server with /// the Start method. /// </summary> /// <typeparam name="T">A class, which has to inherit from ISensorReading /// and supply a parameterless constructor.</typeparam> public class SensorServer<T> : IDisposable where T : ISensorReading, new() { private const int MagicNumber = 0x42fea723; public const int DefaultTcpPort = 3547; private volatile bool alive = true; private Thread listenThread; public event EventHandler<ValuesReceivedEventArgs<T>> ValuesReceived; public event EventHandler<ExceptionOccuredEventArgs> ExceptionOccured; private int numSensorValues; public SensorServer() { numSensorValues = new T().NumSensorValues; } /// <summary> /// Starts listening on the TCP port 3547 and awaits phones that connect to it /// via the Windows Phone 'Sensor Emitter' App. /// /// If any clients connects and sends data, the ValuesReceived event is raised /// for each packet that arrives successfully. If any error occurs in the process, /// or while starting the TCP server, the ExceptionOccured event is fired. /// Make sure to connect to these two events. /// </summary> public void Start() { Start(SensorServer<T>.DefaultTcpPort); } /// <summary> /// Starts listening on the given TCP port and awaits phones that connect to it /// via the Windows Phone 'Sensor Emitter' App. /// /// If any clients connects and sends data, the ValuesReceived event is raised /// for each packet that arrives successfully. If any error occurs in the process, /// or while starting the TCP server, the ExceptionOccured event is fired. /// Make sure to connect to these two events. /// </summary> /// <param name="tcpPort">The port to listen to.</param> public void Start(int tcpPort) { listenThread = new Thread(new ParameterizedThreadStart(ListenForClients)); listenThread.IsBackground = true; listenThread.Start(tcpPort); } /// <summary> /// Thread, that waits for incoming connections. For each connection, a /// seperate thread is started then. /// </summary> private void ListenForClients(object port) { var tcpListener = new TcpListener(IPAddress.Any, (int)port); // Try to start the listener try { tcpListener.Start(); } catch (SocketException sockEx) { OnException(sockEx, Context.WhileStartingListener); } while (alive) { // Blocks until a client has connected to the server TcpClient client = tcpListener.AcceptTcpClient(); // Create a thread to handle communication with connected client var clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm)); clientThread.IsBackground = true; clientThread.Start(client); } try { tcpListener.Stop(); } catch (SocketException) { } } /// <summary> /// Thread, that handles the actual communication after a client connected /// </summary> private void HandleClientComm(object client) { TcpClient tcpClient = (TcpClient)client; NetworkStream clientStream = tcpClient.GetStream(); // Write magic number as a greeting and to signal we are a compatible 'sensor emitter' server clientStream.Write(BitConverter.GetBytes(IPAddress.HostToNetworkOrder(MagicNumber)), 0, 4); // Read from the incoming TCP stream using (BinaryReader br = new BinaryReader(clientStream)) { bool canRead = true; while (canRead && alive) { try { int length = IPAddress.NetworkToHostOrder(br.ReadInt32()); if (length < 0 || length != numSensorValues) { if (length < 0) OnException(new ArgumentOutOfRangeException("The format of " + "the TCP stream is incorrect. It said 'a negative number " + "of items is going to follow this', which obviously does " + "not make sense, as exactly " + numSensorValues + " values are expected."), Context.WhileReceivingData); else OnException(new ArgumentOutOfRangeException("The format of " + "the TCP stream is incorrect. It said '" + length + " items are going to be in this single measurement " + "package', which does not match the expected value. " + "Valid packages from the SensorEmitter App have " + "exactly " + numSensorValues + " values."), Context.WhileReceivingData); canRead = false; } else { // Read all measurement values float[] values = new float[length]; for (int i = 0; i < length; i++) values[i] = ReadNetworkFloat(br); // Fire event, that a new pack of values was read OnValuesReceived(values); } } catch (IOException) { canRead = false; } } } tcpClient.Close(); } /// <summary> /// Reads a float in network byte order (big endian) with the given binary reader. /// This is needed because the IPAddress class doesn't provide an overload /// of NetworkToHostOrder for any floating point types. This is equivalent to: /// /// int ntohl = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(reader.ReadBytes(4), 0)); /// return BitConverter.ToSingle(BitConverter.GetBytes(ntohl), 0); /// /// but with better performance. /// </summary> public static float ReadNetworkFloat(BinaryReader reader) { byte[] bytes = reader.ReadBytes(4); if (BitConverter.IsLittleEndian) { Array.Reverse(bytes); } return BitConverter.ToSingle(bytes, 0); } /// <summary> /// Fires the ExceptionOccured event back to the main thread. /// </summary> private void OnException(Exception ex, Context context) { if (ExceptionOccured != null && Application.Current != null) Application.Current.Dispatcher.Invoke(ExceptionOccured, this, new ExceptionOccuredEventArgs() { Exception = ex, Context = context }); } /// <summary> /// Fires the ValuesReceived event back to the main thread. /// </summary> private void OnValuesReceived(float[] values) { if (ValuesReceived != null && Application.Current != null) { T sensorReading = new T(); sensorReading.SetSensorValues(values); Application.Current.Dispatcher.Invoke(ValuesReceived, this, new ValuesReceivedEventArgs<T>() { SensorReading = sensorReading }); } } /// <summary> /// Stops all associated threads, the TCP Server and clears up ressources of this object. /// </summary> public void Dispose() { alive = false; listenThread.Abort(); } } /// <summary> /// Holds values that were received from the Windows Phone 'Sensor Emitter' App. /// </summary> public class SensorEmitterReading : ISensorReading { public int NumSensorValues { get { return 23; } } public double QuaternionX { get; set; } public double QuaternionY { get; set; } public double QuaternionZ { get; set; } public double QuaternionW { get; set; } public double RotationPitch { get; set; } public double RotationRoll { get; set; } public double RotationYaw { get; set; } public double RotationRateX { get; set; } public double RotationRateY { get; set; } public double RotationRateZ { get; set; } public double RawAccelerationX { get; set; } public double RawAccelerationY { get; set; } public double RawAccelerationZ { get; set; } public double LinearAccelerationX { get; set; } public double LinearAccelerationY { get; set; } public double LinearAccelerationZ { get; set; } public double GravityX { get; set; } public double GravityY { get; set; } public double GravityZ { get; set; } public double MagneticHeading { get; set; } public double TrueHeading { get; set; } public double HeadingAccuracy { get; set; } public double MagnetometerX { get; set; } public double MagnetometerY { get; set; } public double MagnetometerZ { get; set; } public bool MagnetometerDataValid { get; set; } public void SetSensorValues(float[] values) { if (values == null) throw new ArgumentNullException("No array of values given."); if (values.Length != NumSensorValues) throw new ArgumentException("Unexpected length of array. Exactly " + NumSensorValues + " items were expected."); double x = values[0]; double y = values[1]; double z = values[2]; double w = values[3]; RotationPitch = Math.Atan2(2 * (y * z + w * x), w * w - x * x - y * y + z * z); RotationRoll = Math.Atan2(2 * (x * y + w * z), w * w + x * x - y * y - z * z); RotationYaw = Math.Asin(-2 * (x * z - w * y)); QuaternionX = values[0]; QuaternionY = values[1]; QuaternionZ = values[2]; QuaternionW = values[3]; RotationRateX = values[7]; RotationRateY = values[8]; RotationRateZ = values[9]; RawAccelerationX = values[13]; RawAccelerationY = values[14]; RawAccelerationZ = values[15]; LinearAccelerationX = values[4]; LinearAccelerationY = values[5]; LinearAccelerationZ = values[6]; GravityX = values[10]; GravityY = values[11]; GravityZ = values[12]; MagneticHeading = values[16]; TrueHeading = values[17]; HeadingAccuracy = values[18]; MagnetometerX = values[19]; MagnetometerY = values[20]; MagnetometerZ = values[21]; MagnetometerDataValid = values[22] == 1d; } } }
Just include this file in your c# application, and use it like this:
using SensorEmitterServer; { // put this e.g. at your application start: var server = new SensorServer<SensorEmitterReading>(); server.ExceptionOccured += (s, e) => { /* Handle a possible exception */ }; server.ValuesReceived += (s, e) => { /* New sensor data received! Do cool stuff here */ }; server.Start(); }