What is Sensor Fusion?

Find the Latest SoC Solutions
for... Automotive... IoT... Security... Audio... Video

CEVA BLOG - Charles Pao

www.ceva-dsp.com, Dec. 06, 2021 – 

Motion sensing devices are all around us and in several of the electronics that we use on a daily basis. Motion sensors are in earbuds – noticing taps to change songs or pausing music when we take them out of our ears. They're used for head tracking in virtual reality (VR) and augmented reality (AR) headsets used in gaming and training. Motion sensors enable gaming remotes (for orientation), consumer grade robots (for heading), and your phone (also for orientation). These devices' sensing are powered by inertial measuring units (IMUs). IMU sensors are in many commonly used consumer electronics, like the ones listed prior. And their capabilities are unlocked by sensor fusion. In this article we'll explore what sensor fusion is and what it can do.

What's an IMU sensor?

Before we get into sensor fusion, a quick review of the Inertial Measurement Unit (IMU) seems pertinent. An IMU is a sensor typically composed of an accelerometer and gyroscope, and sometimes additionally a magnetometer. By looking at data from these sensors, a device is able to get a more complete picture of its orientation and motion state.

• The accelerometer measures acceleration (change of velocity) in a single direction, like the force you feel when you step on the gas in your car. At rest, an accelerometer measures the force of gravity.
• Gyroscopes measure the angular velocity about its three axes. It outputs its rotational yaw, pitch, and roll at any given moment.
• A magnetometer, quite simply, measures magnetic fields. With proper calibration in a stable magnetic field, it can detect fluctuations in Earth's magnetic field. Through those fluctuations, it finds the vector towards Earth's magnetic North, giving it an absolute heading.

The sensor's information is then used to maintain a drone's balance, improve the heading of a home robot vacuum cleaner, change the orientation of a smartphone screen, and other motion-related applications.

How IMUs are Used in Sensor Fusion

Now that we understand what makes up an IMU, how does it relate to sensor fusion and why do we care? Well, sensors alone are not that "smart". They generate raw data. But this raw data must be processed and packaged to become actionable.

Sensors in an IMU are similar to specialist doctors reading over your patient file – they all have opinions, and their specializations give them insights others don't have, but it's up to you to process their opinions to make your final decision. For instance, if the accelerometer is suggesting that gravity is changing from pointing downward to a more horizontal angle, but the gyroscope is showing that there's almost no motion, which do you trust? Well, in this scenario, the gyroscope should be trusted more as it's not influenced by outside forces. Since the gyroscope tells us that the user frame hasn't changed, it's safe to say that the device is accelerating consistently, like a car driving straight.

In another scenario, if the gyroscope is showing a small, consistent angular velocity, but the accelerometer and the magnetometer show that the device is at rest, then you may trust the opinion of the two "doctors" that agree. And then you can infer that there's some gyroscope bias that's giving a false output.

These examples aim to show how sensor fusion is essential to understanding what the best output is based on the fusion of its sensors' information. This can be used to determine accurate motion, orientation, and heading information.

Exploring the Possibilities

When combined with sensor fusion software, IMUs can be used not only for more accurate motion, orientation, and heading, but also for specialized functionality. Thoughtful fusion of IMU data can create a smooth XR experience with predictive head tracking that minimizes latency effects. For wireless presenting or TV remotes, sensor fusion can directly translate the 3D controller motion into intuitive 2D motion on-screen. The combination of accelerometer and gyroscope sensors can also detect complex in-air shapes and gestures. In human navigation, analysis of the data from the accelerometer and gyroscope, sensor fusion can estimate the direction and distance someone has traveled on foot.

Sensor fusion doesn't have to be done solely with the IMU, but it does often start with it. In the XR space, fusion of controller orientation with linear position from an external camera can create an effective inside-out 6-DOF system. For robot navigation, fusion of an IMU with optical flow and wheel encoder data creates an accurate and robust dead reckoning. If there's motion involved, sensor fusion likely can help.

Sensor Characterization and Calibration

Another piece of sensor fusion is ensuring that the sensors are calibrated appropriately, as IMU sensors are strongly affected by their calibration. Sensor characterization is the process of taking measurements from a sensor under controlled conditions. These measurements can be used to fine-tune how the sensor reacts to various temperatures, operating modes, and motion. Once a sensor has been properly characterized, sensor fusion can help ensure its performance is optimized.

The Sensor Characterization Process

In order to properly characterize a sensor, a statistically significant number of sensors need to be placed on some sort of board that allows communication to change modes and log data. This board should then be placed inside a controlled environment. As an example, they could sit on a two-axis gimbal motor that allows it to move through all three axes of movement. By putting this contraption inside a temperature chamber, we can iterate through permutations of varying temperatures, positions, and operating modes. With knowledge on how each high accuracy motor moved, the temperature changes, and what modes were run, we can acquire a large amount of sensor information to characterize the sensor. To characterize magnetometers, the board can be placed in a Helmholtz Coil to generate a controlled magnetic field.

In order to test how these sensors perform over their lifetime, the sensors can also undergo an artificial aging process by exposing them to extreme conditions of heat and humidity. Then, the same tests can be run with the aged sensors to collect updated data.

A sensor model can be created with all this comprehensive data, leading to a model of a typical (nominal) sensor and – subsequently – the optimization of its performance.

By understanding how the sensor behaves, sensor bias in the accelerometer and gyroscope can also be adjusted for. These sensor biases are related to what the sensors see while at rest. If this sounds familiar, it's because it's the second idea discussed in the IMU/doctor analogy section. Adjusting for these may seem as simple as an offset, but these biases can change with temperature, and show different behavior for the same sensor from the same lot. This bias error is significant, trumping most other errors outside of scale error. However, with the proper sensor fusion algorithms, this calibration can be done dynamically while the device is in use.

Putting the Pieces Together

Using sensors properly requires multiple layers of understanding. There needs to be an understanding of how the base sensors work, how to fuse the data from these sensors to create meaningful information, how to create specialized functionality based on application, and sensor characterization to truly optimize performance.This process is complicated to say the least. In order to simplify it, CEVA implements a large and robust library of sensor fusion functionality in our MotionEngine solution to give your devices the best performance.

CEVA's MotionEngine is to sensor fusion technology what In-n-Out is to fast food burgers (or your regional favorite). Just like a burger, the ingredients (accelerometer, gyroscope, magnetometer, ToF, etc) can be purchased separately. They can be prepared and the pieces put together in-house. Will making one at home satisfy your burger/product needs? Maybe. But it'll take a lot of time and effort even if it does.

Alternatively, by trusting in CEVA's nearly 20 years of sensor fusion expertise, you can get consistent, accurate results with significantly less time and energy. With our comprehensive sensor characterization tools, our solutions are sensor-agnostic – we assure great performance no matter the sensor vendor. We've also spent the time developing specialized functionality for various target applications, from XR to robotics to remote controls to hearables (and more). No matter your application, we're confident we have the secret sauce and methodology to give your end customers the best possible experience.

 Back