Things Ken made.

Collection of software, hardware, and visualizations that Ken's made.

Visualizing spatial data with AR

(I'm moving all my posts from Blogger to this site. You may see some missing links in the process.)

f:id:kenkawakenkenke:20190106213502p:plain

As part of our holiday cleanup, I spent some time optimizing the Wifi layout of our house. To help with this, I built an AR app "AR Sensor" to measure and visualize various spatial data, such as the Wifi signal strength:

f:id:kenkawakenkenke:20190105194138g:plain

The above gif shows what the output of AR Sensor looks like in our house; it shows how the Wifi signal strength received by the smartphone gets better (goes greener) as I go up the stairs (because the router is on the upper floor), and all of the historical measurements are "recorded" in the space.

The way it works is very simple. It uses the positional tracker provided by ARCore (Android's default AR framework) to constantly track the 3D coordinates of the phone inside the house, whilst also measuring other data provided by the smartphone's sensors (like Wifi signal strength). As you move the phone about, it accumulates the measurements which you can view in AR:

 

f:id:kenkawakenkenke:20190106222022g:plain

 This lets us "see" sensor data in its spatial context, letting us better understand its 3D characteristics and distribution in the environment.

What else can it do?

I've added support for 5 different sensors in the first release of AR Sensor: "Wifi Signal Strength", "Bluetooth LE Signal Strength", "Magnetic Field", "Barometric Pressure", and "LTE (Mobile Cell) signal strength". Let me explain how each of these are interesting.

Wifi Signal Strength

Here's a fuller version of the gif posted above. It helps visualize how a flight of stairs or even a single wall is enough to bring the signal (visually) down. I've since moved the Wifi router to the top of the staircase, which let it cover the entire house.

 Bluetooth LE Signal Strength

Similarly for bluetooth, the app lets you visually see how even a piece of aluminimum foil results in drop in Wifi signal strength (not to say that it actually interferes with communication).

Magnetic Field

Using the magnetometer, we can measure the direction and strength of the magnetic field at points in space. Inside a typical room this is fairly unappealing; the particles drift towards North slightly towards the ground (i.e., towards the magnetic north pole):

f:id:kenkawakenkenke:20190105211218g:plain

but when placed near a magnet, it can create a fantastic visualization of the magnetic field around the magnet:

 or around two magnets, the field lines going between the magnets:

Barometric Pressure

Not very interesting, it's basically just the height. Let me skip.

LTE Cell Signal Strength

This visualizes the signal strength of the LTE cell that the phone is connected to. This is great for finding seats with cell connection in those underground bars with crappy Wifi.

 It's a pretty interesting experience walking around the city visualizing little spots of bad connection (e.g., under playground slides).

f:id:kenkawakenkenke:20190106225308g:plain

Various sensors + Arduino + Android

I only found out a few weeks ago that Android doesn't need much setup to talk to Arduino these days; all it takes is a simple USB host cable, and we can get an app talking via serial quickly.

I setup a simple gas sensor TGS2600, to measure the gas density while boiling water on the gas stove. See how the gas density increases as we go near the ceiling:

f:id:kenkawakenkenke:20190105223758j:plain f:id:kenkawakenkenke:20190105224005p:plain

Think about all the other sensors we could attach to the phone, like humidity sensors, CO2 sensors, pollen/dust sensors, etc! I'd love to hear about ideas for things to measure in the comments.

Final thoughts...

For about 10 years I've wanted to create a tool for measuring things and plotting them in 3D, but never really had the time to do implement the positional tracking alternatives that existed at the time, such as placing QR codes all around the house, tracking myself with a Kinect, and writing my own SLAM. In writing this app, I was astounded by how I could just rely on ARCore (and probably the same for ARKit) so that the end-to-end process of positional tracking, sensing, and visualization can all fit into one smartphone, with very minimal code. If you've had ideas that rely on positional tracking, now's probably a great time to implement it!

Get it on Google Play