Using Geospatial API for Augmented Reality Maps

Aug 31, 2022

I have shown you before how I used augmented reality technology to provide indoor maps for our tech conference venue (post, followup post) or for some fun experimentation with virtual physics in the form of an augmented reality game.

I have also shown you several dedicated 360 VR websites; for example, see a sample list of them in the Google VR View post. The latest 360 VR project was about sustainability: I wanted to help map out recycling trash can locations for the California State University Fresno Sustainability Club. The sustainability club ordered stickers to label the trash cans, and I created a 360 website to show their locations around campus. When the campus was deserted during COVID, I went out one weekend day to shoot 360 photos and videos of the trash cans. The currently mapped spots are toward the northeast section of campus, clustering near the science building and surrounding areas.

The 360 website is fun, especially with a VR headset; however, I envisioned a much more usable scenario that I described to interested club members and fellow developer Seth Nuzum when I visited the Meta Store opening. Although the website can still guide a student without a headset, it requires several clicks, and the whole experience could be more immersive. The best solution would be to provide an augmented reality map similar to the Bitwise Industries South Stadium conference venue’s AR map (the late BWSS can still be explored via our 360° virtual tour). That app is for indoor use only, and the distances on campus are large enough that even a small directional drift would displace the pins significantly. Such drift accumulates as the phone moves and shakes in the user’s hand. On top of this, the activation of an indoor map requires a specially crafted image, and I would need to label each trash can with a unique activation photo, along with a guarantee that the photo’s orientation would not change. Unfortunately, that is infeasible: trash cans are moved sometimes, meaning the activation image’s orientation could change and the pins could end up all over the place.

However, you may notice that all of the recycling trash cans are outdoors. We know their GPS coordinates, and when students are looking for them on campus, they are also outside. Why couldn’t we take advantage of this and make an outdoor augmented reality map? Before Google I/O 2022, there were multiple blockers to this:

  1. In the past, an augmented reality anchor could not be tied to a GPS coordinate the way we wanted.
  2. We do not know our precise GPS location and, more importantly, our orientation.

I thought about this a lot: it might be possible to create our own wrapper Anchor class with GPS coordinates and some extra back-end logic. A-GPS technology and, more recently, dual-band GPS chips in modern phones allow relatively precise GPS triangulation. What I was not able to solve properly was the orientation of the student and the phone. Compass sensors usually require a strange figure-eight calibration procedure; I cannot feasibly ask users to do that. Even if I asked a user to walk two yards in a straight line, I would get some kind of heading, but regular hand movements would still mess that up big time.

In the fall of 2020, the end of a blog post and a teaser video suggested that Google was actively working on a solution to this problem. The name “Earth Cloud Anchors” hinted at the fact that the augmented reality team was collaborating with the Google Earth team. It is more than that; all of these technologies are involved:

  • ARCore’s AI algorithms
  • Street View data
  • Google Earth 3D models
  • Google’s Visual Positioning System (VPS)

If your app’s target area has Street View coverage, then the positioning and orientation can be even more precise (by leveraging that data to correct orientation and location). Fortunately, the Fresno State campus has Street View data, although I must note that Street View is entirely optional. I implemented an application following a demonstration codelab. The app is already released on the Google Play Store and featured on the Recycling Trashcan 360 website.

Here is a demo video showing the Recycling Trashcan AR Map app in action on the Fresno State campus:

This browser does not support the YouTube video player. Watch on YouTube

Tech demo of the Recycling Trashcan AR Map app using the Geospatial API on the CSU Fresno campus.

I came across two quirks:

  1. My mid-range OnePlus Nord phone started to stutter after running the app for more than a minute in 100°F+ heat.
  2. The framework’s perceived elevation was about 30 yards lower than the actual elevation. The first time I experimented with the app, I could not find my test pins in my apartment complex. I noticed the perceived elevation discrepancy and then found the pins hovering high in the air.

I was contacted by another developer, Darryl Bartlett, who experienced similar problems. I tried multiple workarounds for my use case:

  1. The first workaround was to determine the perceived elevation at the time of activation. I made sure the user was close enough to the nearest pin, and I assumed that the user was at street level—roughly the same level as the pins. Then I calculated an offset based on this and adjusted the elevation of all pins before placing them into the AR scene.
  2. Then I also noticed that the perceived elevation was offset by the same amount at both my apartment complex and the university campus. For the second workaround, I assumed that the framework simply used a different elevation dataset than Google Maps, Google Earth, and other common elevation calculation services. Besides that, I assumed that this unknown dataset yielded a reproducible offset. So I ended up simply hard-coding the offset, which made the source code a little simpler.

Fortunately, Google engineers must have been working on a solution already, because at the end of August Darryl Bartlett drew my attention to Terrain Anchors. This is an API that does not take elevation (only a supplied hover modifier). I successfully tested this new API, and it works well. The perceived elevation is still off by about 30 meters; however, now that I let the framework itself calculate elevation based on the terrain, it matches up with the real world. I have not tested corner cases so far, such as placing a map pin below a tree canopy (a tree canopy bumps the elevation number on Google Earth).

I summarized my journey in a lightning talk not so long ago at the Pacific Region Google I/O Extended Web Edition; here is my slide deck, and here is a video recording of it on the North America GDG YouTube channel. Let me know if you are also working on any Virtual Reality, 360, or Augmented Reality projects.

Comments loading...