Google Maps Hackathon Prize Win and a New AR Tree Walk Guide

The Geospatial API is a great technology that made the Recycling Trashcan AR Map possible. Some amazing news is that the project won a Google Maps hackathon prize (the Geospatial API is part of the Google Maps Platform). I purchased a GoPro MAX 360 camera with the prize money and used it to record some 360 content. When I visited GDG San Francisco for a lightning talk series of LLM startups (AI SuperShowcase Show and Tell), the GoPro MAX stopped working, so I’ll need to look for another one, but that’s a different story.

Fresno State students were working on redesigning the campus tree walk guide. The new guide contains indigenous or drought-tolerant species, as well as several green spaces. These are all outdoor spots similar to the recycling trash cans and represent an ideal use case for the Geospatial API. I started working on an AR application and companion website for it, and I signed up for another hackathon to keep myself accountable to deadlines.

I was able to use even more Google Maps API features: besides custom pins, I numbered the stops and connected them with lines on the companion website. I wanted the mobile app to appeal to younger generations, so I introduced some gamification: the AR app utilizes Google Play Games, allowing users to receive experience points (XP) and unlock achievements as they visit the tree walk stops.

(Note: The screenshots below were captured while testing the AR mechanics and Geospatial API in my apartment complex parking lot rather than on-site at the CSU Fresno campus.)

On top of that, I wanted to use some EAP (Early Access Program) features. One of these provided segmentation labeling of the camera view. I used that segmentation to identify green vegetation and developed a “virtual watering” feature that yields extra experience points for the user. Another step up compared to the Recycling Trashcan app is that the Tree Walk app is bilingual and can speak and display text in Spanish.

Here is a demo video showing the app in action, including navigation and the semantics-aided virtual watering feature:

As part of the hackathon prize, I had the opportunity to have a one-on-one meeting with a Google Maps engineer. He was very receptive as I shared my use case and feedback, including my experience with the segmentation EAP API. This EAP feature was experimental and has been removed from the current release of the app. Also check out the companion website, which contains 360 photos of the stops as well.

The good news is that eventually, these apps could be part of a sustainability curriculum, and hopefully, I’ll also be able to integrate them with official Fresno State University content so they get more exposure and interest.

There are several future tasks I’d like to involve students in:

1. Just as we haven’t mapped out all the recycling trash cans, the tree walk is not finalized either. The sustainability students need to meet with the arborist to clarify some uncertain spots, and we may also create walk variations for families with small children or seniors.
2. The current app is Android-only. I’d be happy to recruit students to rewrite the app in Unity. The Geospatial API has a Unity plugin; however, it doesn’t have the segmentation EAP feature. Another multi-platform avenue would be to utilize Flutter with SceneView Flutter. Since Flutter Forward, we know that Flutter is no longer limited to 2D, and 3D opens up possibilities for AR/VR as well. Check out this Outpost Game Codelab to see the rich shaders it uses. SceneView is a maintained Kotlin version of the archived Google SceneForm project. Thomas Gorisse keeps it maintained, and hopefully one day the Flutter plugin will come to fruition.
3. There are other issues, like Hmong language support and drinking fountain locations, and we could possibly even integrate the recycling trash can locations as an optional overlay.

I cannot wait to collaborate with students.