Gemini API Developer Competition: Building OpenMMPA on Full Android Watches

At Google I/O 2024 on May 14 at the Shoreline Amphitheatre, Google unveiled the Gemini API Developer Competition. It was an ambitious global challenge calling on developers to build groundbreaking applications powered by the Gemini API, featuring over $1 million in prizes across categories like Web, Android, Flutter, and Firebase.

The centerpiece of the competition—and the talk of the developer community—was its grand prize: a one-of-a-kind, fully custom 1981 DeLorean converted into an electric vehicle (EV).

(The trunk of the custom electric DeLorean exhibited at Google I/O 2024, featuring an illuminated battery bay cover and special Google Gemini pens.)

Seeing the DeLorean exhibited live at Google I/O was unforgettable. Built in collaboration with DeLorean Midwest, this retrofitted masterpiece replaced the original combustion engine with a modern electric drivetrain while preserving the iconic stainless-steel exterior and gull-wing doors. Under the transparent, illuminated cover in the rear, you could admire the high-voltage battery pack alongside custom-branded Gemini pens and gear.

(The frunk of the electric DeLorean at Shoreline Amphitheatre.)

The competition ran through the summer, with the core development sprint taking place between mid-June and the late summer submission deadline, followed by the People’s Choice voting and final announcements in October. With Flutter and Android categories up for grabs, I set out to build something that tackled a major trend in tech.

The Inspiration: The “AI Pin” Craze vs. Full Android Watches

Mid-2024 was marked by peak hype—and swift reality checks—surrounding dedicated AI hardware. Devices like the Humane AI Pin ($699 plus a mandatory $24/month subscription) and the Rabbit R1 ($199) promised a post-smartphone computing paradigm where ambient AI companions would replace screens.

However, once developers and hardware reviewers got their hands on these gadgets, the curtain was quickly pulled back:

1. Both the Humane AI Pin (CosmOS) and the Rabbit R1 (Rabbit OS) were essentially running Android (AOSP) under the hood.
2. The specialized “AI companion” applications were standard Android APKs running on top of mobile system-on-chips (SoCs).
3. The hardware struggled with battery life, thermal throttling, and poor network responsiveness when tethered to cloud-only processing.

This got me thinking: why should users spend hundreds of dollars on fragile, single-purpose proprietary clips and pendants when a far more capable, mature hardware platform already exists?

Enter the Full Android Watch (FAW).

I was first introduced to the FAW movement by Roman Jaquez, a well-known Flutter Google Developer Expert (GDE), international speaker, and GDG lead organizer. Roman had been experimenting with porting Flutter applications to wrist-worn devices, demonstrating how well Flutter’s responsive rendering engine adapts to miniature screens.

What is a Full Android Watch (FAW)?

It is crucial to distinguish a FAW from a standard smartwatch:

• Wear OS is Google’s dedicated smartwatch operating system. While polished and battery-efficient for basic notifications and fitness tracking, it operates under stringent API limitations, background execution restrictions, and a constrained application sandbox.
• Full Android Watches (FAWs), by contrast, run regular, full-blown Android (AOSP, typically versions 8 through 11). They feature a customized watch launcher and power-management tweaks to accommodate the wearable form factor, but at their core, they possess the full Android runtime and API surface.
• Hardware capabilities:
  • Available in both circular and rectangular form factors (from manufacturers like LEMFO, Kospet, and Rogbid).
  • Standalone cellular connectivity via an integrated micro-SIM or nano-SIM slot, allowing direct 4G LTE data connections and standard cellular phone calls without tethering to a smartphone.
  • Built-in microphones and loud speakers for natural two-way voice conversations.
  • Dual camera system: unlike typical smartwatches, FAWs frequently feature two cameras—a front-facing camera for selfies and video calls, and a side/wrist-facing camera positioned along the watch bezel. This side camera is uniquely suited for pointing at real-world objects and capturing environmental photos directly from your wrist.
  • Full Google ecosystem (Google Play Store, Google Play Services) on US and global variants.

The community gathered at the FullAndroidWatch.org forum is the premier hub for anyone getting into this space, offering invaluable support, custom watch launchers, firmware releases, and device-specific optimizations for various FAW models.

With a microphone, speaker, cellular antenna, dual cameras, and a complete Android environment strapped to your wrist, a FAW was already everything the AI pins wished they were. All it needed was an intelligent, multi-modal AI software stack designed specifically for it.

The Solution: OpenMMPA (Open Multi-Modal Personal Assistant)

To bring this vision to life for the competition, I created OpenMMPA (Open Multi-Modal Personal Assistant).

OpenMMPA is an open-source, multi-modal AI companion application engineered to run smoothly on edge form factors—starting with Full Android Watches, but fully responsive and operational on standard Android phones, tablets, and iOS devices.

1. Cross-Platform UI in Flutter

Using Flutter (scaffolded via Very Good CLI), I developed an adaptive user interface that fits circular and rectangular watch displays as well as larger mobile screens. The UI provides quick-action buttons for voice listening, text input, and camera capture, giving users immediate tactile control alongside hands-free voice interaction.

2. On-Device Local Memory & Vector RAG with ObjectBox

A major pitfall of modern AI assistants is sending every ounce of private conversation history and personal notes to third-party cloud vector stores. For OpenMMPA, privacy and low-latency retrieval were top priorities.

I integrated ObjectBox, a high-performance, embedded ACID vector database designed specifically for mobile and IoT devices. OpenMMPA stores embeddings and chat history locally on-device:

• Prior conversations and user preferences are indexed directly in local storage.
• Before querying the cloud LLM, the app performs a local vector search (RAG) to fetch relevant contextual nuggets.
• This ground-truth context is injected into the prompt, giving the assistant a persistent, personalized memory without sacrificing user privacy.

3. Multimodal Intelligence with Gemini 1.5

The core reasoning engine leverages Google’s Gemini 1.5 Flash and Gemini 1.5 Pro models via the Gemini API (and Firebase Vertex AI):

• Natural Voice Chat: Multi-turn dialogue with high-speed reasoning.
• Vision & Multi-modal Exploration: Tapping into the FAW’s side bezel camera, the user can take a snapshot of an object, document, or sign and ask questions in natural language (e.g., “What kind of plant is this?” or “Translate this sign and summarize its directions”).

4. Dual-Engine Speech Architecture (Local + Cloud Functions)

Voice communication on a wearable must balance speed, network bandwidth, and voice quality:

• Speech-to-Text (STT):
  • Supports offline, immediate recognition using Android’s built-in platform speech recognizer.
  • Can seamlessly offload to a custom Google Cloud Function (STTSFunctions repository) running Google Cloud Speech-to-Text v2 with Chirp. Chirp is Google’s billion-parameter foundation speech model that provides automatic language detection and exceptional transcription accuracy across dozens of languages and accents.
• Text-to-Speech (TTS):
  • Falls back to Android’s built-in TTS engine when offline or conserving battery.
  • Routes through Google Cloud Text-to-Speech via Cloud Functions to deliver expressive, high-fidelity GenAI neural voices for the assistant’s replies.

5. Real-World Tool Integrations

To make OpenMMPA truly useful as a daily assistant, I wired in practical tool integrations:

• Geolocation-based weather reports, sunrise/sunset times, and local conditions.
• Real-time fiat currency and cryptocurrency exchange rate conversions.
• Live web research powered by Tavily API.

The Demo Hardware & Video

The demo device running OpenMMPA in the video below is a LEMFO LEM P—a round Full Android Watch equipped with 4GB of RAM and 64GB of storage. With 4GB of RAM, this wrist-worn device provides ample memory to comfortably run the Flutter runtime, microphone audio capture pipelines, on-device ObjectBox vector database, and camera previews without stutter or memory pressure. For fellow developers and tinkerers interested in the specs, firmware, and community discussions around this device, check out the LEMFO LEM P (4GB/64GB) Support Thread on FullAndroidWatch.org.

Here is a full demonstration of OpenMMPA in action on the LEMFO LEM P, showing voice interactions, camera-based multi-modal visual queries, and real-time response generation:

Project Repositories

Both the client application and backend infrastructure are completely open source:

• Mobile / Wearable Client Application: OpenMMPA on GitHub
• Speech & Cloud Functions Backend: STTSFunctions on GitHub

Reflections and Looking Ahead

Shortly after the competition wrapped up, Google held its Made by Google ‘24 event and launched Gemini Live for smartphones. While big tech has focused heavily on cloud-streamed audio sessions on flagship phones, OpenMMPA demonstrated that developers do not have to wait for proprietary hardware to experience wearable AI.

By combining the open Android hardware ecosystem (FAWs), Flutter’s cross-platform UI flexibility, on-device vector databases (ObjectBox), and Google’s Gemini 1.5 multimodal APIs, we can build private, capable, and truly wearable assistants today.

Future plans for OpenMMPA include adding calendar and contact integrations, local on-device small language model (SLM) fallback for zero-connectivity scenarios, and direct SMS/calling tool dispatch. Special thanks to Roman Jaquez for the early inspiration on the potential of Full Android Watches!