Tandem

Tandem is a voice assistant that uses your phone for you. You say what you want the way you’d ask a person, and it taps through the apps while it explains each step. If it starts going the wrong way, you talk over it. If you’d rather learn, say “show me how” and it guides you instead.

We built it overnight at the Boson AI x MScAC Hackathon 2026 in Toronto, and came away as runners-up.

The problem

We asked Gemini on a Pixel to add a medium iced capp from Tim Hortons to our DoorDash cart. It told us it can’t do that.

That felt backwards. Developers like me hand coding agents whole features now and come back when they’re done. Meanwhile my grandparents still get through every app one tap at a time, and call us when an update moves a button. The people who struggle most with new interfaces got the least help from AI.

So I set a few rules before writing any code:

• It runs on a real phone, with no laptop plugged in on stage.
• It works in any app, not a list of supported ones.
• You can interrupt it at any point, mid-sentence or mid-tap.
• It never sends, pays for, or deletes anything without asking out loud.

A voice and a pair of hands

The first decision was to split the conversation from the phone work. Boson’s Higgs Realtime 1.5 handles the conversation really well. It hears you, talks back, and handles turn taking in one session, which is why you can cut it off mid-sentence and it keeps up. But it can’t look at a screen. So Tandem gives it eight tools (start or correct a task, look at the screen, confirm, guide, pause, continue, stop, and instant actions), and a separate operator does the tapping through Droidrun’s Portal.

Picking the model that taps

The operator reads the screen’s accessibility tree and picks the next action. I timed four models on a physical Pixel before picking one, mostly on one hard path: finding Night Light, which is buried a few menus deep in Settings.

These were a few runs each, not a proper benchmark. The trade-off was still clear. Saving half a second per step adds up to a few seconds on a long task, and one wrong tap costs more than that. I went with Sol.

Let it hear the words, not a summary

The first version had the voice model fill in structured fields like timer or nightLight, and the server trusted those over what you actually said. It broke in ways that were almost funny. “Stop one timer, keep the other” started a new timer. “Light mode” became Night Light, which then failed verification.

So I deleted that whole layer: the step planner, the special Settings paths, and the tools that read and set values directly. Now the voice model passes your goal in your own words and the operator decides. The only shortcuts left are four native commands Portal can’t do in one call, like exact brightness or volume. It ended up simpler and worked better.

Talking over it

You can correct Tandem mid-sentence by talking over it. That’s the core of the product, and it’s also a race condition. Your correction might arrive while a model is choosing a tap, or while a tap is already on its way to the phone, and an old decision can’t become a new tap.

One limit: a tap that was already sent can still land. Pausing isn’t undo.

Asking before it counts

Sending, paying, and deleting always get a spoken question first. That broke in testing too. I said “yes” and the voice model sent approve: false anyway. Now Higgs STT 3.1 transcribes your answer, and a clear yes or no decides, not the model’s reading of it. It works in Hindi and Hinglish too. Timers and alarms skip the question entirely, since asking every time gets annoying fast.

Done also has to mean done. A tap that went through, a model saying “finished”, and the result actually showing on screen are three different things. Tandem only counts the last one.

Help in a voice they know

This is my favourite part. Family mode uses Boson’s voice cloning. A grandkid opens Settings, picks a language, and reads a short paragraph aloud. After that, when their grandparents ask for help, that’s the voice they hear. It works in Hindi, English, Spanish, French, German, and Mandarin.

Results

Every demo ran live on stage, on the real phone. One question had it pull up Uber, Lyft, and transit and talk us through the prices in under two minutes.

For this write-up I went back and benchmarked the voice side: 20 fresh sessions, each asking for the iced capp, then correcting it mid-task with “actually, make it a large.”

All 20 turned the request into the right phone task with the whole goal intact, and all 20 came back from the correction with “Order a large iced capp from Tim Hortons on DoorDash.” Nothing got dropped or half-merged. The runs use text input, so they leave out speech recognition and the phone itself.

The repository has an installable APK and a one-click deploy, so anyone with an Android phone, a Boson key, and an OpenAI key can set it up in about ten minutes.

What I’d do next

Tandem works for one person on one phone. The bigger version is help that comes from the whole family.

• Learn from the family. A grandkid does a task once while Tandem watches, like booking a refill in the pharmacy app. After that, Tandem does it the same way and explains each step in their voice.
• Call in a real person. When Tandem gets stuck, it rings that grandkid. They see the screen live and can talk over both of them.
• Catch scams before they land. Tandem already reads every screen it acts on and asks before paying. It could also say out loud when a “bank” wants gift cards or a text asks to install a remote-access app.
• Remember the person. “My usual,” “text my son,” which pharmacy. Fewer questions every time.
• Move it onto the phone. With the operator running on-device, screen contents never leave the phone and Tandem keeps working on a bad connection.