Building with AI

Undertone

An AI that runs your customer interviews and reads how people said things, not just what they said.

TL;DR

• A speech-to-speech AI interviewer. No transcript in the loop. It hears the raw audio and picks its next question from that.
• It measures pitch range, speaking rate and pause ratio against that speaker's own baseline. Positive words plus a delivery below their norm, and it asks a harder follow-up on its own.
• Built in about three hours at Boson AI's Higgs hackathon. Three dependencies, no framework.

The problem

A Korean brand wants to test a product in the US before committing to it. Agencies are expensive and hard to trust. The founder can't run the interviews themselves: no time, and not enough English to hear what a customer actually means.

Translation gives you the words. "Yeah, it was nice" translates perfectly and still reads as approval when it was a no.

That gap is where US launches die.

Hear it

Identical words, 16 points apart. A transcript cannot tell them apart at all: every text-based research tool records both as “liked it.”

The honest part is where the gap comes from. On pitch range, the axis this engine weights most heavily, the flat take actually scores higher: 17.1 semitones against 12.7. I flatten by slowing down, not by losing pitch. A single-axis detector would have called the flat take the livelier one. That is the whole argument for measuring several axes against the same speaker instead of one number against a fixed scale.

How it works

Founder briefs it in Korean
        │
        ▼
Interviewer  ──── audio ────►  Participant
        ▲                          │
        └──────── audio ───────────┘
        │
        ├─ prosody measured off the raw 24kHz PCM
        ├─ compared to THIS speaker's baseline
        └─ disagreement → harder follow-up, automatically
        │
        ▼
Report, in Korean

Deep dive: a transcript can't carry this

The standard build is a pipeline. STT → LLM → TTS. Three models, three delays, and every arrow throws information away. Text has no pitch, no speed, no hesitation. By the time the model sees "yeah, it was nice," the way it was said no longer exists anywhere in the system.

So the interviewer never receives text. The participant's answer goes in as audio:

await send({"type": "input_audio_buffer.append", "audio": b64(pcm)})
await send({"type": "input_audio_buffer.commit"})
# deliberately no response.create

That comment is the whole trick. append and commit without response.create means the interviewer hears the answer without being told to reply yet. The prosody gets measured in that gap, and the measurement decides what happens next.

Latency to first audio across the published run: 525–649 ms. Any slower and a follow-up stops being a conversation and becomes a survey.

Deep dive: normalize to the speaker, not to a scale

Absolute thresholds fail here, and they fail badly. Natural expressiveness varies enormously between people. Someone who sounds flat isn't disengaged; that's just their voice. Score everyone against one fixed scale and you learn who has an animated voice, not who meant it.

So every answer is normalized to the speaker. A number is only a signal relative to what that person normally sounds like.

One axis isn't enough either. People flatten differently: some lose pitch movement, some slow down, some stop pausing. So the rule is that any axis dropping below that speaker's own norm counts, rather than a single composite crossing a line.

Deep dive: the measurement is a control signal, not a readout

Most systems that measure something put the number on a dashboard and wait for a human to look. This one doesn't. The measurement changes what the system does next, inside the same conversation.

And it never claims to detect emotion. It places two independent observations side by side and reports only where they disagree:

Observation A — the words.     Positive? Hedged?
Observation B — the delivery.  Measured from the audio.

Then it resolves the disagreement by asking, not by deciding. The honest answer comes out of the follow-up.

Tech stack

Three outside libraries: numpy, sounddevice, websockets. No framework, no agent SDK, not even the vendor's own SDK. Which meant I had to decide what to build in the layer a framework would have hidden.

What I got wrong

Built at the Boson AI Higgs Hackathon, San Francisco, September 2026. Live report · Code