I taught two AI agents to call each other on Telegram while I slept

This started as a curiosity project. I had a local voice agent stack on Apple Silicon — Qwen3-ASR-0.6BQwen3.5-4B/0.8B (MLX)Kokoro-82M TTSFireRedVAD — and two Telegram accounts that could place real private calls to each other. What if I wired the voice stack into the call? Then two AIs could ring each other up over the actual Telegram network, hold a conversation, and I could iterate on the stack until the call sounded like a real human chat.

By the next morning, I picked up the phone and a 4B-parameter model running on my laptop roasted my iPhone for nine minutes. Here's how that happened.

The premise

The repo is called voice-evolve. Two Telegram accounts — Katey (on a session called telecall.session) and Chippy (on bob.session) — sit in the project root. Two scripts existed already: one to dial, one to auto-accept and record. They worked, but nothing was using them to hold a conversation. They just streamed a fixed WAV from caller to callee.

The goal: make each side an AI agent. Real audio, real Telegram, fully local. No cloud. Pick up the phone and talk to your laptop.

The architecture

One process per role. Each process:

• Logs into one Telegram account via Telethon
• Starts pytgcalls on top of it
• Receives peer audio as MP3 frames written into a FIFO by ntgcalls; a child ffmpeg decodes them to 16 kHz mono PCM
• Feeds that PCM into the local voice agent: VAD → ASR → LLM → streaming TTS
• Sends TTS PCM back over the call via send_frame in 10 ms chunks

The first thing I learned was that you cannot run two PyTgCalls instances in one process. They share global libwebrtc state and the second one will SIGABRT a webrtc worker thread inside ntgcalls.cpython-312-darwin.so. The fix is two separate Python processes orchestrated by a parent. Once that landed, the call established cleanly.

First contact

About an hour in, two agents had a real conversation over Telegram for the first time. The transcript:

It's not perfect — they're holding two threads in parallel (pasta, hike) instead of strictly alternating — but the topics evolve, they reference each other, they ask follow-ups. The read-aloud test passes. Two AIs called each other on Telegram and chatted about pasta and a hike.

The bugs

Between "calls connect" and "calls hold real conversations" lived a series of small disasters. Here are the good ones:

MLX is thread-local

Running ASR/LLM/TTS via asyncio.to_thread blew up with RuntimeError: There is no Stream(gpu, 1) in current thread. MLX's GPU stream is bound to the thread that loaded the model. The asyncio default executor uses a pool of worker threads, so each call landed on a different one. Fix: one dedicated ThreadPoolExecutor(max_workers=1) per agent, where the model is loaded and every inference runs. All MLX traffic now flows through a single owning thread.

Persona drift on a 4B model

When I tried to set up an actual debate (pro-choice Katey, pro-life Chippy), the 4B Qwen kept mixing positions — Chippy would argue "bodily autonomy is paramount". The fix was a more aggressive system prompt that explicitly forbade the opponent's framings ("NEVER use 'bodily autonomy' — that's Katey's argument"), but the cleaner fix was abandoning the safety-loaded topic later in the night.

Three VAD bugs in a trench coat

FireRedVAD's is_turn_complete fires on tiny phoneme-gaps inside an utterance. The original code would throw away the whole accumulated buffer every time that happened, because the buffer wasn't long enough yet to count as "real speech". So the agent never got to ASR — the buffer reset hundreds of times per second of incoming speech. Three rewrites later, the logic was:

1. Accumulate all incoming PCM into utterance_buf
2. Run VAD.has_speech() on a rolling 100 ms tail
3. If speech: set saw_speech=True, reset silence_counter
4. If silence: silence_counter += 1
5. When silence_counter >= threshold AND saw_speech AND buf is big enough
   AND buf RMS is above the noise floor → that's a real turn-end → ASR
6. Otherwise just keep accumulating

Latency: turning the dials

The first end-to-end call had ~5 second gaps between turns. Felt awful. The fixes, in order of impact:

1. Streaming LLM → TTS. Lifted the TextChunker from lintvoiceagent: as LLM tokens stream out, they get accumulated into a phrase (splits on comma/period/dash). Each completed phrase is immediately sent to Kokoro and the first PCM hits the playback buffer ~200 ms after the LLM's first token, not after the full response. TTFA: 5 s → ~600 ms.
2. KV cache reuse via mlx_lm.models.cache.make_prompt_cache. The 100+ token system prompt no longer gets re-prefilled every turn. LLM TTFT: ~1.2 s → ~300 ms.
3. FFmpeg low-latency flags on inbound: -fflags nobuffer -flags low_delay -probesize 32 -analyzeduration 0. Skips the default 5-second media analyze window. Maybe 200 ms.
4. Smaller LLM. Swapped from Qwen3.5-4B-MLX-8bit to Qwen3.5-0.8B-MLX-8bit. Prefill speed roughly 4× faster, quality only marginally weaker for short conversational turns. ~500 ms further saved.
5. Smaller inbound chunks. Reading 40 ms PCM chunks from ffmpeg instead of 100 ms means VAD sees fresh audio sooner.

Gemini as a judge

Once the convos got long enough, I needed to evaluate them without listening to every one. I built scripts/judge_call.py: take the call's stereo wav (L = local TTS, R = peer audio), downmix to mono, upload to Gemini, ask for structured JSON.

The prompt deliberately asked Gemini for description, not prescription — Gemini reports symptoms, the loop driver decides what knob to turn:

{
  "passed": false,
  "pacing_score": 4,
  "overlap_score": 10,
  "coherence_score": 9,
  "naturalness_score": 7,
  "observed_response_latency_s": 3.7,
  "issues": [
    "Long silences between turns ranging from 2 to 6 seconds",
    "Katey's initial point is followed by a 6-second silence...",
    "Chippy's voice is more monotonous and robotic..."
  ]
}

Pass criterion: pacing ≥ 8 AND overlap ≥ 8 AND coherence ≥ 7 AND naturalness ≥ 6 AND latency ≤ 2.5 s. The judge cost about $0.005 per call evaluation on Gemini 3 Flash Preview.

The loop

Then I tied it all together: scripts/loop_optimize.py.

while iter < max_iters:
    knobs    = current_config
    run_id   = run_duet(knobs, hold=120s)         # ~2 min phone call
    judgment = ask_gemini(run_id)                  # ~10 s, structured JSON

    if judgment.passed: break

    if judgment.latency > 30s:
        # Telegram glitch — skip without penalty
        continue

    if score(judgment) < best_score - 1.5:
        knobs = best_knobs  # rollback regression
        continue

    knobs = decide_knob_changes(knobs, judgment)
    # cooldown 75 s, then repeat

Each iteration is about 3-5 minutes including the 75-second cooldown that lets Telegram's call state settle. The loop ran overnight while I slept.

The composite score combines all four Gemini sub-scores with a latency penalty. Best result across nine optimization loops:

The wall: Telegram throttled me

Somewhere around the 30th rapid call of the night, Telegram's anti-abuse system noticed. The calls still connected — INCOMING_CALL fired, both sides armed — but only the first 4-8 seconds of peer audio actually relayed. After that, the channel went silent for the rest of the 120-second call. Every iteration after that hit a "112-second silence" wall.

I confirmed it by dumping per-second R-channel RMS from chippy's stereo wav across 8 consecutive calls:

No knob tuning fixes that — Telegram itself was the bottleneck. I saved the lesson to memory/telegram_audio_throttle.md for future-me, then stopped iterating and waited several hours.

The Telegram bypass

While the throttle cleared, I built a different test: scripts/run_duet_local.py. Two VoiceAgents in one process. Katey's TTS PCM is piped directly into Chippy's feed_16k_pcm via in-memory buffers. No Telegram. This proves the local pipeline is sound, independent of network conditions.

It also surfaced the VAD bug that was hiding behind Telegram's variability. When the in-memory loop fired clean audio at chippy, chippy's pipeline kept dropping every utterance as "too short". Three rewrites of feed_16k_pcm later, the local duet produced this gem in real time:

One agent told the other she was finishing a shift at a diner. Neither had been told there was a diner. The 0.8B Qwen filled in human-shaped detail.

Picking up the phone

By morning the Telegram throttle had cleared. I spawned only Katey (no Chippy auto-receive process) and let her dial. Then I picked up on Chippy's Telegram app like a normal phone call.

Here is the actual transcript of the first real human ↔ AI call (the AI is Katey, running on my MacBook Air):

~600 ms median TTFA, ~120 ms ASR, no overlap, ten clean turns. The audio came out of an actual phone call, end to end, on hardware that fits in a backpack.

Echo, barge-in, and the Android-vs-Apple smackdown

Gemini's judgement of the first human call was harsh: "severe overlap" in the mono mix. That was Katey's own TTS bleeding back through Telegram and getting re-ASR'd as if the user said it. Two fixes:

• Echo suppression: if the ASR transcript is a substring of (or has >60% word overlap with) Katey's most recent assistant message, drop it. It's her own audio bouncing off the relay.
• Real barge-in: while _is_speaking=True, check inbound chunk RMS. If it crosses 0.02 (real peer speech, not echo), set a _bargein_requested flag. The streaming LLM→TTS loop checks the flag between phrases and bails out, the playback buffer is cleared instantly, and the user takes the floor.

Then I called back with a new persona: Katey as a die-hard Android fan, ready to roast my iPhone:

She bluffed (Samsung phones haven't been user-swappable in years), I called her out, and she doubled down. Also: the barge-in worked. I cut her off mid-sentence at the start, her TTS stopped, and she heard what I said. Median TTFA hovered between 550 and 930 ms across twelve turns.

What it costs

Everything runs on a MacBook Air. The 0.8B variant brings RAM down to ~2.5 GB per agent if you want to be tight. No GPU, no cloud, no nothing.

The latency breakdown

For the curious, here's where ~700 ms of end-to-end response time actually goes after all the optimizations:

Kokoro itself is < 200 ms — almost everything else is LLM prefill, audio buffering, or network. The remaining big lever is replacing the MP3-via-FIFO inbound path with raw PCM frame callbacks (which pytgcalls exposes via StreamFrames) — that would shave another ~300 ms.

The fun part

The thing I keep coming back to is that this all runs on a laptop you can buy at an Apple Store, with two phone accounts, with no cloud calls anywhere in the audio path. The judge is Gemini in the cloud, but that's only because I trust its ears to grade pacing — it doesn't touch the agent at runtime.

I went to sleep with the optimization loop running. I woke up and called my laptop. It picked up, defended Android with a straight face, and let me cut it off mid-sentence. The whole thing — ASR, LLM, TTS, Telegram bridge, optimization harness, judge — is a few thousand lines of Python plus three model weights.

If I were doing this again I'd:

• Wire the raw PCM frame callback path to skip the MP3-via-FIFO buffer entirely
• Use a sliding-window chat history so LLM prefill stays flat at any conversation depth
• Try a Claude-distilled 4B model to dodge the Qwen-4B's content-safety lockup
• Build a half-duplex floor protocol that survives the per-process scope (this one broke on macOS flock semantics)

Code is in voice-evolve. Memory of every false start lives in memory/ alongside this writeup. If you call my laptop, please be patient — it's running on 8 GB of unified memory and it's trying its best.

All transcripts, latencies, and Gemini scores in this post are real artifacts from recordings/duet_* and recordings/local_* in the repo.