DeepL's New SI Translates Your Voice Live and Now Carries Your Tone, Too

Anybody who has sat through a meeting with a live translator knows the problem. The words come through. The person doesn’t. A boss who is clearly fired up sounds like a GPS voice reading a grocery list, and a dry joke lands like a weather report.

DeepL, the German company best known for its translation app, says it has a fix. On Wednesday it rolled out Aure, a new SI model inside DeepL Voice, its real-time speech translation product. The pitch is simple: Aure doesn’t just carry your words into another language. It tries to carry how you said them, your tone, your emotion, where you put the emphasis and how fast you’re talking, and it adjusts as those things change in the middle of a conversation, according to DeepL’s announcement.

The old way took one snapshot

DeepL already had a trick that made its translated voice sound like the actual speaker instead of a stock robot voice. But it had a catch. The system listened to the first few moments of a conversation, took that sample, and used it to shape how your translated voice would sound from then on.

That told it who was talking. It couldn’t tell it how. If you started a call calm and then got excited, or got serious, or said something sarcastic, the translated voice stayed stuck on that first impression. DeepL’s own words: a fixed snapshot “can’t follow a change in emotion, emphasis or pace, or convey delivery that contradicts the words, such as sarcasm.”

Aure keeps listening the whole time. It looks at both the translated words and your original audio together and uses them to decide how each translated sentence should come out. Pause before a big announcement, and the translation should pause too. Speed up when you’re pushing a point in a negotiation, and the translated voice should speed up with you. That’s the claim, anyway.

Why this was hard to build

DeepL was fairly open about the tough parts. Real calls are messy. There’s background noise, echo, bad laptop mics and audio that cuts in and out. Aure has to pick out the speaker’s voice and the rhythm and melody of their speech, what linguists call prosody, while filtering all that junk out.

Carrying a voice into another language is also a lot harder than copying it in the same language. Do it badly, and the translated speech comes out with a thick, exaggerated accent or sounds like somebody else. DeepL says it trained the model to pull apart three things, the identity of your voice, the way you’re speaking, and what you’re actually saying, so each can move into the new language on its own. Then it used reinforcement learning, a training method that rewards the model for good results, to keep the translation accurate and the speech natural, “free of accent,” in the company’s words.

There was one more problem, and it’s the one I like best. DeepL says its voice product doesn’t permanently store meeting transcripts or translations and wipes that data from memory when a session ends. That’s good for privacy, but it means the system can’t study a recording of you from last week to learn how you sound. It has to figure you out on the fly, often before you’ve finished your first sentence.

That also means it has to handle the fact that people don’t sound the same every day. DeepL’s example: someone might sound tired one day, enthusiastic the next, or speak differently under stress. Aure is supposed to tell those apart and match how you sound right now.

Kutylowski’s pitch: keep the person

Jarek Kutylowski, DeepL’s founder and CEO, framed it around something everyone already knows from real life. “We’re constantly adjusting our tone, pace, and expression in ways that can completely change how our words are understood,” he said. “For too long communicating across languages has meant compromising how we express ourselves.”

His goal, as he put it, is “a world where language changes, but the person behind it doesn’t.”

Teams, Zoom, Meet and your phone

You don’t have to turn anything on. DeepL says Aure works automatically in three places: DeepL Voice for Online Meetings, DeepL Voice for In-Person Conversations, and the DeepL API for Voice, which other companies plug into things like customer service lines.

The meeting version works with Microsoft Teams, Zoom and Google Meet, according to DeepL’s product page, and the in-person version runs on iPhone, Android and the web.

There’s a limit worth knowing. At launch, Aure only covers eight languages: English, German, Spanish, French, Italian, Japanese, Portuguese and Korean. DeepL Voice handles more than 40 languages in meetings, and everything outside those eight keeps running on DeepL’s older voice models for now. The company says more languages are coming but didn’t give a date.

DeepL named a few companies already using DeepL Voice, including Pioneer Corporation, food-service giant Aramark, Avendra International and the tech services firm Inetum. The company says more than 200,000 business teams use its products overall and that it has more than 900 employees.

Nobody outside DeepL has tested it yet

For now, all of this comes from DeepL. Outlets that covered the launch, including The Decoder and AI Affairs, laid out the same details from the company’s release. Nobody outside the company has published a side-by-side test yet showing how well Aure actually handles sarcasm, a nervous speaker or a noisy conference room.

That’s the test that matters. Live translation that gets the words right but flattens the feeling has been good enough for checking into a hotel. It’s not good enough for closing a deal, calming an upset customer or telling a joke to a room full of people who don’t speak your language.

If Aure does what DeepL says, the gap between talking through a translator and just talking gets a lot smaller. And for the eight languages it covers, the people on the other end should finally hear the same mood you meant to send.