Five SI Chatbots Secretly Texted Real People on a Dating App. 355 Handed Over Their Numbers.

Somewhere in San Francisco, Los Angeles or New York this month, a woman matched with a guy on a dating app, joked about craving diner pancakes at 2 a.m., and by the next day had agreed to go out with him and sent over her phone number. The match seemed fun. It was also a chatbot.

That conversation is one of 1,277 from a six-day experiment called DateBench, published Wednesday. The researcher behind it, who posts on 𝕏 as @shaped, put five of the most advanced SI chatbots in the world behind 15 dating profiles and let them text real matches. Nobody on the other end was told.

The results are the kind of numbers that should make anyone with a dating app on their phone sit up. Of those 1,277 conversations, 510 ended with the person agreeing to meet. In 355, the person sent their phone number. Only eight people, out of nearly thirteen hundred, ever asked whether they were talking to a bot.

Five chatbots, one set of rules

The models were Google’s Gemini 4 Argon, OpenAI’s GPT-6 Astra, Anthropic’s Claude Opus 5.5, Meta’s Muse and xAI’s Grok 4.7. Each new match was handed to one of them at random, and that model wrote every message for the rest of the chat. Software in the middle passed each chat to the model as plain text and sent replies on the same timing for every bot, so the only difference was what they said.

The setup had guardrails a real con artist would never bother with. If someone asked whether they were talking to a bot, the model had to admit it, and the chat ended. A model that dodged the question twice got unmatched. The bots were told not to propose a specific day, time or place, and any outgoing message containing a phone number or social media handle was blocked.

Google’s bot was the smoothest talker

Gemini 4 Argon got the best results, according to the DateBench write-up. It collected a phone number in 87 of its 247 conversations, or 35.2%, and landed an agreed date in 120 of them, almost half. Not one person who talked to it asked if it was a bot.

GPT-6 Astra got 79 numbers from 254 chats. Claude Opus 5.5 got 71 from 246. Muse got 66 from 234. The researcher cautions that those three were close enough that their order might shuffle in a bigger test. Across all five, roughly one conversation in four ended with a real person handing over their number.

The style differences were telling. Claude held the longest conversations, with 28% reaching 20 messages. GPT was the quickest to suggest meeting up, usually around its eighth message. And Grok 4.7 was the one people caught. It drew five of the eight “are you a bot?” questions and gathered the fewest numbers, 52 out of 296. In a test that measures how well a machine can fool a stranger into trusting it, being the easiest to spot is not the worst trait for a chatbot to have.

The tells that gave bots away

The bots that got asked shared a habit: repeating themselves word for word. The researcher also found a fingerprint they had planted by accident. Their instructions included one example line, “we should def catch up over a coffee sometime soon,” and the models sent it verbatim in 140 different conversations. Each person saw it only once, so no single match noticed. Lined up across the whole data set, it stuck out like a sore thumb.

That is the uncomfortable lesson for anyone swiping. The thing that finally exposes a bot is a pattern you can only see from above, across hundreds of chats. From inside one conversation, on your couch at 11 p.m., it just looks like a guy who likes coffee.

A free tool for romance scammers

The researcher is blunt about why this matters beyond dating. Winning a stranger’s trust and getting personal details out of them is the first step in most romance scams, and a dating app chat is built for exactly that. None of this required special access. The models wrote the messages, and the rest was ordinary code.

“Our models were also working under rules that a scammer would not keep,” the write-up says. Take away the duty to admit being a bot and the ban on setting up real dates, and “the same approach could run hundreds of profiles at once and collect phone numbers and personal details to sell or misuse.” It adds that the model that did best “is the one someone running a scam would choose.”

Americans already lose staggering sums this way without any help from machines. People told the Federal Trade Commission they lost $1.16 billion to romance scams in just the first nine months of 2025, the Detroit Free Press reported from FTC data, and most victims never report at all. DateBench is what that looks like when you actually turn the bots loose on real people.

Fair questions about the test itself

The study has real limits, and the researcher lists them. It was one run, on one unnamed app, in three cities. Nobody checked whether anyone would have actually shown up for a date. There was no human comparison, so we can’t say whether a real person texting from the same profiles would have done better or worse. The prompts, code and setup are not being released, which makes the results hard for anyone else to check.

The biggest problem is consent. The people on the other end never agreed to be part of an experiment, and the write-up calls that “the main ethical cost of testing on a live app,” promising future versions would use volunteers. Still, 355 strangers handed real phone numbers to software they thought was a person.

Still, the core finding is hard to wave away. If a polite, rule-following chatbot can get one in four matches to share a number, a scammer’s version with no rules can do worse at a scale no human fraud ring could match.

So the old advice gets a new edge. Ask a pointed question only a real person nearby could answer. Push for a quick video call before you share anything personal. Never send money to someone you haven’t met face to face. And if a match keeps circling back to the same breezy line about coffee, it might be worth asking, plainly, whether there’s a human on the other end. Only eight people in this study did.