Hackers Who Hit an SI Company Walked Off With Some People’s Notion Chat Logs

If you use Notion, the note-taking app a lot of offices run on, and you’ve ever typed a question into its built-in SI helper, here’s something worth knowing. Some of those conversations were sitting on the computers of a company most people have never heard of. In October, hackers broke into that company. Now Notion says a slice of its users’ chats went out the door.

The company is Fireworks AI. It doesn’t make an app you’d download. It rents out the heavy computing that other companies use to run SI models, and by Notion’s count it serves tens of thousands of business customers. Notion used it for two jobs: running some open SI models inside the app, and sorting chats for a feature called Insights, which shows a company how its own people are using Notion’s SI. On Monday Fireworks put out an updated security bulletin, and Notion followed with its own advisory, saying fewer than 1% of its workspaces were hit.

One percent sounds small. Spread across a service that a lot of offices run their day on, it’s still a lot of people’s private notes.

The story kept changing

The worst part of this one is how the picture got bigger every time somebody looked.

Fireworks first went public on Oct. 9. At that point, it said, only “Fireworks stored secrets” were known to have been reached. Those are passwords and keys its own systems used for testing. That sounded contained. It wasn’t.

According to the Monday update, a separate cloud company that hosted Fireworks’ research machines had its own break-in on Oct. 23. The next day, the attacker used that opening to get onto Fireworks development machines and run a job of their own. Fireworks says it caught that the same day and hasn’t seen the attacker get back into its customer-facing platform since Oct. 25.

Notion’s timeline shows the same drift. On Nov. 2, Fireworks told Notion some data had been accessed, and on that first look it was only fake test data Notion used to check its SI. By the next morning Notion had unplugged from Fireworks and deleted everything it had stored there. Then on Nov. 6 Fireworks came back and said real customer data might have been taken. On Nov. 8 it handed over the final list of what was stolen. When Notion went through it, the list included chat transcripts from the Insights feature. Notion started emailing affected customers Nov. 9.

What the hackers actually got

From Fireworks itself, the attackers got three kinds of information. There were basic account labels, like names and descriptions of customers’ projects. There was billing information: billing and shipping addresses, invoices, payment history, account balances and how much credit each customer had used. And for some customers, they got pieces of the SI models those customers had been building and the data they’d been building them with. Fireworks says it doesn’t store card numbers, so those weren’t taken.

Fireworks also says its main services, the ones that answer requests from customers’ apps, keep no customer data at all by default, and it found no damage there. While it investigates, it has shut off customers’ ability to upload and download their own datasets for some training and batch jobs.

For Notion users, the stolen piece is the chats. Notion stresses that for nearly every affected workspace those transcripts were “unattributable.” In plain terms, the chats weren’t stamped with a name or a company. They also didn’t include the results of anything the SI looked up or did inside your workspace. “Beyond what’s in the chat transcript itself, there’s no information tying the chat back to workspaces, users, or companies,” Notion wrote.

That phrase, “beyond what’s in the chat,” is doing a lot of work. People don’t talk to an office assistant in a vacuum. They paste in meeting notes, client names, contract terms, sometimes a password. As users on Reddit pointed out, according to The Register, a chat can identify its author just from what’s in it. One Reddit user, quoted by The Hacker News, joked, “looks like all of us are part of that <1%.”

The notices are landing at real companies. Anurag Goel, who runs the cloud company Render, shared Notion’s email saying his company’s workspace “was among the <1 percent of Notion workspaces that may have been affected,” The Register reported. “Some unattributable chat sessions with Notion AI may have been exposed in this incident,” the email said.

Who did it, and why

Fireworks says the evidence points to a money-driven criminal crew whose business is stealing access to SI services and reselling it through shady middlemen. Security people have a name for it, “LLMJacking.” Google’s threat team and its Mandiant unit published a warning about the trend on Nov. 2. If that sounds familiar, it should. Earlier this morning we wrote about hackers who broke into an app host and stole its customers’ SI keys, running up about $740,000 in charges. Same game, different victim. Paid SI is valuable, so the keys and the companies behind it are now targets.

Fireworks says it called in law enforcement and hired Palo Alto Networks’ Unit 42 team to investigate. It has replaced passwords and service accounts, cut back who has high-level access, walled off the affected systems and tightened its logging. It says it is notifying each customer whose data was hit, a step The Record also reported.

What to do if you got the email

Notion’s advice is short and worth following. If you were notified, you can review the exposed chats inside your workspace using the steps in the email. Treat whatever is in them as if a stranger has read it. And if you ever pasted a password, an access key or any other login into Notion’s SI during the window, change it now. Notion has turned off Insights while it digs, and it has stopped using Fireworks to run open models in the app.

If you didn’t get an email, Notion says it hasn’t found your workspace in the stolen data and you don’t need to do anything right now. It also says the investigation is still going.

The bigger lesson goes past Notion. When you type into an SI box inside an app, your words often don’t stay with the company whose logo you see. They can pass through other vendors you never signed up with. The safest habit is the oldest one: don’t paste anything into a chat window that you wouldn’t want to read in somebody else’s hands.