Weak model leaks hidden thoughts of stronger siblings

One key opened the back door for the whole family

Here is what happened: frontier models from Anthropic, OpenAI, and Google send their internal chain-of-thought reasoning to clients in an encrypted format. They do this to prevent competitors and researchers from looking under the hood and training their own cheaper models on those thoughts.

However, a group of researchers found an amusingly simple flaw: all models within the same family (for example, different versions of the same base model) used the exact same key to encrypt these thoughts. If you captured an encrypted block from the strongest (and most expensive) model, you just had to feed it back into the weakest member of the family, from which you could extract the information using a simple jailbreak.

Uncovered secrets inside hidden blocks

As models take on more complex agentic tasks, providers are increasingly trying to keep their internal logic under wraps. They don’t just want to sell the result; they want to sell the reasoning process itself, guarding it as valuable know-how.

This hack brought a real security issue. By analyzing 6,708 publicly available agent trajectories, researchers reconstructed over 315,000 reasoning blocks. Inside, they found 704 distinct privacy artifacts, including 62 API keys, 33 passwords, 24 access tokens, and 30 email addresses. Out of those, 64 artifacts appeared exclusively in the reasoning blocks and nowhere else in the visible session.

Security through invisibility hits a wall

Providers patched this specific issue relatively quickly. The actual leak of information regarding training data was likely minimal, because finding out how a model thinks is not the same as discovering what data it comes from. However, the leak of private keys and passwords shows that internal thoughts can be very dangerous if they are not truly separated.

Long term solutions will require more than encryption

Separating thoughts won’t be successful just because companies start using different keys. It will happen when they manage to fully isolate the model’s internal reasoning from the output that is sent to the user. Until then, every attempt at hidden thoughts is just another challenge for a community that loves opening boxes they aren’t supposed to look into.

Lilith's verdict

Putting data in opaque envelopes only works until you realize you have the same letter opener for all of them. But the real mess is that the models were secretly packing other people's access passwords into those envelopes.