Open-Weights Models Are Losing Ground to the Closed Frontier

The chasm between the community and the labs is widening again

Wharton professor Ethan Mollick, one of the most diligent practical AI testers, highlights a clear trend: open-weights models are lagging behind the absolute frontier more than we've been accustomed to. Even though new open releases look promising in benchmarks, in practical daily use they simply don't reach the level of models like Claude 3 Opus or Google Gemini Advanced. The race that seemed even last year is now being won by the wallets of the biggest labs.

For local development, this means mandatory compromise

This dynamics has a direct impact on teams building on top of AI who need to keep their data in-house. If the open-weights layer (like Llama 3 or Qwen) doesn't realistically deliver the quality of the best closed models, engineers have to make a compromise: either sacrifice performance and reasoning, or send data out via API. The narrative that the community will soon catch up to OpenAI and Anthropic on its own is taking a serious hit in practice.

Benchmark scores are highly deceptive

When you look at press releases for new open models, the numbers often look comparable to GPT-4. But Mollick's observation echoes what everyone working with AI daily knows: synthetic tests have decoupled from real utility. The ability to hold long context without losing attention, nuance in tone, and reliability in executing complex instructions are exactly the traits where the massive compute power of commercial giants still dominates.

The wait for Meta's next major release will decide

The key event for the open model ecosystem will be the release of a truly massive version of Llama 3 (400B+ parameters) from Meta. This step will show whether the current gap is just a temporary fluctuation caused by the release cycle, or if the open-source curve is flattening and the absolute frontier is permanently pulling away from anyone who isn't investing billions in infrastructure.

Lilith's verdict

The story of a community revolution is nice, but until you can locally run something that holds context like Claude, you're just playing in the minor leagues.