Hy4: Tencent Quietly Catches Up in Open Weight Models

Tencent enters the ring with a 770B text model

Chinese company Tencent has released Hy4 Preview, a new open-weight text model. It represents a massive leap: 770 billion total parameters (49 billion active in a Mixture-of-Experts architecture) and a context window of 1 million tokens. It takes up 1.56 TB on Hugging Face. The previous July version, Hy3, had "only" 295 billion parameters and a 256K context.

An interesting detail noticed by Simon Willison in the chat_template.jinja file points to built-in chain-of-thought capability: the model explicitly expects a reasoning_effort setting of either high (default) or no_think.

Another heavyweight opens up for local developers

Although the model is blind (no vision), its capacity places it at the very top of current open-weight models. For teams building on open models with the necessary hardware, it provides another strong alternative to Llama 3 or models from Mistral and Qwen.

The built-in, natively supported reasoning layer also demonstrates that "thinking" before answering is becoming a standard feature of the largest models, not just the domain of OpenAI and its O1.

Expensive inference and truncated English

Despite the MoE architecture, 1.56 TB of data to download is a huge obstacle. For most local hackers, the model will remain accessible only via API or specialized providers like OpenRouter.

An interesting finding from testing is that the hidden reasoning trace generates thoughts in slightly truncated, ungrammatical English ("Better maybe no."). This is likely the result of optimization for token efficiency. The model does not need perfect grammar in its pre-output reasoning.

How the model holds up in production RAG systems

Whether the model gains traction will depend on its actual ability to utilize the 1-million context without hallucinating. Expect a massive wave of independent Needle In A Haystack benchmarks and stress tests in the coming days, which will show if it's worth provisioning the corresponding hardware for it.

Lilith's verdict

Tencent's massive text model proves that native reasoning is becoming the standard layer for anyone wanting to play in the big leagues.