ERNIE 5.5 Flash Base · ERNIE

Introduction A base model is valuable both for the capabilities that pre-training builds and for the starting point it offers for further training. Different domains, reasoning tasks and agentic applications call for different data, training objectives and behaviors. Releasing the weights at the end of pre-training lets developers build on the general capabilities already in place and choose for themselves where to take the model next.

It was pre-trained from scratch on 33.5T tokens with a training context of up to 128K, and supports a 1M-token context for fine-tuning and inference.

The weights are the checkpoint at the end of pre-training, before any mid- or post-training, and can serve as a common starting point for domain adaptation, instruction tuning and reinforcement-learning research. To be worth building on, a base model has to stay affordable to train and run, so efficiency was a design goal from the outset. Further training samples from the model repeatedly, and agentic applications bring multi-turn interactions and long contexts. Both add to compute and memory costs. ERNIE 5.5 Flash therefore revisits the trade-off between capability and efficiency in every component, from attention and residual connections to the MoE layers and the output head, and weighs activated compute, KV cache and long-context cost together rather than one at a time. The same thinking shaped the data and the training system. Every later stage of training builds on what pre-training provides, so the corpus emphasizes capabilities that are hard to add afterwards, such as repository-scale code and long-context understanding. And because an efficient architecture pays off only if it can also be trained efficiently at scale, we designed the model and the training system together. We hope this release spares more developers the cost of large-scale pre-training, so they can spend their own resources on the capabilities, tasks and training methods they care about.

Key highlights: An open starting point for further training.

Base weights with no mid- or post-training, leaving the community free to pursue its own approaches to domain adaptation, reasoning and agent training.

Built for efficient further training and inference.

Sparse MoE and hybrid sparse attention balance model capacity, activated compute and long-context cost in the architecture itself.

Broad data, efficient training.

Diverse, high-quality pre-training data gives later training a well-rounded foundation, and a training system co-designed with the architecture keeps large-scale pre-training efficient.

A clean research baseline.

A single checkpoint taken at the end of pre-training makes it easy to compare training methods under controlled conditions and to measure what each one adds.

The remaining layers use Heavily Compressed Attention (HCA): each compresses every 128 tokens into a single cached entry, then attends to all of these entries plus a short window of recent, uncompressed tokens.

An HCA layer therefore still sees the whole sequence, while caching only a small fraction of what a full-context layer needs.

Layer-specific head dimensions: Instead of using the same head dimension in every layer, we set it according to how expensive each layer's attention is: cheap layers get wide heads, expensive layers get narrower ones.

A lightweight indexer scores past positions for each query, and core attention is computed only over the top-k highest-scoring positions. The per-query cost of core attention therefore stays constant as the context grows.

This lets heads share information, which improves expressiveness and training quality, and at inference it is fused into the attention kernel with negligible overhead. 2.2 Multi-Router Latent MoE A conventional MoE layer picks experts with a single router and a hard top-k.

We find that a single router's scores correlate only weakly with how useful each expert actually is, especially in the middle and deeper layers.

And because experts left out of the top-k receive no gradient from that token, the router gets no signal that it chose wrongly, so wrong selections tend to reinforce themselves.

Multi-Router replaces the single router with several independently initialized sigmoid routers that score the experts jointly.

The routers develop clearly distinct preferences: an expert can be chosen because one router strongly favors it or because several favor it together, so a wrong choice is easier to correct. The errors of any one router no longer decide the selection, and routing becomes less sensitive to noisy features and distribution shift.

The number of activated experts, expert FLOPs and communication volume stay the same.

Compared with a single router, routing scores correlate noticeably better with how useful each expert is, with the largest gains in the middle and deeper layers.

Latent experts.

Each MoE layer first projects tokens from 4096 to a 2048-d latent space; the experts (intermediate size 2048) run in that space, and a single shared up-projection maps the result back to 4096.

This roughly halves dispatch/combine traffic and expert FLOPs relative to full-width experts.

Stable LatentMoE design with learnable scalars. The MoE blocks follow the Stable LatentMoE design (an RMSNorm before the up-projection and SiTU-GLU expert activations), which keeps expert outputs well scaled across all 43 layers and stabilizes optimization.

In addition, each routed expert's routing weight is multiplied by its own learnable scalar.

2.3 Manifold-Constrained Hyper-Connections (mHC) ERNIE 5.5 Flash replaces the standard residual connection with mHC: each layer reads from and writes to 4 parallel residual streams through a constrained mixing matrix, which gives layers a wider channel for passing information to one another at negligible extra compute. 2.4 MultiMax Output Head A single softmax temperature forces a trade-off: lowering it makes the distribution sparse but also suppresses other plausible tokens; raising it keeps them but fattens the long tail of implausible ones. MultiMax replaces the single temperature with a learnable piecewise temperature, so the output head no longer has to choose: it suppresses the long tail of implausible tokens while keeping several plausible next tokens. The logits are rescaled element-wise inside the chunked, Triton-fused cross-entropy kernel just before the softmax. This adds negligible training overhead and no extra memory, and significantly lowers pre-training loss.

2.5 ECHO: Replacing the MTP Module Conventional multi-token prediction (MTP) attaches extra modules to the backbone to predict future tokens, so the loss on future tokens mostly trains those modules, and the backbone is still trained mainly to predict the next token. ERNIE 5.5 Flash replaces the MTP module with ECHO, which reuses the backbone's own final layer to predict future tokens.

Because there is no separate MTP layer, the loss on future tokens directly trains the same parameters that produce the next token. Learning to look past the next token encourages the backbone to anticipate where the sequence is heading, with clear gains on reasoning and code.