open euro llm logo
Results/Prelude

OpenEuroLLM Prelude

Prelude: a 9B model trained in the open

Prelude is our first experimental 9B dense base model.
Meet the details as we finish it.

Explore checkpointsTraining progress
9.095Bparameters
~10Ttraining tokens
36target languages
1,024GPUs on Leonardo

MULTILINGUAL BY DESING

01

A 10T-token datamix with a European soul

The mix contains mostly language data across 36 languages, covering all official and co-official EU languages. It also contains important code and math datasets.

~80%English
~20%Other languages

TRAINED ON STRATEGIC EU COMPUTE

02

Annealing on Leonardo (CINECA)

Large-scale training on European supercomputing infrastructure.

Nodes
256
Tokens / iteration
~8.39M
Sequence length
4,096
Checkpoint cadence
2,400 iters

FOLLOW THE OPEN PROCESS

Prelude beyond the checkpoints

The data, training config, training progress and intermediate evaluations are public too.

01DATATraining dataComposition, sources and per-language token breakdown.02CONFIGTraining configurationThe production Baby 9B dense experiment configuration.03TRACKINGTraining monitoringLoss curves, throughput and training metrics.04EVALEvaluationOngoing downstream evaluation of intermediate checkpoints.

After Prelude

A 32B dense model is in its opening bars.

Stay tuned!