Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

The project describes a large-scale Indic language pretraining effort using 2.8 trillion tokens. It also introduces an open pipeline for generating training data.

AI Pillar Hub files a short pointer. The reporting lives at the source, Hugging Face Blog.