Where Do Large Language Models Learn From?

What data are models trained on, and what do we know (or not know) about it? In this lesson, we analyze the composition of the data used to train language models. Where do those texts come from? The internet, digitized books, academic articles, forums, code repositories, and specialized databases form part of the large corpora used for their training. We review concrete cases (such as BERT, GPT-1, or GPT-3) to understand how data sources have evolved and what happens when companies stop disclosing this information in more recent versions.

We also explore how models are categorized based on their level of openness and transparency: open-source, open-weights, or closed-source. What does it mean to be able to audit a model? What differences exist in terms of costs, access, and community participation? Through current examples, this lesson invites us to reflect on the relationship between data, transparency, and technological power, and to consider what types of models we want to promote in the educational sphere.

At the end of the lesson, you will be asked to answer a few questions. The answers will be graded, and completing this activity is required to finish and earn credit for the course.

👨‍🏫 Instructor: Guido Ivetta