Section outline

  • LESSON 6 - ARTIFICIAL INTELLIGENCE FOUNDATIONS FOR BROADENING CULTURAL HORIZONS

    Where Do Large Language Models Learn From?

    What data are models trained on, and what do we know (or not know) about it? In this lesson, we analyze the composition of the data used to train language models. Where do those texts come from? The internet, digitized books, academic articles, forums, code repositories, and specialized databases form part of the large corpora used for their training. We review concrete cases (such as BERT, GPT-1, or GPT-3) to understand how data sources have evolved and what happens when companies stop disclosing this information in more recent versions.

    We also explore how models are categorized based on their level of openness and transparency: open-source, open-weights, or closed-source. What does it mean to be able to audit a model? What differences exist in terms of costs, access, and community participation? Through current examples, this lesson invites us to reflect on the relationship between data, transparency, and technological power, and to consider what types of models we want to promote in the educational sphere.

    At the end of the lesson, you will be asked to answer a few questions. The answers will be graded, and completing this activity is required to finish and earn credit for the course.

    👨‍🏫 Instructor: Guido Ivetta

    • Where Do Large Language Models Learn From? URL

      Prefer to watch in a language other than Spanish? Just turn on CC, go to Settings (⚙️) > Subtitles > Auto-translate, and select your language!

    • Below you will find the lesson slides, designed as a reference to revisit the ideas, questions, and tensions raised in the video. (Please note: The slides are in Spanish).

      📚 Key Concepts

      Open Source: Models where everything is made available: the code, the training data, and the model itself. They allow full transparency and offline usage.

      Closed Source: Proprietary models (such as GPT-4 or Gemini) that can only be used through a service or interface. They function as "black boxes" lacking internal transparency.

      Large Language Model (LLM): An artificial intelligence system trained on vast volumes of text to process, generate, and understand human language probabilistically.

      Open Weights: Models that allow downloading and execution (under permissive licenses), but keep their training data confidential.

      Computing Power: The processing capability required to run or train a model. Even if a model is "free," the computational power needed to run it is often costly.

      A creation by Fundación Vía Libre in collaboration with FAMAF – UNC.
    • Assignment: Where Do Large Language Models Learn From? Quiz