Probing LLMs to play games

Contents

• Introduction Introduction

  • Evaluating LLMs: Introduction

    1m 53s
• 1. Foundations of LLM Evaluation 1. Foundations of LLM Evaluation

  • (Locked)
    Topics

    34s
  • (Locked)
    Introduction to evaluation: Why it matters

    13m 24s
  • (Locked)
    Generative versus understanding tasks

    11m 3s
  • Key metrics for common tasks

    19m 15s
• 2. Evaluating Generative Tasks 2. Evaluating Generative Tasks

  • (Locked)
    Topics

    35s
  • (Locked)
    Evaluating multiple-choice tasks

    13m 16s
  • (Locked)
    Evaluating free text response tasks, part 1

    23m 43s
  • (Locked)
    Evaluating free text response tasks, part 2

    26m 54s
  • (Locked)
    AIs supervising AIs: LLM as a judge

    12m 29s
• 3. Evaluating Understanding Tasks 3. Evaluating Understanding Tasks

  • (Locked)
    Topics

    47s
  • (Locked)
    Evaluating embedding tasks

    15m 55s
  • (Locked)
    Evaluating classification tasks

    21m 46s
  • (Locked)
    Building an LLM classifier with BERT and GPT

    24m 2s
• 4. Using Benchmarks Effectively 4. Using Benchmarks Effectively

  • (Locked)
    Topics

    45s
  • (Locked)
    The role of benchmarks

    7m 45s
  • (Locked)
    Interrogating common benchmarks

    20m 36s
  • (Locked)
    Evaluating LLMs with benchmarks

    23m 30s
• 5. Probing LLMs for a World Model 5. Probing LLMs for a World Model

  • (Locked)
    Topics

    37s
  • (Locked)
    Probing LLMs for knowledge

    18m 47s
  • (Locked)
    Probing LLMs to play games

    27m 20s
• 6. Evaluating LLM Fine-Tuning 6. Evaluating LLM Fine-Tuning

  • (Locked)
    Topics

    36s
  • (Locked)
    Fine-tuning objectives

    9m 57s
  • Metrics for fine-tuning success

    11m 42s
  • (Locked)
    Practical demonstration: Evaluating fine-tuning

    25m 54s
  • (Locked)
    Evaluating and cleaning data

    32m 30s
• 7. Case Studies 7. Case Studies

  • (Locked)
    Topics

    43s
  • (Locked)
    Evaluating AI agents: Task automation and tool integration

    18m 41s
  • (Locked)
    Measuring retrieval-augmented generation (RAG) systems

    11m 58s
  • (Locked)
    Building and evaluating a recommendation engine using LLMs

    19m 13s
  • (Locked)
    Using evaluation to combat AI drift

    22m 1s
  • (Locked)
    Time-series regression

    19m 35s
• 8. Summary of Evaluation and Looking Ahead 8. Summary of Evaluation and Looking Ahead

  • (Locked)
    Topics

    40s
  • (Locked)
    When and how to evaluate

    10m 37s
  • (Locked)
    Looking ahead: Trends in LLM evaluation

    6m 35s
• Conclusion Conclusion

  • (Locked)
    Evaluating LLMs: Summary

    54s