In this talk, we discuss construct validity and the common pitfalls benchmarks fall into. This includes a discussion of saturation, the point at which benchmarks are effectively brute forced or memorized by models rather than genuinely solved. We also examine LLM-as-Judge and other grading paradigms, scrutinizing their failure modes to inform best practices for more effective, trustworthy evaluation.
A Guide to Benchmarks: Judging the Judge, Building Better Evals
Join postdoctoral researcher Ryan Marinelli (Department of Informatics, UiO, Norwegian Language Technologies research stream, MediaFutures) in this webinar hosted by NORA Research School
Publisert 17. aug. 2026 11:58
- Sist endret 17. aug. 2026 11:58