In this talk, we discuss construct validity and the common pitfalls benchmarks fall into. This includes a discussion of saturation, the point at which benchmarks are effectively brute forced or memorized by models rather than genuinely solved. We also examine LLM-as-Judge and other grading paradigms, scrutinizing their failure modes to inform best practices for more effective, trustworthy evaluation.
You can find the Teams link here: https://teams.microsoft.com/meet/386304154079844?p=bYjhie3wNtT9VRq37n
Question? Reach out to justine.dickhoff@nora.ai