Search AIMS.
Find AIMS pages, publications, seminars, course projects, and software from one place.
Results for “predictive evaluation”
13 results
The Predictive AI Evaluation Competition at NeurIPS 2026
Predict whether an AI system answered a benchmark question correctly. Submissions are evaluated across fixed test-time label budgets using Brier ALC.
The Predictive AI Evaluation Competition is open
Final codebases and technical reports for the competition are due by 30 October 2026 at 23:59 Anywhere on Earth (AoE).
Investigating Single-Turn Evaluation as a Predictor of Multi-Turn Child Safety
Tracy Y Wei — Child-AI safety benchmarks almost universally rely on single-turn evaluation, but real interactions are conversational. I re-score 225 multi-turn conversations from the ChildSafe corpus (Claude Sonnet 4,…
Signal-Detection IRT for AI Evaluation on Success-vs-Multi-Failure Tasks
Lezhi (Carrie) Tan, Luna Lyu — A recurring class of AI evaluation tasks has an asymmetric label structure: a privileged “success” class and a heterogeneous set of failure subtypes that can co-occur within a single input — multi-label…
Predictive Validity of Objective Audio Metrics for Human Preference in Music Generation
Si Qi Chen — Evaluating music generation models is difficult because the target outcome is often human preference, and there is no single objective metric that fully captures the perceptual and aesthetic quality of…
Latent Capability Estimation Under Sparse AI Evaluation
Jacob Alan Rubenstein, Shane Robinson Mion, Vania Chow — Item Response Theory (IRT) and low-rank matrix completion are both used to estimate latent model capability from (model, item) response matrices, but they are typically benchmarked on dense, curated data rather…
CS 321M: AI Measurement Science at Stanford
Frameworks and methodologies for measuring, benchmarking, and understanding AI systems.
Beyond the Transcript: Reliability and Bias in LLM-Based Depression Judgments
Kerui Lu, Linyin Lyu — LLM-as-judge systems are increasingly used to automate subjective evaluation, but in high-stakes settings it is unclear whether their judgments reflect the intended construct or artifacts of the measurement…
When Model Rankings Diverge: Operational Validity of Fraud Detection Metrics
Deonna Owens — Fraud detection systems are commonly evaluated using aggregate predictive metrics such as AUROC, precision, recall, and F1 score. While these metrics summarize average classification performance, they may not…
AI Measurement Science Workshop
A one-day workshop on the science of measuring frontier AI systems: measurement under interaction, strategic optimization, and non-stationarity.
AIMS Resources
The textbook, the course, the competition, the tools, and the community channels. Pick a starting point.
CS321M Lecture Materials
Browse CS321M lecture slides, discussion notebooks, and videos by class session.
Call for papers: AI Measurement Science Workshop at COLM 2026
AIMS is organizing a one-day workshop on rigorous AI measurement, co-located with COLM 2026 in San Francisco. The research and competition tracks are now open.