论文50

YC 初创推出 FrontierML:5 个前沿模型无一达标

原标题:Synthetic Sciences, a Y Combinator startup, has launched FrontierML, a benchmark so hard that none of the 5 frontier AI models it tested ...

Rohan Paul@rohanpaul_ai发布时间 (北京时间)

Y Combinator 初创公司 Synthetic Sciences 发布 FrontierML 基准,用 25 个开放式 ML 研究问题测试 AI 智能体,5 个前沿模型在所有问题上均未达到强结果线。每个智能体获得 12 小时和真实 H100 GPU,需自行设计进度代理指标并校准其可信度,目标函数全程不可见,最终提交的模型或方法在留出评测上打分。

中文译文尚未完成,先为你展示原文。

Synthetic Sciences, a Y Combinator startup, has launched FrontierML, a benchmark so hard that none of the 5 frontier AI models it tested hit the strong-result bar on any of its 25 ML research problems.

They hide each objective, so agents must build their own tests and decide how much to trust them.

This is a test of whether AI agents can do machine learning research on their own without ever seeing the score they’ll be graded on.

Each agent gets 12 hours and real GPUs, and has to work out for itself how to measure progress.

FrontierML runs each task as a loop with the objective kept out of reach:

The agent gets an open-ended ML problem and 12 hours in Lab0
It designs a local proxy for progress
It runs experiments on H100s and calibrates how far that proxy can be trusted
It submits a model or method
FrontierML scores it on a held-out evaluation

Synthetic Sciences argues step 3 is where much of empirical research happens. Here, a lot of the score rides on it.

Introducing FrontierML, a benchmark for evaluating AI agents on autonomous empirical ML R&D. Most existing benchmarks for ML "research" give the agent a validation set or verifier that closely tracks its final score. In practice, human researchers rarely have that. Much of empirical research is deciding how to measure progress on a problem and how far to trust those measurements. In FrontierML's 25 open-ended problems, agents have to operationalize progress through their own experiments, constructing local proxies and calibrating how far to trust them, while working toward an objective they never directly observe. Their submitted model or method is then scored on a held-out evaluation.
查看原帖
Agent 智能体数据与训练论文研究评测基准#论文/研究#评测/基准#Agent#数据/训练