论文
58
AI 评分(两次独立评分的平均)alphaXiv 发布 OpenResearch Intelligence Index,在 Together AI 上评测 8 个模型
原标题:Open models, put to the test on real research tasks. Thanks to @askalphaxiv for building the benchmark and running them on Together AI!
alphaXiv 发布开源基准 OpenResearch Intelligence Index,用于端到端自动科研评测,并在 Together AI 上运行。该基准包含 20 个 Terminal-Bench-Science 任务、6 个新后训练任务和 PaperFindingBench 子集,覆盖 8 个模型、6 家提供商,token 花费约 $7.4k。Opus 5.5 以 61.8% 得分最高、每任务 $16.46,GPT-6.1 Sol 得分 54.9%、每任务仅 $3.83。
中文译文尚未完成,先为你展示原文。
Open models, put to the test on real research tasks. Thanks to @askalphaxiv for building the benchmark and running them on Together AI!
We created the world’s first open source benchmark for end-to-end autoresearch. Today, its results are live. Which models are best for RSI? Can they understand the literature? Can they conduct complex scientific experiments, while maintaining good taste? We spent about $7.4k in tokens across 8 models, 6 providers, and 3 evaluation tracks to find out. The OpenResearch Intelligence Index evaluates models on scientific knowledge (20 hand-picked Terminal-Bench-Science tasks), post-training ability (6 novel post-training tasks), and literature relevancy/taste (a subset of PaperFindingBench), with each model running in its native harness inside OpenResearch. So far, the index includes Opus 5.5, GPT-6 Astra, GPT-6.1 Sol, Gemini 4 Argon, Fable 5.1, DeepSeek V4.1 Flash, Kimi K3, and Qwen3.8-Flash. Opus 5.5 scores the highest at 61.8%, costing $16.46 per task. GPT-6.1 Sol scores slightly lower (54.9%) at a fraction of the cost (only $3.83 per task). When paired with OpenResearch’s literature search tools, DeepSeek V4.1 Flash performs nearly as well as GPT-6 Astra on the literature relevancy track (47.7% vs 48.7%, respectively). Check out the blog post to learn more: https://www.alphaxiv.org/blog/introducing-openresearch-intelligence-index. More models and source code coming soon.查看原帖
