产品38

LlamaIndex 推出 ParseBench-2 指标实验室

原标题:We launched ParseBench-2 last week as a comprehensive benchmark to evaluate all dimensions of document parsing: reading order, formatting...

Jerry Liu@jerryjliu0发布时间 (北京时间)

LlamaIndex 上周发布 ParseBench-2 文档解析基准,覆盖阅读顺序、格式、表格、图表、表单和视觉定位等维度。此次推出 Metric Lab 界面,用户可自定义各评测指标的权重与严重程度惩罚(默认严重错误扣 10 分),并导出策略 JSON 用于自有文档评测。引用推文补充:调整权重后,19 个解析器会在浏览器中实时重新排名。

中文译文尚未完成,先为你展示原文。

We launched ParseBench-2 last week as a comprehensive benchmark to evaluate all dimensions of document parsing: reading order, formatting, tables, charts, forms, and visual grounding.

While we strive to create some standardization, it's very clear that not everyone weights the same errors equally:
- If you're dealing with backoffice paperwork, you probably don't care about charts, and mostly care about tables and forms
- If you're not dealing with legal documents, you likely don't care about precise formatting.

Also, you may care about different severities differently. By default we penalize "severe" errors by 10 points out of 100, but for instance if getting dollar values wrong is catastrophic, you can turn the slider way up.

Introducing the ParseBench-2 Metric Lab: it's a simple interface where you can define how much you care about different evaluation metrics, and how you weigh different severities. Once you've settled on a dimension, you can export the policy JSON, so that you can use it on evaluations over your own documents!

Come check the Metric Lab out: https://www.parsebench.ai/lab/index.html

For more information on the benchmark itself and overall methodology, come check it out here: https://www.llamaindex.ai/blog/introducing-parsebench-v2

The best parser depends on your documents. A team parsing insurance claim forms and a team parsing earnings decks need different things. One relies on forms and tables, while the other relies on charts. Metric Lab in ParseBench lets you rank parsers by what matters to your team: → Weight the categories you care about: tables, charts, forms, text → Set how much a critical, major or minor error costs → Watch all 19 parsers re-rank live in your browser Start from a default profile or build your own.
查看原帖
产品更新评测基准#产品更新#评测/基准#RAG

同一件事的其他报道1 篇