论文40
LlamaIndex 发布 ParseBench-2 文档解析基准
原标题:ParseBench-2 hit 30k monthly downloads in 3 days, and is #3 on @huggingface trending benchmarks! 📈
LlamaIndex 推出 ParseBench-2 文档解析基准,3 天内月下载量达 30k,登上 Hugging Face trending benchmarks 第 3 名。
中文译文尚未完成,先为你展示原文。
ParseBench-2 hit 30k monthly downloads in 3 days, and is #3 on @huggingface trending benchmarks! 📈
In ParseBench-2, we tried hard to create a comprehensive taxonomy of metrics to evaluate document processing, while creating a severity-based framework to align these metrics with human judgment.
Whether you’re deeply familiar with document OCR and document OCR benchmarks, or just getting started, we encourage you to check out the ParseBench-2 methodology. We’re happy to help you pick whatever subset of metrics you care about if you’re looking to process documents in production.
We’ve merged dozens of PRs since the release date, including fixes/metric updates from some of our peers. We’re aggressively fixing any lingering annotation and metric issues.
Report: https://www.llamaindex.ai/files/parsebench-v2-severity-aware-evaluation-of-document-parsing.pdf
Hugging Face: https://huggingface.co/datasets/llamaindex/ParseBench-2
Github: https://github.com/run-llama/ParseBench/
Today we’re introducing ParseBench-2 - our most ambitious document parsing benchmark to date. It is the only benchmark that provides both extreme granularity into every single type of production parsing error, while also making sure that metric scores are highly aligned with human judgment. We do this by painstakingly assigning severity scores to 330+ error types, and tuning each metric to match the severity scores. Existing benchmarks either use brittle deterministic metrics (e.g. OmniDocBench) or LLM-as-a-judge (ParseArena), which can help recognize equivalent representations but are coarse/stochastic/subject to bias by the prompt+model. ParseBench-2 also contains: ✅ 4.3k+ document pages (double from ParseBench-1). ✅ An extra dimension (forms) ✅ Much better ground-truth across all dimensions compared to ParseBench-1. This is the next evolution of ParseBench, our hugely popular benchmark that has 50k+ monthly downloads, over 150+ OCR solutions benchmarked, and has been adopted by companies like NVIDIA, Cohere, Aleph Alpha, Perplexity, and others. Alongside ParseBench-2, we are also releasing OCRia-1.5 - a new family of document parsing models that powers all our modes in LlamaParse. They represent a massive step up in capabilities from our latest release, and are state of the art in parsing complex tables, forms, charts, formatting, and assigning bounding boxes to complex layouts. Come check out our blog post, technical report, and website. Blog: https://www.llamaindex.ai/blog/introducing-parsebench-v2?utm_source=socials_jerry_liu&utm_medium=li_social&utm_campaign=launch&utm_content=blog_post Technical Report: https://www.llamaindex.ai/files/parsebench-v2-severity-aware-evaluation-of-document-parsing.pdf?utm_source=socials_jerry_liu&utm_medium=tw_social&utm_campaign=launch&utm_content=technical_report Website: https://www.parsebench.ai/ If you’re interested in trying out our OCRia-1.5 capabilities, come sign up for LlamaParse: https://cloud.llamaindex.ai/查看原帖
