Papers
arxiv:2509.08031

AU-Harness: An Open-Source Toolkit for Efficient and Unified Evaluation of AUdio-LLMs

Published on Oct 23
ยท Submitted by
Aman Tiwari
on Sep 12, 2025
Authors:
,
,
,
,
,
,
,

Abstract

AU-Harness is an efficient evaluation framework for large audio language models that accelerates processing, supports multi-turn dialogue analysis, and enables standardized large-scale assessment.

Large Audio Language Models (LALMs) are rapidly advancing, but evaluating them remains challenging due to inefficient and nonstandardized toolkits that limit fair comparison and systematic assessment. Existing evaluation frameworks exhibit three critical limitations: (1) slow and inefficient processing pipeline that bottlenecks large-scale studies, (2) inadequate multi-turn dialogue support, leaving fundamental questions about cross-turn context integration and performance dynamics over extended conversations in LALMs unanswered; and (3) the absence of unified and scalable evaluation framework capable of keeping pace with the rapid growth of both LALMs and audio benchmarks. To address these issues, we introduce AU-Harness, an efficient and comprehensive evaluation framework for LALMs. Our empirical studies demonstrate that AU-Harness achieves 77% reduction in evaluation processing time over existing evaluation toolkits, enabling large-scale LALM evaluations previously considered impractical. We provide standardized evaluation protocols and flexible configurations for fair model comparison across diverse scenarios. AU-Harness unlocks a range of in-depth analyses difficult to conduct without a unified foundation, including multi-turn dialogue dynamics, enabling the study of true audio reasoning capabilities in existing LALMs. AU-Harness provides both practical evaluation tools and insights into model limitations, advancing systematic LALM development.

Community

Paper author Paper submitter

Voice is becoming central to AI assistants as the ultimate UI. But evaluation has remained fragmented, narrow, and slow.

AU-Harness brings it all together and is:
โšก Blazing fast and inference-efficient
๐Ÿ› ๏ธ Customizable for accents, languages, long-form audio, and multi-turn dialogue
๐Ÿ“Š Broad task coverage: ASR โ†’ paralinguistics โ†’ understanding โ†’ reasoning โ†’ safety
๐Ÿ“ฆ Modular & extensible for easy experimentation
๐Ÿ‘‰ 50+ datasets | 380+ subsets | 21 tasks | 9 metrics

@librarian-bot recommend

ยท

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2509.08031
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2509.08031 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2509.08031 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2509.08031 in a Space README.md to link it from this page.

Collections including this paper 1