AU-Harness: An Open-Source Toolkit for Efficient and Unified Evaluation of AUdio-LLMs
Abstract
AU-Harness is an efficient evaluation framework for large audio language models that accelerates processing, supports multi-turn dialogue analysis, and enables standardized large-scale assessment.
Large Audio Language Models (LALMs) are rapidly advancing, but evaluating them remains challenging due to inefficient and nonstandardized toolkits that limit fair comparison and systematic assessment. Existing evaluation frameworks exhibit three critical limitations: (1) slow and inefficient processing pipeline that bottlenecks large-scale studies, (2) inadequate multi-turn dialogue support, leaving fundamental questions about cross-turn context integration and performance dynamics over extended conversations in LALMs unanswered; and (3) the absence of unified and scalable evaluation framework capable of keeping pace with the rapid growth of both LALMs and audio benchmarks. To address these issues, we introduce AU-Harness, an efficient and comprehensive evaluation framework for LALMs. Our empirical studies demonstrate that AU-Harness achieves 77% reduction in evaluation processing time over existing evaluation toolkits, enabling large-scale LALM evaluations previously considered impractical. We provide standardized evaluation protocols and flexible configurations for fair model comparison across diverse scenarios. AU-Harness unlocks a range of in-depth analyses difficult to conduct without a unified foundation, including multi-turn dialogue dynamics, enabling the study of true audio reasoning capabilities in existing LALMs. AU-Harness provides both practical evaluation tools and insights into model limitations, advancing systematic LALM development.
Community
Voice is becoming central to AI assistants as the ultimate UI. But evaluation has remained fragmented, narrow, and slow.
AU-Harness brings it all together and is:
โก Blazing fast and inference-efficient
๐ ๏ธ Customizable for accents, languages, long-form audio, and multi-turn dialogue
๐ Broad task coverage: ASR โ paralinguistics โ understanding โ reasoning โ safety
๐ฆ Modular & extensible for easy experimentation
๐ 50+ datasets | 380+ subsets | 21 tasks | 9 metrics
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence (2025)
- DIFFA: Large Language Diffusion Models Can Listen and Understand (2025)
- LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model (2025)
- SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models (2025)
- CodecBench: A Comprehensive Benchmark for Acoustic and Semantic Evaluation (2025)
- Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan (2025)
- AHELM: A Holistic Evaluation of Audio-Language Models (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2509.08031 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper