Author Identifier (ORCID)
Abstract
The rapid advancement of machine learning has led to an unprecedented expansion of model ecosystems, making it increasingly difficult to assess the reliability of newly released models on unseen and unlabeled data. Existing evaluation pipelines typically rely on costly annotation, repeated fine-tuning, or assumptions that do not generalize well to new models. We introduce MetaEvaluator, a cost-effective, model-agnostic framework for fast, label-free evaluation of unseen models across diverse architectures and modalities. MetaEvaluator meta-learns over a pool of reference models to acquire an effective initialization for accurate assessment of unseen models, thereby amortizing evaluation cost and eliminating the need for per-model retraining. To the best of our knowledge, this is the first model-agnostic framework that evaluates new models on unlabeled datasets. Extensive experiments demonstrate that MetaEvaluator delivers stable and accurate performance estimates at substantially lower cost than conventional approaches, enabling scalable benchmarking on unlabeled datasets for emerging models. The code is available at: https://github.com/phkhanhtrinh23/MetaEvaluator.
Keywords
meta-learning, model evaluation, unlabeled data, unseen models
Document Type
Conference Proceeding
Date of Publication
8-8-2026
ISSN
2154817X
Volume
2-B
Publication Title
Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Publisher
Association for Computing Machinery
School
School of Science
RAS ID
100714
Funding received from the Australian Research Council (ARC)
DP240101108, DP260100326, LP240200546
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
ISBN
[9798400722592]
First Page
3978
Last Page
3987
Recommended Citation
Pham, T., Huynh, V., Yin, H., Nguyen, Q. V. H., & Nguyen, T. T. (2026, August). Learning to evaluate: Cost-effective model evaluation on unlabeled data with meta-learning. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 3978-3987). Association for Computing Machinery. https://doi.org/10.1145/3770855.3818104