HSplitLoRA: A heterogeneous split parameter-efficient fine-tuning framework for large language models

Authors/Creators

• Zheng Lin
• Yuxin Zhang
• Zhe Chen
• Zihan Fang
• Xianhao Chen
• Praneeth Vepakomma
• Wei Ni, Edith Cowan UniversityFollow
• Jun Luo
• Yue Gao

Author Identifier (ORCID)

Wei Ni’s ORCID record

Abstract

Recently, large language models (LLMs) have achieved remarkable breakthroughs, revolutionizing the natural language processing domain and beyond. Due to immense parameter sizes, fine-tuning these models with private data for diverse downstream tasks has become mainstream. Though federated learning (FL) offers a promising solution for fine-tuning LLMs without sharing raw data, substantial computing costs hinder its democratization. Moreover, in real-world scenarios, private client devices often possess heterogeneous computing resources, further complicating LLM fine-tuning. To combat these challenges, we propose HSplitLoRA, a heterogeneous parameter-efficient fine-tuning (PEFT) framework built on split learning (SL) and low-rank adaptation (LoRA) fine-tuning, for efficiently fine-tuning LLMs on heterogeneous client devices. HSplitLoRA first identifies important weights based on their contributions to LLM training. It then dynamically configures the decomposition ranks of LoRA adapters for selected weights and determines the model split point according to varying computing budgets of client devices. Finally, a noise-free adapter aggregation mechanism is devised to support heterogeneous adapter aggregation without introducing noise. Extensive experiments demonstrate that HSplitLoRA outperforms state-of-the-art benchmarks in training accuracy and convergence speed.

Keywords

distributed learning, large language model, parameter-efficient fine-tuning, split learning

Document Type

Journal Article

Date of Publication

10-1-2026

E-ISSN

15580660

ISSN

15361233

Volume

25

Issue

10

Publication Title

IEEE Transactions on Mobile Computing

Publisher

IEEE

School

School of Engineering

Funding Information

Natural Science Foundation of Shanghai (Grant Number: 25ZR1401016), Research Grants Council of Hong Kong (Grant Number: 27213824 and CRS HKU702/24).

Copyright

subscription content

First Page

15543

Last Page

15559

Recommended Citation

Lin, Z., Zhang, Y., Chen, Z., Fang, Z., Chen, X., Vepakomma, P., Ni, W., Luo, J., & Gao, Y. (2026). HSplitLoRA: A heterogeneous split parameter-efficient fine-tuning framework for large language models. IEEE Transactions on Mobile Computing, 25(10), 15543–15559. https://doi.org/10.1109/TMC.2026.3680521