Publication

Enhancing Parameter-efficient Fine-tuning with Simple Calibration Based on Stable Rank

返回学术发表
Enhancing Parameter-efficient Fine-tuning with Simple Calibration Based on Stable Rank 论文配图

Peiyu Liu, Ze-Feng Gao, Xiao Zhang, Wayne Xin Zhao#, Ji-Rong Wen

Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024

摘要

In this paper, we proposed both theoretical analyses and experimental verification for the proposed calibration strategy. Considering efficiency, we further propose time-aware and structure-aware strategies to determine the most crucial time to commence the fine-tuning procedure and selectively apply parameter matrices for lightweight fine-tuning, respectively.