Name: Xiao Yao (PhD 1.5Y)
Eduction: 2013-2017 Tianjin U
2019-2022 Shanghai JiaoTong U
2023-Now SUTD
Supervisor: Lu wei, Li Xiaoli
Working Experience: Ximalaya, Meituan
Current interest: LLM reasoning、Alignment、MoE
Date: 0703
Decomposed Prompt Tuning via Low-Rank Reparameterization
Xiao Yao、Xu Lu、Li JiaXi、Lu Wei and Li XiaoLi
EMNLP 2023
Background
- Update a small amount of parameters
- Achieve comparable (even better) performance than full finetuning
✅ Less storage
✅ Less gpu memory
✅ Less Computational Overhead (Faster training speed)
Parameter Efficient Finetuning
Background
Prompt Tuning
PLM
Bert/T5/GPT
Trainable
Frozen
Soft Prompt e c
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440084/pasted-from-clipboard.png)
Input n c
Observation
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440257/pasted-from-clipboard.png)
The motivation of this initiation is that soft prompt flexibly adjust its rank.
Observation
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440492/pasted-from-clipboard.png)
Observation results are consistent.
Method
In practice, c is 100, e is 512, 768, and 1024 for T5-Small, T5-Base, and T5-Large respectively. We set b as 10.
ec >> eb + bc
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440406/pasted-from-clipboard.png)
Based on the observation above, we directly decompose the soft prompt into two low-rank trainable matrix.
Results
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440432/pasted-from-clipboard.png)
Model: T5-Small, T5-Base, T5-Large
Datasets: SuperGlue
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440451/pasted-from-clipboard.png)
Few-shot Results
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440443/pasted-from-clipboard.png)
8-shot、16-shot and 32-shot
WiC、CB、RTE、and COPA
Ours are consistently better.
Bottleneck
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440454/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440463/pasted-from-clipboard.png)
If we increase b to a large number, performance will be unstable.
Prompt Length
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440471/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440476/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11440483/pasted-from-clipboard.png)
Ours can work even when the length of soft prompt is extremely short.
Same idea
DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
UCL
ICLR 2024
Self-Adaptive In-Context Chains-of-Thought for Enhanced Mathematical Reasoning
Xiao Yao、Xu Lu、Li JiaXi、Lu Wei and Li XiaoLi
EMNLP 2024 under review
Motivation
Recently, researcher explored the self-improvement approaches of LLM. These work can be classified into two categories:
- update the parameters of LLM: RFT、DPO
- ICL without parameter update: use external feedback like compiler error or gold label
Can LLM self-improve without parameter update or external feedback?
Method
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11450203/pasted-from-clipboard.png)
Intuition: Why It Works
Similar questions have similar solutions. Though generated solution may have mistakes, it can reason through the similar solutions to help solve the question.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11450265/pasted-from-clipboard.png)
Results
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11450236/pasted-from-clipboard.png)
Random: choose n generated samples randomly
Long: choose n longest generated samples
Results
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11450281/pasted-from-clipboard.png)
If we feed the generated samples to other models (especially the weak one), we find significant empirical gains.
Effects of Question Generator
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11450303/pasted-from-clipboard.png)
The more powerful the question generator is, the better the performance.
Effects of Generated Solutions' Accuracy
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11450335/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11450346/pasted-from-clipboard.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11450347/pasted-from-clipboard.png)
The more accurate the generated solution is, the better the performance.
Robustness
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2575653/images/11450348/pasted-from-clipboard.png)
The performance is relatively stable.
Thank you!
deck
By Yao
deck
- 36