Name: Xiao Yao (PhD 1.5Y)
Eduction: 2013-2017 Tianjin U
2019-2022 Shanghai JiaoTong U
2023-Now SUTD
Supervisor: Lu wei, Li Xiaoli
Working Experience: Ximalaya, Meituan
Current interest: LLM reasoning、Alignment、MoE
Date: 0703
Xiao Yao、Xu Lu、Li JiaXi、Lu Wei and Li XiaoLi
EMNLP 2023
✅ Less storage
✅ Less gpu memory
✅ Less Computational Overhead (Faster training speed)
PLM
Bert/T5/GPT
Trainable
Frozen
Soft Prompt e c
Input n c
The motivation of this initiation is that soft prompt flexibly adjust its rank.
Observation results are consistent.
In practice, c is 100, e is 512, 768, and 1024 for T5-Small, T5-Base, and T5-Large respectively. We set b as 10.
ec >> eb + bc
Based on the observation above, we directly decompose the soft prompt into two low-rank trainable matrix.
Model: T5-Small, T5-Base, T5-Large
Datasets: SuperGlue
8-shot、16-shot and 32-shot
WiC、CB、RTE、and COPA
Ours are consistently better.
If we increase b to a large number, performance will be unstable.
Ours can work even when the length of soft prompt is extremely short.
UCL
ICLR 2024
Xiao Yao、Xu Lu、Li JiaXi、Lu Wei and Li XiaoLi
EMNLP 2024 under review
Recently, researcher explored the self-improvement approaches of LLM. These work can be classified into two categories:
Can LLM self-improve without parameter update or external feedback?
Similar questions have similar solutions. Though generated solution may have mistakes, it can reason through the similar solutions to help solve the question.
Random: choose n generated samples randomly
Long: choose n longest generated samples
If we feed the generated samples to other models (especially the weak one), we find significant empirical gains.
The more powerful the question generator is, the better the performance.
The more accurate the generated solution is, the better the performance.
The performance is relatively stable.