TY - GEN
T1 - EFFICIENT FINE-TUNING OF DEEP NEURAL NETWORKS WITH EFFECTIVE PARAMETER ALLOCATION
AU - Wallis, Phillip
AU - Song, Xubo
N1 - Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - It's commonplace in modern deep learning to achieve SOTA performance by fine-tuning a large, pretrained base model. Recent successes in natural language processing, attributed in part to knowledge transfer from large, pretrained, transformer-based language models, have sparked a similar revolution in computer vision via the introduction of Vision Transformers. As modern deep neural networks increase in performance, they also tend to increase in size. Key issues associated with fine-tuning such enormous models include storage overhead, as well as memory and/or latency requirements. Parameter efficient fine-tuning is a fairly recent paradigm which has been evolving alongside massive neural networks in part to address these issues. We showcase the effectiveness of parameter efficient fine-tuning of vision transformers, and introduce a simple yet effective method for learning a non-uniform parameter allocation given a fixed budget. We demonstrate our approach across a range of benchmark tasks in image classification and semantic segmentation.
AB - It's commonplace in modern deep learning to achieve SOTA performance by fine-tuning a large, pretrained base model. Recent successes in natural language processing, attributed in part to knowledge transfer from large, pretrained, transformer-based language models, have sparked a similar revolution in computer vision via the introduction of Vision Transformers. As modern deep neural networks increase in performance, they also tend to increase in size. Key issues associated with fine-tuning such enormous models include storage overhead, as well as memory and/or latency requirements. Parameter efficient fine-tuning is a fairly recent paradigm which has been evolving alongside massive neural networks in part to address these issues. We showcase the effectiveness of parameter efficient fine-tuning of vision transformers, and introduce a simple yet effective method for learning a non-uniform parameter allocation given a fixed budget. We demonstrate our approach across a range of benchmark tasks in image classification and semantic segmentation.
UR - http://www.scopus.com/inward/record.url?scp=85146707173&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85146707173&partnerID=8YFLogxK
U2 - 10.1109/ICIP46576.2022.9897314
DO - 10.1109/ICIP46576.2022.9897314
M3 - Conference contribution
AN - SCOPUS:85146707173
T3 - Proceedings - International Conference on Image Processing, ICIP
SP - 3510
EP - 3514
BT - 2022 IEEE International Conference on Image Processing, ICIP 2022 - Proceedings
PB - IEEE Computer Society
T2 - 29th IEEE International Conference on Image Processing, ICIP 2022
Y2 - 16 October 2022 through 19 October 2022
ER -