Sparse High Rank Adapters (SHiRA) Support

[kkb-code]

Feature request

We would like to add code for Sparse High Rank Adapters (SHiRA) which was published at NeurIPS 2024 (PAPER LINK). We have created a pull request at #2584.

Motivation

This is an alternate type of adapter and has been found to have significant advantages over the low rank adapters. Specifically, SHiRA achieves better accuracy than LoRA for a variety of vision and language tasks. It also offers simpler and higher quality multi-adapter fusion by significantly reducing concept loss, a common problem faced by low rank adapters. Please see the paper for more details.

Your contribution

We have implemented the SHiRA technique and created a pull request at #2584.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[BenjaminBossan]

not stale

[BenjaminBossan]

Resolved via #2584, thanks for your contribution.