ENH Add PEFT method papers in a structured way#3323
Open
BenjaminBossan wants to merge 2 commits into
Open
Conversation
Right now, there is no structured way to find the paper corresponding to a specific PEFT method. But having this can have advantages, e.g. to create a list of paper links for PEFT methods (see huggingface#3317). To create this PR, I asked a coding agent to first write a script to extract the papers using heuristics like regexes from the PEFT code and docs. The resulting file was used to ground the AI. Once implemented, I manually checked each paper and title (there were indeed a few errors or information that was out-of-date). As for the design, I went with a dict of lightweight dataclasses on the PEFT method's config. In general, it will contain a single entry, which is a dataclass with the URL and title of the paper. The key will correspond to the value of the PeftType. If there are multiple papers, the dict has multiple entries, where one is considered the 'main' paper based on the key. For variations, the dict may also contain multiple entries. E.g. for DoRA, it resides in the LoraConfig's papers attribute under the DoRA key. There is also a very brief description for the variants that describes how to use them. When it comes to URLs, I prefered HF papers > Arxiv > other sources. When the paper title changed over time, I took the most up-to-date title. I omitted papers that are not implemented as a PEFT method or PEFT method variant (so e.g. no QLoRA paper). PR huggingface#3317 will be updated based on this PR.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Right now, there is no structured way to find the paper corresponding to a specific PEFT method. But having this can have advantages, e.g. to create a list of paper links for PEFT methods (see #3317).
To create this PR, I asked a coding agent to first write a script to extract the papers using heuristics like regexes from the PEFT code and docs. The resulting file was used to ground the AI. Once implemented, I manually checked each paper and title (there were indeed a few errors or information that was out-of-date).
As for the design, I went with a dict of lightweight dataclasses on the PEFT method's config. In general, it will contain a single entry, which is a dataclass with the URL and title of the paper. The key will correspond to the value of the
PeftType. If there are multiple papers, the dict has multiple entries, where one is considered the 'main' paper based on the key.For variations, the dict may also contain multiple entries. E.g. for DoRA, it resides in the
LoraConfig's papers attribute under the"DoRA"key (there is no correspondingPeftTypeso the spelling doesn't need to be all caps). There is also a very brief description for the variants that describes how to use them.When it comes to URLs, I preferred HF papers > Arxiv > other sources. When the paper title changed over time, I took the most up-to-date title. I omitted papers that are not implemented as a PEFT method or PEFT method variant (so e.g. no QLoRA paper).
Even though the
papersattribute lives on the config class, it is IMO better not to include it in theadapter_config.json. Otherwise, we create bigger files with arguably unnecessary info (not needed to load the PEFT adapter) and all existingadapter_config.jsons would be updated, resulting on warnings when loading them with older PEFT versions.PR #3317 will be updated based on this PR.