Skip to content

ENH Add PEFT method papers in a structured way#3323

Open
BenjaminBossan wants to merge 2 commits into
huggingface:mainfrom
BenjaminBossan:enh-add-paper-attributes-to-peft-configs
Open

ENH Add PEFT method papers in a structured way#3323
BenjaminBossan wants to merge 2 commits into
huggingface:mainfrom
BenjaminBossan:enh-add-paper-attributes-to-peft-configs

Conversation

@BenjaminBossan

Copy link
Copy Markdown
Member

Right now, there is no structured way to find the paper corresponding to a specific PEFT method. But having this can have advantages, e.g. to create a list of paper links for PEFT methods (see #3317).

To create this PR, I asked a coding agent to first write a script to extract the papers using heuristics like regexes from the PEFT code and docs. The resulting file was used to ground the AI. Once implemented, I manually checked each paper and title (there were indeed a few errors or information that was out-of-date).

As for the design, I went with a dict of lightweight dataclasses on the PEFT method's config. In general, it will contain a single entry, which is a dataclass with the URL and title of the paper. The key will correspond to the value of the PeftType. If there are multiple papers, the dict has multiple entries, where one is considered the 'main' paper based on the key.

For variations, the dict may also contain multiple entries. E.g. for DoRA, it resides in the LoraConfig's papers attribute under the "DoRA" key (there is no corresponding PeftType so the spelling doesn't need to be all caps). There is also a very brief description for the variants that describes how to use them.

When it comes to URLs, I preferred HF papers > Arxiv > other sources. When the paper title changed over time, I took the most up-to-date title. I omitted papers that are not implemented as a PEFT method or PEFT method variant (so e.g. no QLoRA paper).

Even though the papers attribute lives on the config class, it is IMO better not to include it in the adapter_config.json. Otherwise, we create bigger files with arguably unnecessary info (not needed to load the PEFT adapter) and all existing adapter_config.jsons would be updated, resulting on warnings when loading them with older PEFT versions.

PR #3317 will be updated based on this PR.

Right now, there is no structured way to find the paper corresponding
to a specific PEFT method. But having this can have advantages,
e.g. to create a list of paper links for PEFT methods (see huggingface#3317).

To create this PR, I asked a coding agent to first write a script to
extract the papers using heuristics like regexes from the PEFT code
and docs. The resulting file was used to ground the AI. Once
implemented, I manually checked each paper and title (there were
indeed a few errors or information that was out-of-date).

As for the design, I went with a dict of lightweight dataclasses on
the PEFT method's config. In general, it will contain a single entry,
which is a dataclass with the URL and title of the paper. The key will
correspond to the value of the PeftType. If there are multiple papers,
the dict has multiple entries, where one is considered the 'main'
paper based on the key.

For variations, the dict may also contain multiple entries. E.g. for
DoRA, it resides in the LoraConfig's papers attribute under the DoRA
key. There is also a very brief description for the variants that
describes how to use them.

When it comes to URLs, I prefered HF papers > Arxiv > other
sources. When the paper title changed over time, I took the most
up-to-date title. I omitted papers that are not implemented as a PEFT
method or PEFT method variant (so e.g. no QLoRA paper).

PR huggingface#3317 will be updated based on this PR.
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants