Migrate data.gov data from Harvard Innovation Lab to HDV

[sbarbosadataverse]

300K datasets rescued from data.gov by Harvard Innovation Lab, to move into a collection on HDV
Requires a Harvard faculty owner

Contact:
Cushman, Jack jcushman@law.harvard.edu

[jggautier]

Related:

• Harvest over 2 million datasets from the US Government (data.gov) #56
• https://github.com/IQSS/dataverse-HDV-Curation/issues/568

[cmbz]

2025-11-05

• This task needs a great deal of discussion, given the potential impact of loading this many datasets into production (and also hosting this many datasets).
• @landreev and @sbarbosadataverse (and others) will set up a time to discuss implications, considerations, and plans.

[jggautier]

Does this mean we shouldn't or can no longer harvest datasets from data.gov, and we should close #56? Just adding as another question maybe for @landreev, @sbarbosadataverse and others.

[landreev]

@jggautier A good rule of thumb is that any issue that's been open since before COVID should be closed quietly.
A less flippant/more serious answer is we don't know without more research. I haven't spoken to the "rescuers" of the data yet. I don't know if they meant it as a one time operation, or if the plan is to build an ongoing replication system to keep grabbing new data they make available. ... But I'd say there is a good chance we'll want to close #56 as obsolete. It was opened specifically with their OAI service in mind; and that looks like it was a short-lived experiment that's been long discontinued. Even the "2 million datasets" part is no longer true, as they re-organized their holdings into fewer combined datasets.

[landreev]

(also, let's not assume that this migration/rescue has already been done. the scale of the undertaking may prove to be a serious challenge)