Create new harvesting clients to harvest from a few different repositories

[jggautier]

This GitHub issue is to track harvesting of records from repositories that we had to stop harvesting and that we couldn't start harvesting while @landreev addressed issues that prevented Harvard Dataverse from harvesting reliably, like what he described at IQSS/dataverse#11828.

@landreev has restarted weekly harvesting from most of the other repositories that we had to stop harvesting from. See #408.

And he asked me to create this GitHub issue and include links to the email threads where folks have asked us to harvest from additional repositories. The repositories, existing GitHub issues if there are any, and links to the email threads in our RT support email system are:

• DANS Data Stations (#266): https://help.hmdc.harvard.edu/Ticket/Display.html?id=360607
• CIRAD Dataverse: https://help.hmdc.harvard.edu/Ticket/Display.html?id=365436
• SRDA (#243) : https://help.hmdc.harvard.edu/Ticket/Display.html?id=287243

@landreev, I made you owner of those three RT tickets. Let me know if there's anything else I can do to help.

[landreev]

Could you please create collections for these repos?
Except for SRDA of course, that we already have.

[jggautier]

Cool okay, I'll create collections for those repos today

[jggautier]

Here are the collections:

• https://dataverse.harvard.edu/dataverse/dans_archaeology_harvested
• https://dataverse.harvard.edu/dataverse/dans_ssh_harvested
• https://dataverse.harvard.edu/dataverse/dans_phys_techsciences_harvested
• https://dataverse.harvard.edu/dataverse/dans_life_sciences_harvested
• https://dataverse.harvard.edu/dataverse/cirad_harvested

[landreev]

Cool, I will try to harvest these today.

[landreev]

dans_archaeology: I want to put more thought into/ask if harvesting one of the smaller subsets would suffice.
dans_ssh is also on the larger side; I will try and harvest it overnight.
With the remaining 3 - harvested a decent number of datasets. But there are quite a few failures (esp. w/ dans_phys_techsciences), so I'd like to take a closer look before contacting them.

[landreev]

The dans_ssh_harvested collection is now showing a few datasets at the top that were published just yesterday, so that's nice. On the other hand - more than 50% failure rate. So, something isn't right there.
It looks like it'll be an effort and will take some time before we get these new collections to import fully. But, having some of their datasets is better than none.

[jggautier]

Great to hear!

It looks like you're using oai_ddi for this new harvesting. Think the repository might be able to harvest more if you used oai_dc, instead?

[landreev]

The other way around - oai_dc is the most primitive/least metadata-rich of the supported formats. Very few fields, and no files, only datasets.
oai_ddi is in the middle, meaning there is a possibility to get a bit more searchable metadata from our proprietary dataverse_json. In reality, importing dataverse_json is much more likely to fail; more so between different versions of Dataverse.
But, now that it's not obscenely expensive to drop and re-harvest large collections, I am going to try dataverse_json at some point too. On an off chance that the failures are somehow ddi-specific, and dataverse_json will actually fair better.
The exceptions in the logs are unfortunately not specific at all; just "failed validation" and "unknown".

[jggautier]

Ah right, sorry, I meant: Think the repository might be able to harvest more datasets if you used oai_dc, instead? A while back, I think the thinking was that the simpler the metadata format, the more likely we'd be able to harvest more "dataset" metadata, even though we'd get less searchable metadata. We were even thinking of codifying this thinking into the sort of policy that we were talking about in IQSS/dataverse-pm#100.

But yeah, if harvesting dataverse_json instead means being able to harvest more "datasets", that sounds awesome!

[landreev]

My bad, I read that as "more metadata". It is true, we are very likely to harvest more datasets if we try oai_dc. But there will also be zero searchable files (vs. 350,000 files indexed in dans_ssh now, for example). So, that would be a call to make, which is more important.

But yeah, if dataverse_json instead means being able to harvest more "datasets", that sounds awesome!

While it's quite unlikely, that dataverse_json will do better than ddi, I just feel like it may be worth trying - now that redoing a full harvest can be done in somewhat reasonable time.