Feature Request: Email notification about draft datasets

[emmaw786]

Overview of the Feature Request
This feature would enable an automatic reminder email to be sent to a deponent for any datasets left in draft status after a certain period of time (i.e. to remind them to either publish the draft, or submit for review, depending on the configuration of the particular installation). It would be ideal if the managers of a particular dataverse (e.g. institutional collection) could choose whether or not to turn this on, and at what time interval the reminders should be sent (e.g. 6 months, 12 months etc.).

The text in the email could reflect the already existing yellow banner that is displayed above draft datasets and hence say something like "Your dataset XX is still a draft version and is not yet published. Please revisit your dataset, and when you are ready to share, it either publish it or submit it for review. If you do not intend to publish this draft dataset, please delete it".

What kind of user is the feature intended for?
This would be a feature aimed at a depositor, but configurable by the manager of a particular collection.

What inspired the request?
We (at DataverseNO) have experienced that datasets are often left in draft status for a long time (presumably because the deponent started making a draft and then forgot about it, or they did not realise that they needed to press "submit for review").
There is a yellow banner above a draft dataset reminding the deponent of this, but they have no reminder when they come out of the system. Thus, a reminder email could be an effective way to get the deponent to log on to Dataverse again and deal with any lingering draft datasets. This suggestion actually came from a discussion of the issue on the Dataverse Users Community Google Group (from KU Leuven, as they have experienced similar issues).
See thread here: https://groups.google.com/g/dataverse-community/c/7jaVkOLpqFk

What existing behavior do you want changed?
We hope that this will remind deponents to take more responsibility for any draft datasets that they may have forgotten about, and hence make it less time consuming for collection managers to follow up about such datasets.

Any open or closed issues related to this feature request?
#7675 - This closed issue however, relates to in system reminders, whereas here we are talking about email (i.e. external) reminders.

[pdurbin]

Good idea. It should be straightforward to determine if the depositor has permission to publish or not, so in the message we should be able to specify the next step, be it publishing or submitting for review.

@emmaw786 is it safe to assume that in addition to an email notification, an in-app notification should be sent as well?

[bryanmgee]

Dryad has done this for a while with a system of different automated emails that are conditioned on the status/history of the deposit (e.g., different emails for a draft dataset that needed to be submitted for curation vs. an already curated dataset that had been returned and needed to be revised). Especially for institutions that require datasets to be submitted for review, there may be a need/benefit to define multiple conditions with different associated emails (e.g., a dataset returned to the depositor from review that hasn't been actioned in 30 days).

At UT Austin, we've had some discussions about developing something like this for the Texas Data Repository, but the fuzzy draft implementation would be more bespoke given our multi-institutional configuration (different institutions might want to apply different thresholds). I also have some reservations around a fully automated system if it isn't possible to easily disable/reset recurring notifications for a specific dataset (ideally as an admin through the UI). Some of the reminders at Dryad would recur after an unpublished dataset hit the time threshold (e.g., 6 months post-creation), but an author could have a good reason why they are just sitting on a draft (e.g., stuck in peer review), potentially for years, and would get irritated about still getting these reminders, leading them to put all of our emails in the spam filter or start ignoring anything from the email domain.

[pdurbin]

@bryanmgee thanks for all your thoughts above. One "out" people have who get annoyed by too many emails is the ability to mute them by type:

• New feature for muting email and/or in-app notifications #8530

[philippconzett]

@bryanmgee thanks for sharing your thoughts and for pointing to the Dryad way of handling notifications.
@pdurbin is there a similar folder in the Dataverse GitHub with all notifications?

[pdurbin]

@philippconzett https://guides.dataverse.org/en/6.7.1/admin/user-administration.html#mute-notifications is a good place to get a sense of the various notification types. The proposed notification about draft datasets would be a new type.

As for the messages themselves, I think they all begin with "notification.": https://github.com/IQSS/dataverse/blob/v6.7.1/src/main/java/propertyFiles/Bundle.properties#L237

[philippconzett]

@pdurbin Thanks for pointing to these resources! So, the content of the notifications are hard-coded?

[pdurbin]

@philippconzett correct. Here's an issue to make the welcome email configurable, at least:

• Feature Request: ability to customize welcome e-mail, or disable it entirely #11688

[emmaw786]

Thanks everyone for your input!
@pdurbin, yes, I think it would make sense to have an in-app notification as well. As per my suggestion above (but now with some minor tweaks), the text could read something like:

"Notification: Your dataset XX is still a draft and is not yet published. Please revisit your draft dataset, and when you are ready to share it, either publish or submit it for review. If you do not intend to publish this draft, please delete it".

Ideally this could be configured to a particular time interval, depending on the wishes of the particular installation, e.g. reminders sent 1 year after creation, 18 months after creation etc.

@bryanmgee, I understand the point about datasets being stuck in peer-review and that this could create some annoyance, but hopefully those cases will be in the minority? Such a reminder will be helpful to those who simply forgot that they created a draft dataset in the first place, which seems to be a major reason for lingering draft datasets in our case at least.

[bryanmgee]

@emmaw786 I don't know one way or another what the typical distribution of dataset 'lifespans' is, and I imagine this will vary quite a lot between repositories - could be an interesting exercise to find out since it should be pretty easy to get this information, and that might be a useful way for each repository to make a more data-informed decision about what threshold to start sending reminders at (if it's only possible to configure one type of email to be auto-sent). If it seems like the majority of datasets in a given repository could be published at that time but are being left in draft status, there might be a need for a different, much earlier notification or some tweak to the existing 'dataset creation' email to more clearly indicate "hey you made this but it is NOT YET PUBLIC" (or something to that effect).

[emmaw786]

Hi everyone, sorry for the late follow-up here. We have had an internal discussion about this within the management group of DataverseNO and this is something that we would like work on, and hence sponsor, i.e. an email (and in-app) notification that notifies a deponent to the existence of a draft dataset, and which can be configured, by a repository manager, to be set to trigger at a particular time interval. Are there any other comments/input before we move forward with this feature?

[pdurbin]

@emmaw786 that's great news! As you may know, we are rewriting the frontend in React as an SPA (Single Page App). Here's how notifications will look there (assuming this pull request is merged without changes):

• 775 account page notifications dataverse-frontend#825

I think the backend logic shouldn't be overly affected by the the frontend switch from JSF to React but heads up that it uses a new inAppNotificationFormat introduced in this PR:

• Notifications API extension and new notifications generated by API operations #11696

Questions I have:

• When you say repository manager, do you mean superuser? Or do you mean the admin of this or that collection?
• Are you ok with creating API endpoints to allow the feature to be configured? This is ideal because the SPA will use them.
• Do you want a UI in JSF for configuration? I'm ok with configuration being API-only since JSF will be retired in favor of React some day.
• When configuring the time interval, are you familiar with @Schedule? That's what we used in Add timer for saved searches #7152 but from what I understand EJB is being deprecated so maybe there's a CDI equivalent we can use. Not sure.

I hope this helps! You're welcome to join a tech hours if you like! We have them on Tuesdays.

[philippconzett]

Thanks @pdurbin! @emmaw786 and I had a chat to discuss your questions. Here's what we found out:

When you say repository manager, do you mean superuser? Or do you mean the admin of this or that collection?

We mean a superuser / admin of the whole installation. However, we'd like the feature to be configurable at collection level, so that it could be turned on or off for individual collections, and that the timing could be configured differently for each collection where the feature is turned on.

Are you ok with creating API endpoints to allow the feature to be configured? This is ideal because the SPA will use them.

Yes, that's ok.

Do you want a UI in JSF for configuration? I'm ok with configuration being API-only since JSF will be retired in favor of React some day.

No need for UI support at the current stage.

When configuring the time interval, are you familiar with @Schedule? That's what we used in Add timer for saved searches #7152 but from what I understand EJB is being deprecated so maybe there's a CDI equivalent we can use. Not sure.

We're not developers, so we're not familiar with the technicalities here, but would like to suggest choosing a solution that makes it well into the future, also after switching to the new SPA.

Some other things we'd like to be addressed in the feature:

• The notification timing should be based on the dataset last edited time, which might differ from created time.
• The notification should include contact information, similar to other messages that are sent to depositors. It should contain contact information about the collection and the whole repository. Here is an example of this type of information from an existing automatic message:

You may contact the collection administrator for more information: abc@test.mail

You may contact us for support at support@dataverse.no.

Thank you,
DataverseNO Support

Our plan is to sponsor the development of this feature through GDCC.

Please let us know if there anything else we need to clarify. Thanks!