[ENG-8064] Add New Notifications Data Model (Refactor Notifications Phase 2) #11151

Johnetordoff · 2025-05-20T14:09:45Z

Purpose

This system is designed to formalize and consolidate OSF Notifications under one system in order to better facilitate collaboration between Product, Engineers and QA and end persistent problems with notification email related tech debt. This project is the result of an extensive audit of all OSF emails and combined email digests and seeks to move the NotificationSubscription model out of it's transitional state having been migrated from a document based model, into it's final data model which is more regular for a relation our current relational database (django's postgress db).

In order to do this I have taken @mfraezz design documents and implemented them with minor changes. This means the responsibility for sending notifications in osf.io is shared by three new models, which will have the old data migrated into them via a migration script and management command. The models are:

NotificationType
- Similar to RegistrationSchemas or Waffle flags these are in db instrances which are poulated from static config documents in this case yaml.
- Since these are synced on migration with notification.yaml a developer will be able to see at a glance if where a notification template is and what it's purpose is.
NotificationSubscription
- This model is somewhat analogous to the earlier EmailDigest model, this holds references to potentially many Notifcations models that can be compiled into a single digest this is sent at a periodic basis.
Notification
- Holds message information and context
- Unlike earlier implementations this will record each Notification creation and sending for metrics purposes.

Changes

Combines worker with beat in docker-compose
creates notifications.yaml to contain all notification types info it is populated via postmigrate hook
creates new Notification NotificationSubscription and NotificationType
adds migrations to rename legacy tables and a managment command to populate the new ones.
Deletes old send_mails method and replaces it with emit of NotificationType
adds tests and updates old mocking
updates views and permission classed
A slight change to handle_boa_error to pass the already decanted user object.
adds new data model for Notification, NotificationTypes and Subscriptions
creates a notifications.yaml for data dependency notificationtypes
add migrations
updates notifications to use NotificationTypes
updates Admin app to control email templates
updates metrics reporter to count notifications sent etc.
updates tests to all use capture_notifications mocking util
Removes queued_mail
Removes EmailDigest
Removes detect_duplicates for duplicate subscriptions

QA Notes

Please make verification statements inspired by your code and what your code touches.

Verify
Verify

What are the areas of risk?

Any concerns/considerations/questions that development raised?

Documentation

Side Effects

Ticket

https://openscience.atlassian.net/browse/ENG-8064

Allow the User Preprints endpoint (/v2/users/me/preprints) to be filterable by title (?filter[title]=example), similar to how the User Nodes endpoint (/v2/users/me/nodes) supports this filter capability

…terForOpenScience#11210) Enable Product Team to Force Archive Registrations in the Admin App

Johnetordoff · 2025-08-31T11:17:46Z

addons/base/views.py

@@ -830,7 +860,7 @@ def addon_deleted_file(auth, target, error_type='BLAME_PROVIDER', **kwargs):
            'version_id': None,
            'file_guid': file_guid,
            'file_id': file_node._id,
-            'provider': file_node.provider,
+            'preprint_word': getattr(file_node.target.provider, 'preprint_word', None),


@Johnetordoff check

Johnetordoff · 2025-08-31T11:18:28Z

addons/boa/tests/test_tasks.py

@@ -190,7 +183,19 @@ async def test_submit_success(self):
            assert self.mock_job.refresh.call_count == 4
            assert mock_async_sleep.call_count == 4
            mock_close.assert_called()
-            self.mock_send_grid.assert_called()
+            mock_send_mail.assert_called_with(


Added these back in based on previous CR

Johnetordoff · 2025-08-31T11:29:11Z

addons/base/views.py

    return {'status': 'success'}


 @file_signals.file_updated.connect
-def addon_delete_file_node(self, target, user, event_type, payload):


Notice event_type is gone as that was always derived from the payload anyway, cleans up the logic a lot.

Johnetordoff · 2025-08-31T11:30:40Z

admin/common_auth/views.py

@@ -69,16 +69,6 @@ def form_valid(self, form):

        # create AdminProfile for this new user
        profile, created = AdminProfile.objects.get_or_create(user=osf_user)
-
-        for group in form.cleaned_data.get('group_perms'):


@Johnetordoff Should get done on demand now

Johnetordoff · 2025-08-31T11:42:13Z

api/subscriptions/fields.py

+
+class FrequencyField(ser.ChoiceField):
+    def __init__(self, **kwargs):
+        super().__init__(choices=['none', 'instantly', 'daily', 'weekly', 'monthly'], **kwargs)


@Johnetordoff keep for now

…/centerforopenscience/osf.io into add-new-notifications-data-model * 'add-new-notifications-data-model' of https://github.com/centerforopenscience/osf.io: revert some syntax changes to shrink diff revert ResourceContributorsCreateSerializer refactor update tests

…/centerforopenscience/osf.io into add-new-notifications-data-model * 'add-new-notifications-data-model' of https://github.com/centerforopenscience/osf.io: update moderation notification tests

…/centerforopenscience/osf.io into add-new-notifications-data-model * 'add-new-notifications-data-model' of https://github.com/centerforopenscience/osf.io: update subjects and mailhog tests

…/centerforopenscience/osf.io into add-new-notifications-data-model * 'add-new-notifications-data-model' of https://github.com/centerforopenscience/osf.io: fix more test stuff # Conflicts: # osf/models/schema_response.py

Johnetordoff force-pushed the add-new-notifications-data-model branch 4 times, most recently from 3a8b414 to 57b0bd1 Compare May 21, 2025 14:42

Johnetordoff force-pushed the add-new-notifications-data-model branch from 2b8ba0d to c449599 Compare June 9, 2025 13:40

Johnetordoff changed the base branch from feature/pbs-25-10 to refactor-notifications June 13, 2025 14:40

Johnetordoff force-pushed the add-new-notifications-data-model branch 3 times, most recently from ad18e9d to 300524c Compare July 3, 2025 14:20

Johnetordoff marked this pull request as ready for review July 8, 2025 20:05

Johnetordoff force-pushed the refactor-notifications branch from 2dee1b2 to 2dbcbf7 Compare July 10, 2025 15:50

Johnetordoff force-pushed the add-new-notifications-data-model branch 4 times, most recently from 9238258 to 37b419a Compare July 10, 2025 20:24

Johnetordoff force-pushed the refactor-notifications branch from afc3815 to b6bbeed Compare July 11, 2025 13:02

Johnetordoff force-pushed the add-new-notifications-data-model branch 3 times, most recently from 12a9842 to 4c69f7e Compare July 15, 2025 16:58

ihorsokhanexoft added 2 commits July 16, 2025 08:38

API: Allow /v2/users/me/preprints list view to filter by title

95be0e0

Allow the User Preprints endpoint (/v2/users/me/preprints) to be filterable by title (?filter[title]=example), similar to how the User Nodes endpoint (/v2/users/me/nodes) supports this filter capability

[ENG-8224] Fixed force archive template with registration addons (Cen…

cc51cb3

…terForOpenScience#11210) Enable Product Team to Force Archive Registrations in the Admin App

Johnetordoff force-pushed the add-new-notifications-data-model branch from bb4f441 to 106679b Compare July 16, 2025 14:29

fix embargo sanctions typo

d43de2b

Johnetordoff force-pushed the add-new-notifications-data-model branch from 106679b to d43de2b Compare July 16, 2025 14:39

Johnetordoff added 3 commits July 16, 2025 10:49

add brand relationship to collectionprovider

99e26cb

fix mocks for institutional access tests

9776d92

fix file added updates for notifications

cfc1f97

Johnetordoff force-pushed the add-new-notifications-data-model branch 2 times, most recently from 4c3adc7 to e7266b9 Compare July 16, 2025 22:11

Add collections scopes to FULL_READ and FULL_WRITE

91672c0

Johnetordoff commented Aug 31, 2025

View reviewed changes

update tests

05aad5e

Johnetordoff force-pushed the add-new-notifications-data-model branch from b241b26 to 05aad5e Compare August 31, 2025 11:25

Johnetordoff commented Aug 31, 2025

View reviewed changes

revert ResourceContributorsCreateSerializer refactor

282defc

Johnetordoff commented Aug 31, 2025

View reviewed changes

Johnetordoff and others added 3 commits August 31, 2025 08:03

revert some syntax changes to shrink diff

499d823

update moderation notification tests

d3c238c

Johnetordoff force-pushed the add-new-notifications-data-model branch from 920ac12 to d3c238c Compare August 31, 2025 15:10

update moderation notification tests

8937680

Johnetordoff force-pushed the add-new-notifications-data-model branch from d3c238c to 8937680 Compare September 1, 2025 13:46

John Tordoff added 2 commits September 1, 2025 09:46

fix submission templates

de56134

Johnetordoff force-pushed the add-new-notifications-data-model branch from 1a4976f to de56134 Compare September 1, 2025 14:01

John Tordoff and others added 6 commits September 1, 2025 10:21

update subjects and mailhog tests

7981aaf

update subjects and mailhog tests

bf9a444

clean-up submissions collection tests

0a46af1

add more test fixes

698ce5d

update tests

6816040

Johnetordoff force-pushed the add-new-notifications-data-model branch from 22e81be to 6816040 Compare September 1, 2025 17:58

fix mailhog tests

6733701

Johnetordoff force-pushed the add-new-notifications-data-model branch from 3ca708c to 5d0b6b4 Compare September 2, 2025 00:20

fix more test stuff

b8fdc80

Johnetordoff force-pushed the add-new-notifications-data-model branch from 5d0b6b4 to b8fdc80 Compare September 2, 2025 00:35

John Tordoff added 2 commits September 1, 2025 20:47

fix more test stuff

89dc4ed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ENG-8064] Add New Notifications Data Model (Refactor Notifications Phase 2) #11151

[ENG-8064] Add New Notifications Data Model (Refactor Notifications Phase 2) #11151

Uh oh!

Johnetordoff commented May 20, 2025 •

edited

Loading

Uh oh!

Johnetordoff Aug 31, 2025

Uh oh!

Johnetordoff Aug 31, 2025

Uh oh!

Johnetordoff Aug 31, 2025

Uh oh!

Johnetordoff Aug 31, 2025

Uh oh!

Johnetordoff Aug 31, 2025

Uh oh!

Uh oh!

[ENG-8064] Add New Notifications Data Model (Refactor Notifications Phase 2) #11151

Are you sure you want to change the base?

[ENG-8064] Add New Notifications Data Model (Refactor Notifications Phase 2) #11151

Uh oh!

Conversation

Johnetordoff commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Changes

QA Notes

Documentation

Side Effects

Ticket

Uh oh!

Johnetordoff Aug 31, 2025

Choose a reason for hiding this comment

Uh oh!

Johnetordoff Aug 31, 2025

Choose a reason for hiding this comment

Uh oh!

Johnetordoff Aug 31, 2025

Choose a reason for hiding this comment

Uh oh!

Johnetordoff Aug 31, 2025

Choose a reason for hiding this comment

Uh oh!

Johnetordoff Aug 31, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Johnetordoff commented May 20, 2025 •

edited

Loading