Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New notification system: implementation #10922

Merged
merged 103 commits into from
Jan 4, 2024
Merged

Conversation

humitos
Copy link
Member

@humitos humitos commented Nov 23, 2023

Initial work towards the new notification system. I started adding some comments and deleting old code to understand the scope of the project.

Decisions made

  • Create an endpoint on APIv2 to be used from the builders. Builders use an API key to have write access on specific resources. We don't have this permission/authentication process implemented in APIv3 and there is no easy way to do it right now. Builders configure the API client using v2 also, so we should migrate all together. This is out of the scope of this project.
  • The font-end will use APIv3 to list and update notifications following the new patterns.
  • Some notification require "extra data" to render them properly (e.g. concurrency limit notification needs to render the actual limit number). These extra data is passed when the notification is created using format_values attribute and they are stored in a JSONField in the database, so they can be used to render the message at display time.
  • Old admin actions to send notification to owners were removed to reduce complexity. We haven't been using them in a while now and they don't seem to be super useful since we have most of them automated now.
  • Created a message "registry" so each application can register their own messages IDs to attach to the notification they send to users. This is done by creating a <app>/notifications.py and defining and registering the messages there. Then this file has to be imported from <app>/apps.py or <app>/__init__.py or similar approach.
  • Use a custom models.Manager that exposes a .add() method with de-duplication capabilities.
  • Hack old dashboard templates with a small location.reload() JS to avoid re-implementing the Knockout dynamic capabilities in the old dashboard.
  • I copied&pasted the current messages and quickly adapted some of them. However, I'm expecting to improve them during the review process by getting feedback or even in a different PR to avoid blocking it.
  • The work for "new messages" (that didn't exist before) won't be covered by this PR since I'm considering out-of-scope. For now, I'm sticking to the plan of implementing the new notification system and migrate the current notifications into the new system.
  • Build.error field is not populated anymore from the builder. Those error are now attached as Notification objects to the build itself.

ToDo

The work listed here is not mandatory of this PR, but I'm writing it down to keep track of it and don't forget:

  • create a periodic Celery task to check for notifications that should be removed since the status has changed (e.g. subscription was paid, custom domain was verified, or similar). See Notifications: cancel notifications when status changes #10981
  • "cancel" notification when state changes via Django forms
  • implement the same location.reload() hack in the new dashboard until we write the proper Knockout code using APIv3
  • remove Django messages extend dependency in case it's not used anymore
  • remove SiteNotification class completely
  • implement this changes in readthedocs-corporate at https://github.com/readthedocs/readthedocs-corporate/pull/1699
  • implement APIv3 to PATCH notifications and mark them as read for example
  • finish the migration of all the ConfigError notifications shown at build time
  • consider renaming Notification.format_values to .context which is more common in the Django world
  • implement this changes in ext-theme at Implementation of new notification system ext-theme#254
  • review messages from readthedocs-ext and implement the new notification system there as well
  • make all the old tests to pass
  • write down new test cases if needed (in case the old ones not cover current behavior)
  • write a migration for Project.skip=True that create Notification object for those (see Notifications: create notifications for projects with skip=True #10980)

Examples in screenshots

Notification error attached to a Build object

Screenshot_2023-12-21_12-24-17

Notification error attached to a Project object

Screenshot_2023-12-21_12-24-09

Notification error attached to a User object

Screenshot_2023-12-21_12-33-19

Notification on business

Screenshot_2023-12-21_13-08-30

Notification on new dashboard

Screenshot_2023-12-21_13-37-49

Closes #10917

We are not sending any kind of notification from the Django admin.
We used to use this in the past but we are not using it anymore.

I'm deleting it now to avoid re-implementing it using the new notification
system.
I'll come back to this later.
This allows to store values required for this notification at render time
Use `Notification.objects.add()` to create a notification with de-duplication
capabilities. If the notification already exists, it updates all the fields with
the new ones and set the `modified` datetime to now.

This commit also migrates the "Your primary email address is not verified" to
the new notification system, which it was when I implemented the de-duplication logic.
Use a small hack on build detail's page to refresh the page once the build is
finished. This way, we don't have to re-implement the new notification system in
the old Knockout javascript code and we focus ourselves only in the new
dashboard.
@humitos
Copy link
Member Author

humitos commented Jan 3, 2024

I'm merging this PR to avoid conflicts. I'll make any required work in following PRs.

We are using the `Notification`s attached to the object now.
These old errors are migrated to the new system by a one-time script.
@humitos
Copy link
Member Author

humitos commented Jan 3, 2024

We decided to merge this after today's deploy since I wasn't able to check everything before the deploy.

I created the script to migrate the old build.error to the new Notification objects at #10980 (comment). We can run it in production after deploying this PR.

@agjohnson
Copy link
Contributor

I think these notifications have to be attached to the correct object

Definitely, I agree with this modeling and was instead thinking we'd use a separate query like Notification.objects.filter(attached_to=ANY, news=True), or rename the news field to make more sense. I'm guessing this is only really a boolean, we don't see us needing a 3rd or more level of priority.

show notifications attached to Organization to owners in all the pages

This seems like a good first step, and solves the more immediate problem of the organization notifications. We can always change these messages later to be flagged important, and use a more general pattern like a important=True flag.

show notifications attached to Project to admins in project detail's page

👍 I also can't think of any messages that are important yet immediately. User notifications are special cased though, and could use this same pattern instead of a dedicated query/template context.

Oh also, I realized we can maybe even show project error notifications in project listings, using an error icon for any projects with error notifications. That would at least give some higher level view into projects that have problems.

create a notification page that list all the notifications

I don't recall if we've talked about this, but it could be a great UI to add. I'd say this is maybe a later phase of work, it's nice to have but not essential to the current refactor.

Copy link
Member

@stsewd stsewd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a quick look. The main concern is that we are rendering the notifications as | safe, but they include user content, that can be abused.

readthedocs/api/v2/views/model_views.py Outdated Show resolved Hide resolved
readthedocs/api/v2/views/model_views.py Outdated Show resolved Hide resolved
(UNREAD, UNREAD),
(READ, READ),
(DISMISSED, DISMISSED),
(CANCELLED, CANCELLED),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe call this "resolved"?

Copy link
Member Author

@humitos humitos Jan 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong preference. Either one is fine I think.

Resolved conveys the "problem was solved" which may not be true. Cancelled, on the other hand, doesn't. So, 🤷🏼

readthedocs/projects/notifications.py Show resolved Hide resolved
@agjohnson
Copy link
Contributor

I created the script to migrate the old build.error to the new Notification objects

Noted in chat, but there were some issues with the notification migration and it sounds like we're probably leaning towards skipping this. The template code and KO view would stay the same as it is currently, or wouldn't change much if we supported the list of notifications with the same logic.

@agjohnson
Copy link
Contributor

created all the issues that are out-of-scope from this PR to put in the roadmap and work in the near future

Thanks for creating all of those! I've added them to our roadmap and put the API changes on this sprint. The API pieces will be required to make progress on any front end pieces and can happen before we get to any HTML/JS.

I opened up readthedocs/ext-theme#259 to discuss how to proceed on the actual front end work, but I'm still feeling that these will be relatively minor compared to the last of the work on the backend.

This reverts commit d4951a8.

We decided to keep this code around for now.
See #10980 (comment)
If the message ID is not found we return a generic message to avoid breaking the
request and we log an internal error to fix the issue.
@humitos
Copy link
Member Author

humitos commented Jan 4, 2024

I applied all the suggestions. I'm merging this so other people can start testing it locally in their day to day development. Please, let me know if anything doesn't work as you expected and I will try to fix it before deploying it next week.

@humitos humitos merged commit e778db5 into main Jan 4, 2024
2 of 4 checks passed
@humitos humitos deleted the humitos/new-notification-system branch January 4, 2024 10:58
humitos added a commit that referenced this pull request Jan 8, 2024
* Notifications: small fixes found after reviewer

Use single `{}` since we are using `.format()` to format these strings.

* Handle missing `.format()` key

Avoid internal error and log the exception so we can figure it out how to solve
it. This happens when the `Notification` does not have _all_ the required `format_values`.

* Protection agasint XSS when rendering notifications

See #10922 (comment)

* Test for missing key in format values and XSS protection

* Update common/

* Lint

* Document we only support `str` and `int` for now in `format_values`

We don't support nested dictionaries in `format_values` or random objects.
Only `str` and `int`. That should be enough for now.

Skip all the values that are not `str` or `int` from the format values to render
the messages.

* Typo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Notifications: implement the new system
3 participants