feat!: Drop the block_structure.storage_backing_for_cache WaffleSwitch #35185

feanil · 2024-07-25T20:01:47Z

This work is part of DEPR openedx/public-engineering#32

See the above link for rational. This PR make is so that the course structure
cache is always backed by persistent storage.

BREAKING CHANGE: The block_structure.storage_backing_for_cache will no
longer exist and its value will be ignored. If you have this switch set
in your instance you can remove it. The backing cache is now always ON.

Questions

Q: Do we need to run any commands to generate the initial
cache for all existing operators? Are there docs we can link to or create for
this?

A: Unclear but, as a part of the get method on the BlockStructureStore if the structure is
not found in the cache/storage, it wll be retrieved from the backing store and added
to the cache. But currently the same check is not happening for getting the structure
from the model. Should we update this? I think operators will need to run the
generate_course_blocks management command before they deploy this code or
the will see errors in their deployment.

eg. ./manage.py lms generate_course_blocks --all_courses --settings=production

feanil · 2024-08-20T14:37:33Z

@ormsbee per our conversation last week, I dug a bit deeper into the query count differences and it looks like a lot of the extra queries are savepoints around updating a block blockstructure and then also setting the modified date on the parent block when a block structure changes. I've collected the before and after queries for the test that changed by more than one in this file.

ormsbee

Some minor questions and requests.

In terms of the deployment risk for this: I agree that operators should run the backfill command when upgrading. Honestly, operators should in general probably run a simulate_publish on all their courses every time they upgrade, because not all devs will be as conscientious as you are about thinking through their upgrade process.

I suspect that whether to lazily re-generate the data on read or to just fail the request has to do with the failure mode when something goes wrong with the publish process. The collect process is so expensive for large courses that it's probably better for a high volume site to fail the request altogether than to try to dynamically regenerate on the fly.

Unclear but, as a part of the get method on the BlockStructureStore if the structure is not found in the cache/storage, it wlll be retrieved from the backing store and added to the cache. But currently the same check is not happening for getting the structure from the model. Should we update this?

I don't think we should update it. Grabbing a completely generated blob of data from S3/filestore and stuffing it into memcached/redis is relatively cheap. Regenerating the collected data is much more expensive.

At edX, the publish process would sometimes break block structure collection for a certain course because of some content edge case. The result would be that some data would be unavailable or out of date (e.g. the outline not showing a newly added subsection). It's frustrating for users, but it's better to show that stale data quickly than to constantly hammer the site to break on the same errors over and over again.

cms/djangoapps/contentstore/management/commands/import.py

openedx/core/djangoapps/content/block_structure/tasks.py

ormsbee · 2024-09-09T14:48:35Z

lms/djangoapps/grades/rest_api/v1/tests/test_gradebook_views.py

+        # Review Question: Should we be doing this here? Does it make sense to do
+        # this in the xmodule/modulestore/tests/factories.py BlockFactory class
+        # as a part of the publish there?
+        bs_api.update_course_in_cache(self.course_data.course_key)


I think doing it here manually is okay. I'd probably just enable the published signal for this test case and let it build everything–to test part of the machinery as well, since the triggers for that may change over time. It might involve splitting off this test from the rest of the test case though.

I wouldn't put it in xmodule/modulestore/tests/factories.py though, since that's called by a lot of things that may not require this functionality.

Makes sense, I think I'll leave it as is then.

ormsbee · 2024-09-09T14:56:09Z

lms/djangoapps/grades/tests/integration/test_events.py

@@ -76,44 +77,45 @@ def setUp(self):
        CourseEnrollment.enroll(self.student, self.course.id)
        self.instructor = UserFactory.create(is_staff=True, username='test_instructor', password=self.TEST_PASSWORD)
        self.refresh_course()
+        # Since this doesn't happen automatically and we don't want to run all the publish signal handlers
+        # Just make sure we have the latest version of the course in cache before we test the problem.
+        bs_api.update_course_in_cache(self.course.id)


Again, I'd probably emit the signal and let everything run (I think this is the most expensive part of it anyhow), but I think either way is okay.

So this class already has the course_published signal enabled and so I thought he issue might be that the CourseFactor needed the emit_signals parameter set to true, but even with that this did not work. How important is it that this be figured out for this PR? I can dig into it further if needed.

Does it work when you test the functionality locally via browser? If it doesn't happen automatically, it's possible that something is going wrong (e.g. it's going to trigger later on lazy-load). I'm really sorry to keep dragging this PR out, but so many things rely on block structures that I think it's worth investigating why it doesn't refresh.

@ormsbee I've tested it locally and created a new course and made various changes to the course sections/subsections and I can see them all propagate correctly to the course outline page. Let me know if there is anything else you want me to test.

lms/djangoapps/course_api/blocks/transformers/tests/test_milestones.py

openedx/core/djangoapps/content/block_structure/tests/helpers.py

Any places where the storage backing cache for block structures was on conditionally previously it will be ON by default.

Since the cache is now always on, remove test cases that test the case when it's disabled.

Remove the `enable_storage_backing_for_cache_in_request` function and its uses in the platform. The function is no longer needed because the storage backing for the block_structure cache will be ON by default moving forward. BREAKING CHANGE: This `enable_storage_backing_for_cache_in_request` function no longer exists and any calls to it should be removed. The cache it enables is now always ON.

This work is part of DEPR openedx/public-engineering#32 Now that we've removed all uses for this switch remove the decleration as well. BREAKING CHANGE: The `block_structure.storage_backing_for_cache` will no longer exist and its value will be ignored. If you have this switch set in your instance you can remove it. The backing cache is now always ON.

Previously, we were not caching BlockStructures to the database when we were adding them to the store by default. Now that we are doing that, the BlockStructureFactory test failed because it was taking a shortcut that would no longer work. It was just creating a blockstructure that had relations but not any block information. Now that we're always persisting updates to the database, this broke because to persist the structure to the database, we have to look up the block information from the block_structure which now fails. This change updates the test mixin to add more data so that the content can be persisted to the database successfully as a part of this test.

We don't call the modulestore or update the cache here now that we are backed by the database model. Previously the cache would change because the `_encode_root_cache_key` function in `BlockStructureStore` class used the `BlockstoreBlockData.VERSION` as a part of the cache key when the data was not being cached in a DB model.

Now that the model backed cache is on by default, we don't need to keep the StubModel object around.

With the new block_structure model cache enabled by default, we're donig queries to it as a part of other tests.

Because signals are disabled by default for performance reasons, this doesn't happen automatically. So we manually refresh the course in the cache after all the changes have been made so that the course in the cache matches the latest version in the modulestore.

This test had a redundant call to get the course data from the store because that already happens at the end of the setup function. And also because expected call structure was being built inside the assert, it made it harder to inspect when debugging. Make the code a little bit easire to debug in case we're back here in the future.

Provide a link to more docs from the code in case we need more context in the future.

…le switch.

edx-pipeline-bot · 2024-11-06T17:42:22Z

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

edx-pipeline-bot · 2024-11-06T18:01:26Z

2U Release Notice: This PR has been deployed to the edX production environment.

edx-pipeline-bot · 2024-11-06T18:22:00Z

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

edx-pipeline-bot · 2024-11-06T18:39:48Z

2U Release Notice: This PR has been deployed to the edX production environment.

edx-pipeline-bot · 2024-11-06T19:04:10Z

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

edx-pipeline-bot · 2024-11-06T19:23:27Z

2U Release Notice: This PR has been deployed to the edX production environment.

edx-pipeline-bot · 2024-11-06T19:23:27Z

2U Release Notice: This PR has been deployed to the edX production environment.

feanil force-pushed the feanil/blockstructure_cache branch from 88f76e1 to 092f565 Compare July 29, 2024 17:44

feanil force-pushed the feanil/blockstructure_cache branch 2 times, most recently from 18d943f to 2548c2c Compare August 8, 2024 18:39

feanil requested review from ormsbee, bradenmacdonald and kdmccormick August 8, 2024 18:40

feanil force-pushed the feanil/blockstructure_cache branch from 2548c2c to fd1e239 Compare August 9, 2024 14:26

feanil marked this pull request as ready for review August 9, 2024 17:24

feanil force-pushed the feanil/blockstructure_cache branch from fd1e239 to 8f4ae4e Compare August 13, 2024 18:31

feanil force-pushed the feanil/blockstructure_cache branch from 2e03a8e to 854b490 Compare September 4, 2024 14:37

ormsbee requested changes Sep 9, 2024

View reviewed changes

kdmccormick removed their request for review September 16, 2024 17:13

feanil force-pushed the feanil/blockstructure_cache branch from 854b490 to a64dad5 Compare September 23, 2024 17:28

feanil requested a review from ormsbee September 23, 2024 18:21

feanil force-pushed the feanil/blockstructure_cache branch 2 times, most recently from 5541aae to 9fb8492 Compare October 2, 2024 14:13

feanil added 13 commits November 5, 2024 10:44

feat: Turn ON the storage backing for cache by default.

16d440a

Any places where the storage backing cache for block structures was on conditionally previously it will be ON by default.

test: Remove test variants that test without a cache.

57cdbfa

Since the cache is now always on, remove test cases that test the case when it's disabled.

feat: Remove the unused StubModel class.

e5f698a

Now that the model backed cache is on by default, we don't need to keep the StubModel object around.

test: Update query counts.

db266e2

With the new block_structure model cache enabled by default, we're donig queries to it as a part of other tests.

docs: Cross link to wiki docs related to block structures.

8741417

Provide a link to more docs from the code in case we need more context in the future.

fix: Send the course publish signal on course import.

80ed5ee

fixup! test: Update the course in the cache after it's got new content.

d58dea3

feanil added 3 commits November 5, 2024 10:44

fixup! test: Update the course in the cache after it's got new content.

a6e5b81

fixup! test: Update the course in the cache after it's got new content.

398160c

fixup! feat!: Drop the block_structure.storage_backing_for_cache waff…

4768568

…le switch.

feanil force-pushed the feanil/blockstructure_cache branch from 9fb8492 to 4768568 Compare November 5, 2024 16:09

ormsbee approved these changes Nov 6, 2024

View reviewed changes

feanil merged commit 5904b54 into master Nov 6, 2024
49 checks passed

feanil deleted the feanil/blockstructure_cache branch November 6, 2024 15:52

feanil mentioned this pull request Nov 6, 2024

[DEPR]: block_structure.storage_backing_for_cache in edx-platform openedx/public-engineering#32

Closed

robrap mentioned this pull request Nov 7, 2024

Remove waffle switch data for block_structure.storage_backing_for_cache edx/edx-arch-experiments#848

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: Drop the block_structure.storage_backing_for_cache WaffleSwitch #35185

feat!: Drop the block_structure.storage_backing_for_cache WaffleSwitch #35185

feanil commented Jul 25, 2024 •

edited

Loading

feanil commented Aug 20, 2024

ormsbee left a comment

ormsbee Sep 9, 2024

feanil Sep 23, 2024

ormsbee Sep 9, 2024

feanil Sep 23, 2024

ormsbee Oct 7, 2024

feanil Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

feat!: Drop the block_structure.storage_backing_for_cache WaffleSwitch #35185

feat!: Drop the block_structure.storage_backing_for_cache WaffleSwitch #35185

Conversation

feanil commented Jul 25, 2024 • edited Loading

Questions

feanil commented Aug 20, 2024

ormsbee left a comment

Choose a reason for hiding this comment

ormsbee Sep 9, 2024

Choose a reason for hiding this comment

feanil Sep 23, 2024

Choose a reason for hiding this comment

ormsbee Sep 9, 2024

Choose a reason for hiding this comment

feanil Sep 23, 2024

Choose a reason for hiding this comment

ormsbee Oct 7, 2024

Choose a reason for hiding this comment

feanil Nov 6, 2024

Choose a reason for hiding this comment

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

edx-pipeline-bot commented Nov 6, 2024

feanil commented Jul 25, 2024 •

edited

Loading