Switch v2 libraries to Learning Core data models

[ormsbee]

Description & Supporting Information

This moves the Content Libraries V2 backend from Blockstore over to Learning Core. For high-level overview and rationale of this move, see

• [DEPR]: Blockstore public-engineering#238

Supersedes:

• Import v2 libraries content into openedx-learning data models #33577 and
• Read support for v2 library content using openedx-learning models #33579.

Requires:

• Changes for v2 content libs openedx-learning#143
• refactor: normalize component types with ComponentType openedx-learning#148
• Combine RawContent and TextContent into Content openedx-learning#149

A note on backwards incompatibility

On the Blockstore DEPR, we did not hear that anyone is relying on Blockstore-backed V2 Content Libraries in production. So, we are applying this port in-place without any migration path from Blockstore to Learning Core. If you were using V2 Content Libraries in production: your libraries will stop working. However, we are not yet dropping the the ContentLibraries.bundle_uuid field, so it is still possible to manually migrate Blockstore data over to Learning Core. If you're in the situation, reach out to us. The ContentLibraries.bundle_uuid field will be dropped in Sumac.

Status

The biggest functional gap here is static asset support. I'm gong to try to merge this once I have tests fixed, even if it means we take a hit in terms of a regression in some functionality like static asset support–with the understanding that we're going to keep iterating to make this ready for Redwood.

TODO in this PR:

• Convert remaining Learning Core model manipulation to use api module calls instead.
• Squash the new migrations into one.
• So much test removal/cleanup/rewriting.

TODO in follow-up PRs:

• Follow-ups from Learning Core Switchover #34283

Bugs I'm seeing (not sure if I introduced them or not):

• There's sometimes a JS error when a new block is created, about not being able to set display_name on null. But the value is written correctly into the database and shows up correctly when reloading.
• The ProblemBlock editor seems to revert to code editing after the initial creation–did I change the OLX serialization?

FYI @kdmccormick, @bradenmacdonald

Testing

This PR's sandbox can be compared against:

• [No Merge] Sandbox to test the migration from Blockstore to LC #33936

If you want to run this on your devstack, you must put the following in your cms/envs/private.py:

from .devstack import FEATURES

FEATURES['ENABLE_LIBRARY_AUTHORING_MICROFRONTEND'] = True

PR sandbox

Settings

PLUGINS:
    - mfe
    - grove
    - s3
    - library-authoring-mfe

Tutor requirements

git+https://github.com/overhangio/tutor.git@nightly
git+https://github.com/overhangio/tutor-mfe.git@nightly
git+https://gitlab.com/opencraft/dev/tutor-contrib-grove.git@main
git+https://github.com/hastexo/tutor-contrib-s3.git@main
git+https://github.com/openedx/openedx-tutor-plugins@main#subdirectory=plugins/tutor-contrib-library-authoring-mfe

[ormsbee]

Okay, I've updated to the changes in openedx/openedx-learning#149, and it seems to be working locally. In the morning, I'll squash what I have here, do a rebase to what's on master today, and then see what explodes in terms of conflicts.

[bradenmacdonald]

Personally, I would say that this is definitely "nice to have" in the UI (indicate which libraries have unfinished changes, so you can find them and finish your work), but definitely not a requirement for MVP.

But there are other ways to work around this from a UX perspective - for example, trying to go for something more like Google Docs, where changes are encouraged to be published immediately and if you close a library page / XBlock editor without publishing your changes to any given XBlock it will prompt you "are you sure you want to leave without publishing your changes?". In that scenario, unpublished changes are rare and we don't need to highlight them as much.

[ormsbee]

There's one major outstanding bug that I'm seeing, and that is that it looks like the Learning Core XBlock runtime is serializing the default values into the metadata dict that comes across the fields REST API call that the editor uses.

Looking back, the XBlock runtime for Learning Core is something I copy-pasta'd from Blockstore, with some things slashed out. There are parts of the original blockstore runtime that I don't really understand as well as I should. @bradenmacdonald: May I grab an hour of your time on Monday to walk through the Blockstore XBlock runtime and FieldData?

[ormsbee]

@kdmccormick, @bradenmacdonald: The openedx/core/djangoapps/content_libraries/views.py module is ready for review.

[open-craft-grove]

Sandbox deployment successful.

Sandbox LMS is available at https://pr-34066-2432b1.sandboxes.opencraft.hosting
Sandbox Studio is available at https://studio.pr-34066-2432b1.sandboxes.opencraft.hosting

Instance Tutor config : https://axim-sandbox-build-logs-4626178240.nyc3.digitaloceanspaces.com/pr-34066-2432b1-51564656-6156678534-tutor-config.yml
Instance Grove config : https://axim-sandbox-build-logs-4626178240.nyc3.digitaloceanspaces.com/pr-34066-2432b1-51564656-6156678534-grove-config.yml
Instance Python requirements : https://axim-sandbox-build-logs-4626178240.nyc3.digitaloceanspaces.com/pr-34066-2432b1-51564656-6156678534-tutor-requirements.txt

[ormsbee]

@kdmccormick, @bradenmacdonald: The re-worked openedx/core/djangoapps/xblock/runtime/learning_core_runtime.py is ready for review.

[ormsbee]

Removed the create-sandbox label so that the sandbox gets destroyed. Will remake it with a more backwards compatible migration later.

[kdmccormick]

I just reviewed learning_core_runtime.py. Two suggestions, and one thing I think is actually broken (the get_learning_context_impl all).

[ormsbee]

@kdmccormick, @bradenmacdonald: openedx/core/djangoapps/content_libraries/api.py is ready for review.

[ormsbee]

@kdmccormick: incorporated your feedback on learning_core_runtime.py

[ormsbee]

xblock/api.py is ready for review.

[ormsbee]

force-pushed a rebase onto the latest master

[ormsbee]

@kdmccormick, @bradenmacdonald: Everything here should be reviewable now.

[ormsbee]

The last actual issue I'm aware of is a set of unit tests that fails in CI but not locally. It involves signals and mocks, so I don't know if there's something weird going on there. These same tests were working earlier.

[open-craft-grove]

Sandbox deployment failed 💥
Please check the settings and requirements.
Retry deployment by pushing a new commit or updating the requirements/settings in the pull request's description.
📜 Failure Logs
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[open-craft-grove]

Sandbox deployment failed 💥
Please check the settings and requirements.
Retry deployment by pushing a new commit or updating the requirements/settings in the pull request's description.
📜 Failure Logs
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[open-craft-grove]

Sandbox deployment failed 💥
Please check the settings and requirements.
Retry deployment by pushing a new commit or updating the requirements/settings in the pull request's description.
📜 Failure Logs
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[bradenmacdonald]

Those changes look great @kdmccormick 👏🏻

[open-craft-grove]

Sandbox deployment failed 💥
Please check the settings and requirements.
Retry deployment by pushing a new commit or updating the requirements/settings in the pull request's description.
📜 Failure Logs
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[open-craft-grove]

Sandbox deployment failed 💥
Please check the settings and requirements.
Retry deployment by pushing a new commit or updating the requirements/settings in the pull request's description.
📜 Failure Logs
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[kdmccormick]

OK, I haven't the faintest idea why this PR's sandbox is failing migrations, but I did confirm that:

• migrations applied fine on tutor local
• migrations applied fine on this PR's sandbox, which was originally created using master, and then was updated with this PR's code.
  • The Blockstore-authored libraries render as title-less cards which spin when you click on them. I think that's completely reasonable for now. If, in the future, we want to degrade more gracefully when there's no backing learning_package for a content_library, then we could do that.
    

[open-craft-grove]

Sandbox deployment failed 💥
Please check the settings and requirements.
Retry deployment by pushing a new commit or updating the requirements/settings in the pull request's description.
📜 Failure Logs
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX production environment.

[edx-pipeline-bot]

2U Release Notice: This PR has been deployed to the edX production environment.

[ormsbee]

🎉 @kdmccormick, @bradenmacdonald: Thank you for getting this over the line! It feels so nice to finally have the learning core foundations in place under libraries!

[kdmccormick]

Hey @ormsbee , is setting def_id=None here the long-term intention?

Context: I'm on a quest to type-annotate XBlock, and I'm looking for the correct annotation for def_id. Currently I have:

DefinitionId: t.TypeAlias = DefinitionKey | UsageKey | ObjectId | LocalId | str

and that handles that ModulestoreRuntimes. But in order to handle the LearningCoreRuntime, looks like I'll need to add | None to that. Happy to do that, just want to check your instincts before proceeding.

cc @bradenmacdonald

[ormsbee]

Yeah, it's a long term intention. I think that the def_id as it exists today is an unhelpful leak of Split's internal storage details, and that it's more trouble than it's worth. I'd like to eventually kill it off entirely.