[DEPR]: Course Discovery Service

[bradenmacdonald]

Proposal Date

23 September 2024

Ticket Acceptance Date

23 October 2024

Technology Removal Date

2025-2027

First Open edX Named Release Without This Functionality

TBD

Rationale

The Course Discovery Service provides a variety of optional functionality in Open edX, some of which is required for other services like the edx-enterprise suite. However, the fundamental architecture of the discovery service has proven to be problematic, and it is currently a limiting factor that makes it much more difficult than it should be to develop and integrate custom functionality for the Open edX platform such as ecommerce, enterprise integrations, subscriptions, and programs.

Here are some of the problems with its current architecture:

• Non-default: The discovery service is not installed by default on most instances of Open edX, which means that "core" functionality is never built using its APIs.

• Out of Sync Course Data: Fundamentally, the discovery service maintains at least two separate sets of models about courses (i.e. the Course model in its MySQL database, and the CourseRunDocument in Elasticsearch). Though there are various systems for keeping this data in sync with Studio (which holds the actual "source of truth"), they are imperfect and synchronization issues are a frequent source of bugs and errors.

• Bi-directional sync: Because the discovery service is the source of truth for some pieces of data, it actually syncs its data to Studio as well as from Studio. This has caused a variety of issues, including bugs with an infinite cycle of updates.

• Other Redundant/Conflicting Data Models: The discovery service has its own database tables for storing important data that is already stored by Studio/LMS (Discovery Organization vs. Core Organization, Mode vs. CourseMode, Person vs. UserProfile, CourseEditor vs. CourseAccessRole). Often, these models have either subtle or substantial differences from the Studio/LMS version, and there is no guarantee that they are in sync at all.

  • Part of the reason this problem has arisen is that there was historically no good mechanism to extend the models in the core platform (and their corresponding APIs) using plugins.

• Competing APIs: some operations like "list courses available to me" can be done with the core platform APIs, the discovery service APIs, or the edx-enterprise APIs. Needless to say, they have different behavior, functionality, caching, etc.

• Kitchen Sink / edX-Specific Functionality: The Course Discovery Service includes, in no particular order:

  • hard-coded edX-trademarked program types
  • hard-coded course run types like paid-bootcamp and bootcamp-2u (not to be confused with Tracks, Modes, or Seat Types which are all related)
  • Subject
  • Topic
  • Fact
  • CertificateInfo
  • SEO Metadata
  • "Taxi" (2U form library) lead capture data
  • random 2U "additional fields"
  • Collaborators
  • Product Value data (for Cost-per-click??)
  • product GeoLocation
  • Course location restriction
  • "CourseReview model to save Outcome Survey analytics coming from Snowflake"
  • Entitlements
  • Corporate Endorsements
  • Program Subscriptions
  • Degrees
  • Curriculum
  • Pathways
  • Person's Social Networks
  • an "API Client...to communicate with google spread sheets and drive images"
  • some Salesforce thing
  • various functionality built on Algolia
  • and many more things.

  It may be the case that the platform lacked ready-to-use extension points at the time these features were required, but they should have been built as extensions regardless.

• Problematic Programs implementation: If you want to group courses together into a "Program", you can do so using the discovery service programs feature. However: it is not well documented; it is limited in functionality; its only management UI is the Django admin; it is not flexible and only supports 2U's original use cases; and it depends on various other apps like a marketing site, ecommerce, etc.

Removal

The entire Course Discovery microservice and the corresponding code in edx-platform that integrates with it. However, see below - this is not going away anytime soon.

Replacement

To rectify this situation, this proposal calls for a series of improvements to the Open edX core (edx-platform and openedx-learning, etc.) that add a "catalog" feature and focus on improving platform extensibility, so that the kitchen sink of features listed above could be largely built using plugins. At the same time, the Discovery service would be officially deprecated.

New Flexible Catalog App

To replace one of the core features of the Discovery Service, a new "Catalog" application should be built into the core platform. The purpose of this app is to provide a unified API for answering the question of "What courses are available to me?". It should be extensible using plugins so that things like Subscriptions and Enterprise Licensing can be accounted for.

This catalog app may also have a flexible concept of "programs" which are essentially groups of related courses, but where each program can define requirements like "pass course A then B, then either C and/or D in order to complete the program." Significant product research and planning is required to define such requirements in more detail.

The catalog app should be developed with "micro-courses" in mind as well - courses significantly smaller than the average MOOC and more like a "Section" of today's typical courses.

The detailed design of the catalog app will happen in future ADRs (and/or an OEP) if this DEPR proposal is accepted. There is already a draft of one initial ADR to add an edx-platform representation of "Catalog Courses".

Extending Core Models

Many of the other "kitchen sink" features listed above were developed in the Discovery service because there wasn't a better place to put them at the time. Instead of finding the least bad place to put specialized functionality like this, we should modify the platform so that the core models can be extended by plugins.

For example, a plugin should be able to add custom fields to the UserProfile model, and have these fields (optionall) show up anywhere user profiles are seen or edited. Likewise the core Course model should be extensible, so a plugin could add (e.g.) a "Price" field to each course and choose to have it show up in the catalog API data, the default catalog UI (when user's browse for courses), and the default course "Advanced Settings" UI. The Open edX Hooks Extension Framework already provides some mechanisms for this, but others will need to be proposed in future ADRs/OEPs.

Roles & Permissions

Achieving a good implementation of a core catalog app and extensible core models will almost definitely depend on a major refactor of roles and permissions, the lack of which has been a long-standing pain point in Open edX.

Deprecation

Deprecating the Discovery Service now means three things:

First, proposing any new functionality for the Open edX Project whose implementation depends on the Discovery Service would be rejected.

Second, maintenance of the Discovery Service would become entirely the responsibility of those organizations that [still] use it (e.g. 2U).

Third, eventually the corresponding hooks and integration code in edx-platform (e.g. sync code, "programs cache, etc.) would be removed. This is likely many years away (see below). However, if this DEPR is accepted, we will make an effort to flag all of this code and corresponding APIs so that it's clear it will be eventually removed, and should not be used for implementing new functionality.

Migration

For any functionality that currently depends on the Discovery Service (e.g. "Programs"), there will be no effort to remove corresponding integration code from edx-platform until an alternative approach along the lines proposed has been developed, stablized, and released for enough time that organizations have a reasonable migration path. Details of that will depend on the implementation. Those who currently depend on the discovery service for functionality critical to their organization may continue to use it for foreseeable future, provided they are able to contribute to its ongoing maintenance.

The goal of this DEPR proposal is not to take the discovery service away as soon as possible, but to focus community resources on developing a better alternative, and to encourage stakeholders to avoid building any new features on top of the discovery service from this point forward.

[felipemontoya]

I'm all for the end-goal of this proposal.

I think every provider has met issues with course-discovery and would like to see a replacement that is actually part of the core, is organization agnostic and can be extended. I know as a provider I definitely want that. I even proposed a pattern of hooks to address this issue among others when I was looking at the possible roadmap for hooks for this year and beyond.

Implement Hook Pattern in Core Components of IDAs
Implement a standardized pattern of hooks in the platform's core components and other IDAs, which currently lack extensibility entry points such as course about page, course progress, search, homepage, login/registration, learner-dashboard, profile page, certificates, credentials, programs page, ecommerce APIs, discovery, catalog and enterprise views. This will involve adding filters at the beginning of each process and firing events at the end, allowing organizations to customize core logic to fit client needs.

Taken from this hooks roadmap document

Something that this proposal is not considering is how the information of the catalog will be displayed. I understand that 2U uses the catalog as an API to feed into their marketing site which is a whole other system also not in the core. Perhaps we could leverage a Course search MFE such as the one proposed here

Some specific comments:

• new catalog app: should this proposal take off, I believe we should put the core primitives like programs or catalog and write the most basic implementation possible. Then leave every bit of product development to plugins. I don't think that requirements for courses or anything so specific should be implemented in core. Mostly a models and calls to extension points should live there.

• extending core models: I love the idea. I however don't know how we could bring this into the hooks framework in a performant way. I'll be happy to collaborate on this until we reach a good solution.

• roles: this is a whole project on itself.

[bradenmacdonald]

Something that this proposal is not considering is how the information of the catalog will be displayed. I understand that 2U uses the catalog as an API to feed into their marketing site which is a whole other system also not in the core.

@felipemontoya My thinking is that a basic "Find Courses MFE" should be included with the platform (similar to how the LMS currently has a super basic page to browse available courses):

It would just use the new core catalog service to show courses to the current user (or to users who aren't yet logged in). I expect that it would be super basic. However, because it's using the new catalog service, it would correctly account for any plugins (subscriptions, enterprise, permissions) that are affecting the catalog - it just might not have any ability to show details of those plugins in the UI. So anyone who needs something more elaborate or custom can either (A) extend it using FPF, (B) replace it with their own custom MFE, or (C) turn it off completely and just integrate the REST API data into their existing external website or external LXP.

[felipemontoya]

Yes, this makes a lot of sense. In that case it would be even more important to lay the correct groundwork during the MFE conversion (platform-roadmap/issues/375) . CC @GlugovGrGlib

[tecoholic]

@bradenmacdonald A huge thanks for this proposal. As someone provision course-discovery and edx-enterprise suite, the recommendations for a new catalogue app and extension of the data models makes a lot of sense.

@felipemontoya

new catalog app: should this proposal take off, I believe we should put the core primitives like programs or catalog and write the most basic implementation possible. Then leave every bit of product development to plugins.

200% agree with this. If I boil down course-discovery and enterprise-catalog, it is simply a set of data and business rules made for a single business, that everyone else trying to use the software have to agree to or work around to satisfy their requirements. I am spending an unorthodox amount of hours just trying to get courses to show up for the learner in enterprise-learner-portal MFE. Business rules of which course gets displayed for which user should be left to the individual organisations to maintain as simple openedx-filter plugins. (side thought: maybe someone should build a low-code/GUI tool that will allow this to be done dynamically without having to put things down in Python config and requiring redeployment)

Some of the relevant work at OpenCraft

While Braden's proposal comes with the broader experience with the stack. I want to highlight a couple of things that are relevant to this proposal.

Note: I don't want this to be taken as an OpenCraft ad of sorts. Trust me, I am not authorised to do it. I merely want to highlight these things so, someone will tell me if there is a better way to do these things or at least provide an opportunity for the community to say, instead of doing this individually, let's do this together.

The catalog app should be developed with "micro-courses" in mind as well - courses significantly smaller than the average MOOC and more like a "Section" of today's typical courses.

There is a plugin that does this from existing courses. - https://github.com/open-craft/section-to-course. It takes a section of a course and publishes it as an independent course. I believe it also allows syncing the content if the parent course is ever updated (might be manual at the moment, but could be automatic).

This catalog app may also have a flexible concept of "programs" which are essentially groups of related courses, but where each program can define requirements like "pass course A then B, then either C and/or D in order to complete the program." Significant product research and planning is required to define such requirements in more detail.

https://github.com/open-craft/learning-paths-plugin/. This is a very early stage project that tried to implement "Programs" as a plugin. "Programs" is a merely a collection of a course, whereas with learning-path we are trying to implement a sequential -or- non-sequential collection with possible inclusion of micro-credentials like Badges.

[bradenmacdonald]

Here is a recent proposal from the community that gives an example of wanting to extend the core platform course model with custom fields: openedx/edx-platform#35593 - seems like a very relevant example.

[regisb]

Thanks a lot for putting this together @bradenmacdonald! I'm looking forward to not having to run a separate discovery service.

The great thing about this DEPR announcement is that it doesn't really require us to build anything new -- that is, until someone steps forward and tells us that Discovery is a required feature for them. Let's use this announcement to gauge community usage. We can then decide whether we want to backport some features in the core, and how.

Wouldn't you want to name a specific release as a deadline? Otherwise I'm afraid there won't be a sense of urgency in the community. I propose to target Teak.

[OmarIthawi]

I support this removal. Discovery, credentials, edx-enterprise and licensing-manager has been all 2U-specific.

It's more likely than not, that Discovery can be better implemented in a customer Marketing Site (e.g. Drupal or similar) than into an edx-platform tied service.

[kdmccormick]

Thanks Braden!

Could you clarify what happens if this is Accepted? Would we archive course-discovery, so that it is not part of Teak?

[bradenmacdonald]

@kdmccormick At a minimum, there will be a ban on developing any new functionality on top of the discovery service. I'd like to say that it won't be officially supported at all as of Teak (but perhaps an org like 2U may continue to maintain and use it if needed for their org's requirements). But I want to hear about who's using it, how they'd be affected, and how willing they are to help build replacement(s) for it and on what timeframe.

[feanil]

I think we should plan on dropping course-discovery by Teak unless there is a lot of community interest in reforming it. If 2U wants to continue to use it and maintain it, I think that can be done off of a fork like with some other services.

[scottrish]

Apologies for coming to this conversation late. While I generally agree with the proposal I think it's important that there is a migration path off the current Discovery service before it is deprecated. In particular I am not in favor of dropping official support until such time that the proposed replacement is available.

[regisb]

@scottrish Why not? Are there features of course-discovery that are required for you?

[DawoudSheraz]

Hi. Representing 2U Phoenix/Inspire squad (@openedx/2u-phoenix) currently managing and contributing to course-discovery, this service is an integral part of 2U business needs and powers major parts of edx.org marketplace. 2U has been committing code to this codebase for a while. There have not been any major community-driven initiatives/features solely for course-discovery in quite a while. Most of the additions have been by edX/2U teams. It is understandable why Axim/Open edX community would want to deprecate this service.

We don't have immediate plans to fork this repository unless Axim / Community decides to make changes that will impact 2U's business needs. We will closely follow this ticket for deprecation status and share any action we are taking regarding forking or updating maintainership.

cc: @julrusak

[scottrish]

@regisb We currently use eComm coupons (we don't support purchasing but do use it for fulfilling purchases made elsewhere), enterprise catalogs, enterprise enrollment URLs, enterprise coupons and also offer the enterprise enrollment APIs although nobody is currently using them.

[bradenmacdonald]

Hi everyone. While I'm hearing broad support for this proposal, I guess my proposal has been too vague about the actual implications of this, so here is a more detailed proposal based on what I've heard so far:

Soon (before end of year):

• The discovery service repo will be moved from the openedx org to the edx (2U) org, and 2U will be responsible for maintaining it, with support and collaboration from other users like Pearson.
• Discovery service will no longer be part of the Open edX release process (Teak, etc.) nor will it be part of any Open edX programs like Maintenance or Core Contributor. The tutor plugin will be archived, though the community can fork it and maintain it as a contrib plugin if needed.
• The catalog discovery service client in edx-platform will be marked as deprecated, but will continue to be part of the platform, maintained by 2U.
• Everything that 2U and Pearson are currently using will continue to work as is.
• We start to compile a list of all the discovery service usage in edx-platform, and create GitHub issues to track each one and discuss if/how to replace it. Examples include:
  • any UI pieces gated by ENABLE_COURSE_DISCOVERY
  • the api_admin catalog APIs
  • things like get_pseudo_course_overview which uses the discovery service but it exists only to show certificates granted to deleted XML courses on edx.org - this could be replaced with a hook in the "get user certificates" code.

Next steps (timeline TBD):

• We start to push forward the design of a new catalog app, extensible core models, etc.
• We add more hooks to the platform, and as we do so the discovery integration code can be removed, and the discovery service can be updated to use generic events and filters.
• Once we develop the new core catalog app, we can create a plan for a successor to the "programs" feature and potentially changing "enterprise" to use the new catalog service instead of discovery, and/or developing a completely new enterprise integration.

"U" release:

• My hope would be that by the "U" release, we can fully remove the catalog discovery service client and all references to the discovery service from edx-platform. This doesn't mean that the discovery service no longer works, just that the platform no longer explicitly makes calls to it. Discovery would use hooks/filters/events and UI plugins/slots, just like any other integration.

What do you think? (Feel free to suggest something different if you don't like this plan)

[e0d]

I am supportive of this proposal and have begun speaking to folks to coordinate building a plugin for the LMS for providing a catalog for courses and programs.

I'd like to have that plugin available, ideally, before we deprecate Discovery or before we remove it minimally.

If you have use cases for a catalog plugin, please reach out to me.

[e0d]

Thanks a lot for putting this together @bradenmacdonald! I'm looking forward to not having to run a separate discovery service.

The great thing about this DEPR announcement is that it doesn't really require us to build anything new -- that is, until someone steps forward and tells us that Discovery is a required feature for them. Let's use this announcement to gauge community usage. We can then decide whether we want to backport some features in the core, and how.

Wouldn't you want to name a specific release as a deadline? Otherwise I'm afraid there won't be a sense of urgency in the community. I propose to target Teak.

I have heard from a number of folks that this is a need. I also believe a number of organizations have done some work to develop such a thing. For example,

https://github.com/academic-innovation/platform-plugin-course-series/tree/basic-series-support

[deborahgu]

@OmarIthawi

I support this removal. Discovery, credentials, edx-enterprise and licensing-manager has been all 2U-specific.

Credentials is definitely not 2U-specific. However, I'm not sure about the integration with discovery that in credentials. I think it's primarily used for copy-catalog, and I assume that non-2U credentials hosts just...don't run copy-catalog? @GlugovGrGlib, do you know?

[deborahgu]

for what it's worth, even though 2U is going to continue to use course-discovery for the immediate future, I would passionately like to be involved in a rethinking of Programs. I have absolutely no meaningful insight into the business constraints on 2U, but from a developer perspective, as a maintainer of the programs dashboard and the program certificate process, I would shave years off my eventual ulcer if there were an architectural reassessment.

@felipemontoya said

new catalog app: should this proposal take off, I believe we should put the core primitives like programs or catalog and write the most basic implementation possible. Then leave every bit of product development to plugins.

Agreed. Rather than spinning off different pathways, and programs with explicit enrollment, and programs without explicit enrollment, and all of the other myriad "program" alternatives, I'd like to see some core APIs for communication with some abstract, high level classes behind them, and rely on plugins for the logic. (eg. credentials IDA shouldn't have to have its own logic to find out if a learner is in a program, which relies on knowing the business logic of what a "program enrollment" means)

[bradenmacdonald]

I'd like to have that plugin available, ideally, before we deprecate Discovery or before we remove it minimally.

@e0d In the meantime then, what does it mean to accept this DEPR proposal? Or are you saying we should leave it pending until we have some replacements?

[felipemontoya]

I'm happy to see that many have agreed with my take on the models being super lightweight and very much extendable.
We at edunext are currently in the process of structuring two projects each with a different feature set, but both depending on the existance of catalogs/course_series/programs.

I spend some time reviewing some implementations of catalogs that we have in.

• course_discovery/apps/catalogs/models.py
• enterprise-catalog/blob/master/enterprise_catalog/apps/catalog/models.py
• academic-innovation/platform-plugin-course-series/blob/basic-series-support/course_series/models.py

and in the process trying to extract from the requirements we have a minimum set of functionality that would serve as a foundation for us.
Our needs would be solved by a plugin that does something like this:

FlexibleCatalogModel
    course_runs: as manyToMany to CourseOverview
    query_string: a string to match course_runs dinamically
    slug: url friendly
    name: for humans
    get_course_runs: the method that either does fetches the course_runs
                     directly or that uses the query string for the dynamic
                     asignment. Implemented through filters so that

    search: a method that does flexible search. This would be implemented via
            filters so that the default implementation can be swapped. I would
            also vouch for shipping with a mysql only implementation and one with
            meilisearch

    model specific openedx-filters: before meaningful changes are made
    model specific openedx-events: after the catalog changes in a meaningful way


API REST:
    CRUD on the FlexibleCatalog
        the API must allow to set the course_runs or the query
        the API to list the catalogs should use get_course_runs so that filters are respected
    Search API that uses the search method so that filters are respected

Python API
    get_catalogs: a public api method that includes filters to do access control
    get_course_runs: for a catalog without exposing the model (for stability for those that can manage to only use this)
    exposing the search api
    CRUD on the Catalog

And with all this I would expect the usual quality level:
docs, serializers, forms, admin pages, QA suite... all the goodies.

I would be trilled to collaborate with other teams on this.

---

Edit: copied to https://openedx.atlassian.net/wiki/spaces/OEPM/pages/4610129922/Lightweight+catalog+implementation+via+extensions for feedback in the models or other technical specs.

[bradenmacdonald]

@felipemontoya Could you move that to the wiki or to a new GitHub issue? I don't want to get this thread off topic, but I'd love for you to receive feedback.