Proposal: [Catalog Domain] Course About page, Index Page, and Course Catalog MFE conversion

[GlugovGrGlib]

Abstract

This proposal aims to transition the Course About, Catalog, and Index pages of the Open edX platform from legacy architecture to MFE. Historically, catalog and about pages are provided supporting functionality to Open edX learning, which is clearly described in DDD bounded contexts, therefore no major works or new features were introduced to these pages for a long time. As other parts of the system are converted to MFE any further maintenance and customizations are growing in cost.
The goal is to address maintenance challenges, improve user experience, and increase the platform's competitiveness. By adopting MFE, we will enable easier integration of new features, improve performance, and increase retention rate.

Detailed Product Proposal

No response

Context & Background (in brief, if a Product Proposal is linked above)

Overview of Current Challenges with Legacy Architecture for Course About and Catalog pages.

Maintenance Complexity: Maintaining both legacy and MFE pages whithin the OeX is time-consuming and costly. As other interfaces has already transitioned to MFEs, this pages require legacy approach for customizations which should be maintained and supported separately from other solutions in OeX system. Clients often seek systems with lower maintenance costs.

Customization Difficulty: Course catalog and about pages are challenging to customize in the current system due to limitations of legacy code, and tech debt around those functionalities. Course creators cannot modify search filters/facets without additional development to these pages, leading clients that don’t want to invest into 3rd-party systems like Wordpress to opt for other LMSs that allow such customizations for the catalog.

Integration Limitations: The legacy architecture lacks necessary attributes for smooth service integration. New features like a video player or taxonomy are challenging to incorporate into legacy pages, and it requires maintaining separate solutions to integrate other core features into these pages.

Pluginization Issues: The legacy architecture prevents the creation of new frontend and backend plugin types for these pages, that are more distinct from comprehensive theming, and can be reused in the Open edX community.

Adaptability on Mobile App: Course About page on mobile apps is not adaptive due to hard usage of custom HTML and specific layout of the page. It is difficult for students using mobile app to find information they need about the course.

The customizations to these pages aren't straightforward, but not that uncommon. For example, Raccoon Gang has done many projects to customize these pages, including rewriting the Course Catalog and Course About pages to make them unique and meet specific business needs.

Scope & Approach (in brief, if a Product Proposal is linked above)

Beneficiaries of this enhancement include:

Educational Institutions: With a more advanced API layer to support an easier integration with other systems, reduced maintenance burdens, and improved overall platform performance for catalog pages, educational institutions can meet more effectively their students' needs.

Students: Improved UX and performance will enable students to quickly find the information they need about courses, enhance course search functionality, and facilitate easier discovery of courses they are interested in.

Open edX Community: Moving from legacy architecture to MFE is likely to attract more community contributions, leading to the development of new plugins for the Course About and Course Catalog pages. This will also unblock new developments in Catalog domain, enabling the development of new features at a lower cost.

Open edX partners and Operators: Decreased maintenance costs may improve user retention rates, making Open edX a more attractive option compared to other LMS systems. As the customization complexity is decreased, Open edX vendors and operators will be able to easily adjust Catalog and About pages to unique business needs for the clients. This will benefit Open edX partners by potentially increasing their client base.

The high-level scope includes

• A new MFE application is created to host Catalog domain related user interfaces.
• Refactoring API Layer in edx-platform to integrate with new MFE interfaces for Course Catalog, Index, and Course About pages.
• Enhancements for the integration capabilities of Open edX
• Addition and integration of new features into new MFE pages in Catalog domain

Approach

Based on Raccoon Gang experience and previous work around conversion of LMS and Studio pages to MFE, the standard approach can be taken, using existing API, components and solutions whenever possible to ease the initial migration.
To save on cost for initial implementation, the previous solutions developed by Raccoon Gang can be taken as the basis. However, some features for enterprise or ecommerce may be considered obsolete and should be integrated into the core solution using plugin functionality.

Value & Impact (in brief, if a Product Proposal is linked above)

We believe the transfer of Course Catalog and Course About pages to MFE can have the following effect:

Increase Adoption Rate for the Open edX System:

Maintenance Cost Reduction: Moving to MFE will decrease maintenance costs, influencing user decisions to select Open edX over other systems.

Pluginization: Users can develop, share, and add plugins to the Course About and Catalog pages, meeting their needs without significant investments in customizations or integration with third-party systems.

Improved Integration: Enhanced APIs layer will facilitate easier integration with other systems, such as external CRM and CMS.

Consistent Architecture: A unified MFE architecture will streamline maintenance and customization efforts in the community.

Enhance User Experience:

Mobile Responsiveness: The Course About page will be more mobile-friendly, providing a better experience for students accessing the platform from the mobile app.

Improved Search Functionality: Enhanced search features will support students in quickly finding the courses they need. Integration of new search features developed for learning and course authoring MFEs might be much easier.

Support Development of Mobile Initiatives:

Adaptive Design: The division of fields in the Course About page enables the creation of a mobile-responsive design that improves student understanding and engagement.

Performance Improvements:

Performance: The load on the Course Catalog and Course About pages will decrease, enhancing overall platform performance. MFE solution will allow us to switch between different pages without full-page reload, which significantly enhances UX for end-users.

Advanced Caching: Use of the cache techniques for API responses on the frontend for the anonymous users.

Milestones and/or Epics

Milestone 1: Legacy to MFE & Pluginization Enablement

• A new MFE application is created to host Catalog domain related user interfaces.
• Refactoring API Layer in edx-platform to integrate with new MFE interfaces for Course Catalog, Index, and Course About pages.
• Index page reimplementation as MFE.
• Redesign Course Catalog page using Paragon Design System, and achieve feature parity with the legacy Course Catalog page in MFE.
• Reimplement Course About page in MFE, by preserving the core functionality.
• Enable frontend pluginization on Course About, Course Catalog, and Index pages, to meet user needs without extensive customizations.
• Allow Operators to opt out of these pages in favor of third-party CMS integration

Milestone 2: Functional Enhancements

• Improve API functionality for native catalog and 3rd party integration.
• Enhance search capabilities, by utilizing new search, taxonomies and course attributes features.
• Redesign Course About page, by dividing Course Overview HTML into several fields for better indexing and mobile adaptability. More information in the draft proposal - Improved enrolment views and mobile discovery

Futher development

• WYSIWYG for course about page

Named Release

Teak

Timeline

Milestone 1

Autumn 2024 — Winter 2024

Milestone 2 and further customizations

Winter 2025 — Spring 2025 by Teak release.

Proposed By

Raccoon Gang

Additional Info

No response

[github-actions]

Thanks for your submission, @openedx/openedx-product-managers will review shortly.

[sarina]

Hi @GlugovGrGlib - I volunteered to coordinate this as I'm interested in it. However I'm not sure who the biggest stakeholders are, as I believe many sites use their own catalog functionality. Any thoughts?

[jmakowski1123]

Just noting that this proposal carries contingencies for mobile.

[marcotuts]

This work is a precursor to improved course / enrollment / discovery views for Mobile as well as improved authoring for enrollment pages out of the box for Open edX (web + mobile). Additional details are here: #369

I think a few providers who have spent cycles improving / modifying course discovery views would be able to provide input here on the value of MFE conversion. I can reach out to a few and ask for input.

[felipemontoya]

I agree that bringing the "external" pages up to standard is necessary next step. Also I am all pro the pluginization and the inclusion of extensibility cases from the ground up. I believe we should have a very simple implementation of the course search/course about pages and leave every single extra feature to be build in plugins.

I don't think that we should strive for feature parity here as much as in other MFE conversions. These APIs and UIs have received no upgrades in a long time and we could take this opportunity to turn them in a better foundation for the extensibility case. Such as the replacement for the catalog service (course discovery) outlinined in course-discovery/issues/4449.

[GlugovGrGlib]

@sarina That's great! I also believe providers have significant exposure to various catalog customization requests and could provide valuable input.
Based on our experience and initial research, most Open edX installations either rely on the default catalog functionality or have made attempts to customize it—especially within educational institutions. Only the largest, commercially focused installations tend to use external CMS/e-commerce integrations for their catalogs.

@felipemontoya I completely agree, that's the right approach.

[OmarIthawi]

Thanks for the proposal.

I've been out of the loop for a while when it comes to DEPR and MFEs.

I see that SEO isn't mentioned here, which unlike other MFEs is important for the Course Catalog.

If we'd like to keep the platform lightweight and works out of the box, I think our standard Micro-frontend stack isn't useful and we need to think about the SEO and marketing experience in this specific MFE.

[sarina]

Apologies for the late review here. I agree with @felipemontoya

I believe we should have a very simple implementation of the course search/course about pages and leave every single extra feature to be build in plugins.

and would propose a milestone 0 that is the design of the new APIs and the MFE pages (with plugin slots specified, and mocks). That would end up being an extension of this product proposal for the community to review.

@OmarIthawi I'm not sure how to address SEO concerns. A part of me feels that an instance focused on SEO should potentially use a 3rd party CMS to focus on SEO needs, and by making clear and easy to use APIs we make it easier to leverage a 3rd party CMS.

[sarina]

@marcotuts @felipemontoya would you be able to review this plan? I think if accepted, we move on to a milestone 0 which itself will have another review/comment period of the technical specs and visual mocks.

[OmarIthawi]

@OmarIthawi I'm not sure how to address SEO concerns. A part of me feels that an instance focused on SEO should potentially use a 3rd party CMS to focus on SEO needs, and by making clear and easy to use APIs we make it easier to leverage a 3rd party CMS.

My concern is that we're making Open edX more costly to adopt for small instances with limited budgets by removing basic public pages from the core of Open edX into an MFE.

@sarina Last time I checked, using pure React.js app isn't good for the SEO and harm crawling. So to put the home page as a pure JavaScript/React.js app without pre-rendering can be hamful to the most basic SEO and hurts Open edX adoption.

If we're going the MFE route and making edx-platform as a backend-only, I'd recommend investing in a Next.js app which is a bit more friendly for SEO as well as the added benefit of caching and providing pre-rendered pages for non-authenticated users. It doesn't have to be Next.js, WordPress/Drupal can be a good replacement as well.

I know this is a change from our MFE stack, so perhaps we should make this specific MFE replaceable as much as possible similar to how wer'e investing in making ecommerce replaceable via WooCommerce plugin.

[sarina]

@GlugovGrGlib could you please weigh in on Omar's above comment?

[pdpinch]

Google's crawler has been parsing JavaScript for some time so using react.js, by itself, isn't a blocker for Google SEO.

But, you may want to consider social media. In my experience, Twitter, Slack, etc. don't go to the same effort to process JavaScript when you share URLs. It may be worth some effort to be sure that distinct about page URLs present distinct metadata and text when they are shared.

[OmarIthawi]

Google's crawler has been parsing JavaScript for some time so using react.js, by itself, isn't a blocker for Google SEO.

Yes, thanks @pdpinch for pointing that out.

But, you may want to consider social media. In my experience, Twitter, Slack, etc. don't go to the same effort to process JavaScript when you share URLs. It may be worth some effort to be sure that distinct about page URLs present distinct metadata and text when they are shared.

Yup, that's the issue. I think we probably need to invest in a server-side rendered solution, unless we expect every installation to re-implement their own solution.

Popular solutions are: Next.js, WordPress, Drupal, Django app (server side rendering), and static page generators like GitHub pages Jeykll. Anything but a pure MFE in my opinion would work.