[VEDA2 Refactor] Spike - Replace Parcel Resolver?

[hanbyul-here]

Parcel Resolver has been used to serve the contents of the VEDA from MDX files. The resolver made the metadata of the contents (frontmatter) instantly available through the faux module veda while the contents of each file lazily available through its loader. (So the built assets don't have to hold a large amount of content.)

While the resolver has served its purpose so far, it is explicitly written for Parcel, and we hope VEDA-UI can be agnostic from the bundler.

Documentation about Parcel Resolver & Faux Modules: https://github.com/NASA-IMPACT/veda-ui/blob/main/docs/development/ARCHITECTURE.md#veda-modules

Here are some pointers that helped me get more thoughts on what is going on bts:

Where the mdx data objects are generated ({data : ${FrontMatterData}, content:${FileLoader}}): https://github.com/NASA-IMPACT/veda-ui/blob/main/parcel-resolver-veda/index.js#L66-L81

Where frontmatter gets processed (ex. we have some prefixes that need special handling for frontmatter, such as ::markdown, ::js) : https://github.com/NASA-IMPACT/veda-ui/blob/main/parcel-resolver-veda/stringify-yml-func.js

This ticket is a spike for researching how we can access this problem. The answers that we need to get from this spikes are

• How can we load the contents data with the benefits that Parcel Resolver gave us (synchrouns loading of metadata, lazy loading of contents, preprocess of yaml data) without it? If we need to trade some of the benefits, what are they?
• What kind of API should we expose to users so that they can bring their own content and use our data parsers to feed the data to VEDA UI components?
• What content data format should we ask for users to bring? only MDX? or we can be more flexible?

[danielfdsilva]

Besides making the different contents available one big task of the veda resolver is to create list of values.

It creates a general list of datasets so then individual ones can be loaded and also of all the existent taxonomies for the filters by looking through all the files. Doing this during build allows you not to have a hardcoded list elsewhere and not needed to load all content during runtime to create this.

These are things that are very basic when you have a database for example, or even when you use a static site generator like gatsby, or something like Next. It becomes much more complicated when you're building an SPA that needs arbitrary data from an arbitrary number of files without using server tech.

[oliverroick]

I feel like I’m missing a lot of detail for how VEDA works internally and how the content is managed so I’m just going to throw out some questions, hoping that it clarifies some questions for me without stirring the pot too much.

• Is MDX even the best way to manage the content? Are there alternatives that allow users to manage the content in a more convenient way, like a database-driven CMS?
• What the reason for not using a database-driven solution if it’s easier to generate the content structure we require?
• Can we use a static-site generator like Gatsby or Netlify, and how would that change the way we approach generating a list of content and values for filtering?
• Do we have to replace the parcel builder? If so, why?

[hanbyul-here]

@oliverroick These are all good questions and should be part of Spike. To lay out the contexts that I am aware of...

• is MDX the best way?
  MDX incorporates Custom components relatively simply into markdown. I do not know if MDX is the best way to do this, but I could not find other options for exposing React-based components to non-technical users. (I am trying to think if this is the only reason that we have to stick with MDX.. MDX does offer other good things such as frontmatter as metadata and markdown formats, but I feel like these traits can be found in other systems.)

• Why not using db ?
  Assuming that we do not want to build a CMS from scratch, we looked around to see if there is any good third-party CMS for VEDA. We could not find a good solution for incorporating the custom components/ async loading the contents.

  • Adjacently related: There have been requests for GUI for writing experience. Erik experimented with some ideas about it (not involving any back-end or db) detailed in this issue: Plan/Prototype Scrollaborate (working title) #546. So, if we are willing to build CMS with db, we might be able to get buy-in. (Something like EO-CMS? but we would need more people to build and maintain this system.)

• Using other static site generators?
  I am not very experienced at React-based static site generators, but to overly simplify, it is basically getting' a list of posts' for SSG, and I don't see why it won't be possible. (Please correct me if I am wrong or missing anything.)

• Replacing Parcel builder?
  Parcel as a bundler, you mean? (Then I don't see why we have to replace it.) or resolver? If you mean later, this is part of the answer I am trying to find at the end of this Spike. Imo, it will be likely that we would need to replace it. If we don't replace Parcel resolver, we would at least need to move it to the configuration level (because configuration will feed the data to UI), then users would be locked into using Parcel as an application builder.

I hope this gives you more ground to delve into the issue @oliverroick

[sandrahoang686]

Slack Thread on needing to make a decision wether we want to use a static-page generator like NextJS or stick with what we’re currently using...
https://developmentseed.slack.com/archives/C04HM9LQ9UM/p1712011779163419

[hanbyul-here]

Working on the Next test instance, #947 gave me some insights about what is needed to replace Parcel Resovler (To be clear, I am not talking about Parcel as a bundler. I am talking about Faux module 'Veda' that Parcel resolver outputs. These are being used as import { datasets } from 'veda' in the current code base.

1. Parse the front matter and contents of MDX files: This is being tested as a part of [VEDA2 Refactor] Build NextJs Instance Prototype #947 . Thanks to the communities that evolved around MDX; I expect this can be done relatively straightforwardly, Ex. The current progress on parsing the MDX data on Next test instance: https://github.com/developmentseed/next-veda-ui/blob/main/app/blog/utils.ts#L145

2. Expose types that VEDA depends on: This is also a matter of moving the types to somewhere in the source code.

3. Expose the datasets/stories that can be globally accessible throughout the applications: This is the element that I am still unclear how to tackle. We moved some components towards the direction that the component will get this data as a prop. (ex. data catalog https://github.com/NASA-IMPACT/veda-ui/blob/main/app/scripts/components/data-catalog/container.tsx#L23) However, this approach is not possible with MDX components. Ex. BlockMap component expects a user to pass dataset-id and layer-id, then gets all other information needed through globally accessible datasets: (https://github.com/NASA-IMPACT/veda-ui/blob/main/app/scripts/components/common/blocks/block-map.tsx#L146. - it is layered but it eventually depends on global datasets, https://github.com/NASA-IMPACT/veda-ui/blob/main/app/scripts/components/common/blocks/block-map.tsx#L186). Some of the ideas (these options are not mutually exclusive to each other.)

• Keep the pattern. Ask users to pass the list of datasets like other components - Then how can a component get the list of datasets in MDX file?
• There is almost always a central point to customize the rendering of MDX file, such as https://github.com/NASA-IMPACT/veda-ui/blob/package-node-v/app/scripts/components/common/mdx-content.js for VEDA and https://github.com/developmentseed/next-veda-ui/blob/main/app/components/mdx.tsx for Next Instance. Can we expose the list of datasets through this central point?
• Use Context. Ask users to fill the provider with the list of datasets and wrap the MDX component central point with this provider. Since the context is for client components, does this mean that the page won't be able to be generated anymore (no benefit for SEO)?

I'd like to spend more time on 3, but I wanted to lay out the problem in a more scoped way first.

[hanbyul-here]

To replace what Parcel Resolver does for 3 (Expose the datasets/stories that can be globally accessible throughout the applications), we narrowed down the approaches worth trying. (Thanks @dzole0311 for the insights 🙇 )

Approaches

1. Use Context. Expose the ProviProviderask Users to fill up this proviProvider wrap the MDX Renderer with this Provider. We should examine how SEO will be affected from this approach.
   Some resources

   • When Server components are nested with Client components: https://nextjs.org/docs/app/building-your-application/rendering/composition-patterns#supported-pattern-passing-server-components-to-client-components-as-props
   • Server only context https://www.npmjs.com/package/server-only-context

2. Get the datasets on MDX Renderer level and pass them as props. In this approach, rewriting the component so the dataset resolutions can happen on a higher container level will help avoid deep prop drilling.

---

There was an additional option to expose datasets and stories through an MDX file through using the syntax below.

import datasets from 'somewhere-that-instant-can-get-datasets-list';
<BlockMap dataset-id={...} ... datasets={datasets} />

But this will complicate the editor experience, so this approach won't be worth pursuing.

[hanbyul-here]

As we narrowed down what to do to replace Parcel Resolver, I will close this ticket and open the items with action items. (such as #975)