async renderer support

[taylorchu]

It is helpful if one needs to use http to retrieve rich data for rendering.

[notslang]

Is there any case where marked actually needs to perform an async operation during compilation?

[taylorchu]

what do you mean by "compilation"?

[notslang]

The operation of turning markdown into HTML.

[taylorchu]

if rendering requires to call a function that is not synchronous or this function will run for a while.

this will be similar to how code highlighting works. making all rendering functions async will be more consistent.

[jacargentina]

My use case is this: i want to override paragraph to make a call to mongo, fetch a document by id (async) and build the output with that.

[TiboQc]

+1 for this, I need some enhanced custom markdown that will fetch data from the server to get the info to render the final HTML.

e.g.
The head of IT, [function:HEAD_OF_IT], will be in charge of...
Should render:
The head of IT, John Smith, will be in charge of...
This needs to get the person from the server. That will always be accurate when displaying the page, even when the person that's assigned as Head of IT changes.

[pod4g]

+1 for this

I define a special markdown syntax for our editor

the syntas like this:

\`\`\`sku
 1120
\`\`\`

the number is a id， I should fetch data by ajax (http://x.x.x.x/?id=1120)，and then i will convert the data to html , but marked not support async

What should I do?

I think this solution is pretty good #798

[janosh]

Async renderers would be very helpful! My use case is turning a markdown image responsive by fetching multiple srcsets as well as the image's width and height (to avoid layout shifts) from a CMS based on its href. Something like this:

import marked from 'marked'

const renderer = {
  async image(href, title, text) {
    if (href?.includes(`mycms.tld`) && !href.endsWith(`.svg`)) {
      const id = href.split(`myCmsId`)[1]
      const query = `{
        asset(id: "${id}") {
          width
          height
        }
      }`
      const { asset } = await fetchFromCms(query)
      const { width, height } = asset
      title = title ? `title="${title}"` : ``
      return `
      <picture>
        <source media="(min-width: 1000px)" type="image/webp" srcset="${href}?w=1000"&fm=webp"" />
        <source media="(min-width: 800px)" type="image/webp" srcset="${href}?w=800"&fm=webp"" />
        <source media="(min-width: 600px)" type="image/webp" srcset="${href}?w=600"&fm=webp"" />
        <source media="(min-width: 400px)" type="image/webp" srcset="${href}?w=400"&fm=webp"" />
        <img src="${href}?w=1000" alt="${text}" ${title} loading="lazy" width="${width}" height="${height}" style="width: 100%; height: auto;" />
      </picture>`
    }

    return false // delegate to default marked image renderer
  },
}

marked.use({ renderer })

[UziTech]

This is on the v2 project board but does not have a PR yet. If someone would like to start a PR this would go faster.

[janosh]

@UziTech I'm happy to try it if pointed in the right direction.

[UziTech]

basically we just need to change /src/Parser.js to call the renderers asynchronously without slowing down the parser.

[janosh]

So it's just adding await to every call to this.renderer.[whatever] in Parser.js and then await Parser.parse wherever that gets called?

[UziTech]

Just awaiting every render call will probably slow down the parser quite a bit.

[janosh]

I see. But what’s the alternative?

[UziTech]

Not sure. That is what needs to be figured out to get async renderers working.

[janosh]

Would it make sense to have both a sync and an async renderer (the latter being undefined by default) and only using the async one if the user defined it?

import marked from 'marked'

const renderer = {
  async imageAsync(href, title, text) {
    // ...
  },
}

marked.use({ renderer })

Similar to how fs has readFile and readFileSync.

[UziTech]

That could work. Or maybe have an async option and use a different parser that awaits the renderer calls. That way the default renderer doesn't change speed, and have a note in the docs about async being slightly slower.

I don't think it should be a problem with async being slightly slower since any async action (api call, file system call, etc.) will be a much bigger bottleneck for speed than awaiting a synchronous function. I just don't want to slow down the users that don't need it to be async.

[janosh]

You mean like this marked(mdStr, { async: true })? And then something like

  try {
    // ...
    if (opt.async) return await Parser.parse(tokens, opt);
    else return Parser.parse(tokens, opt);
  } catch (e) { ... }

[UziTech]

Ya it might work better to create a separate parser so we don't have the conditional logic on each renderer call.

 try {
    // ...
    if (opt.async) return await AsyncParser.parse(tokens, opt);
    else return Parser.parse(tokens, opt);
  } catch (e) { ... }

[OzzyCzech]

Is there any case where marked actually needs to perform an async operation during compilation?

It will be very useful have async support in renderer (speaking as an author of static site generator that uses Marked)

There is a plenty of examples:

• use oembed for generating smart links (with titles, description)
• check existence of URL before made active link (<del><a href="not existing url">someething</a></del>)
• resize images - that's a little bit crazy :)
• syntax highlighters - most of them are async
• API calls during rendering content - eg. IMDB get star rating for movies

[acarl005]

I'm also building a static site that uses Marked. My use case is that I'd like to execute code in the code blocks and render the output in my HTML page. I need async capabilities in my renderer to call a child process or remote host.

[trustedtomato]

I have made an async renderer extension for markdownit in my project called decharge, I'll extract it when I have some free time:) Probably a similar method can be used for marked. Until I extract it, feel free to take some ideas, it's located at examples/project-website/src/utils/async-render-markdown.ts

…

On Thu, Mar 10, 2022, 8:59 PM Andy ***@***.***> wrote: I'm also building a static site that uses Marked. My use case is that I'd like to execute code in the code blocks and render the output in my HTML page. I need async capabilities in my renderer to call a child process or remote host. — Reply to this email directly, view it on GitHub <#458 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEMIERCJOLTATBHJHZR3OWTU7JIA7ANCNFSM4ASHCU6Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[UziTech]

I started implementing a solution for this in #2405

[jimmywarting]

I would like to lazy import syntax highlighting library only when needed.
any possibility to turn the callback into promises instead?

looking at this example:

marked.setOptions({
  highlight: function(code, lang, callback) {
    require('pygmentize-bundled') ({ lang: lang, format: 'html' }, code, function (err, result) {
      callback(err, result.toString());
    });
  }
});

marked.parse(markdownString, (err, html) => {
  console.log(html);
});

would be nice to do:

marked.setOptions({
  async highlight (code, lang) {
    const xyz = await import('...')
    return xyz(code, lang)
  }
})

await marked.parse(markdownString)

[jimmywarting]

Just awaiting every render call will probably slow down the parser quite a bit.

it's also possible to look at the function if it's a async function:

async function f() {}
f.constructor.name === 'AsyncFunction' // true

so based on that you could do some if/else logic... but i guess this would be bad for sync functions that return promises or have some thenable function

found this if somebody is intrested:
https://stackoverflow.com/questions/55262996/does-awaiting-a-non-promise-have-any-detectable-effect

[jimmywarting]

as part of making it async, i would like to wish for something like a async stream to output data to the user as data flows in, i would like to do this within service worker:

self.onfetch = evt => {  
  evt.respondWith(async () => {
    const res = await fetch('markdown.md')
    const asyncIterable1 = res.body.pipeThrough(new TextDecoderStream())
    const asyncIterable2 = marked.parse(asyncIterable1, {
      async highlight(code, lang) {
        return await xyz()
      }
    })
    const rs = ReadableStream.from(asyncIterable2)
    return new Response(rs, { headers: { 'content-type': 'text/html' } })
  })
}

I ask that you do support asyncIterable (Symbol.asyncIterator) as an input parameter so it can work interchangeable between both whatwg and node streams so that it isn't tied to any core features.
(all doe node v17.5 will have whatwg stream exposed globally so that would be suggested over node streams if you where to support it)

[jimmywarting]

instead of having a async/sync method you could instead do things with symbol.asyncIterator and symbol.iterator

// Uses symbol.asyncIterator and promises
for await (const chunk of marked.parse(readable)) { ... }
// Uses symbol.iterator (chunk may return promise if a plugin only supports asyncIterator)
for (const chunk of marked.parse(readable)) { ... }

It could maybe also be possible to have something like:

const iterable = marked.parse(something, {
  highlight * (code, lang) {
    yield someSyncTransformation(code, lang)
  }
})
for (const str of iterable) { ... }

// or 
const iterable = marked.parse(something, {
  async highlight * (code, lang) {
    const transformer = await import(lib)
    yield transformer(code, lang)
  }
})
for (const chunk of iterable) {
  if (typeof chunk === 'string') console.log(chunk)
  else console.log(await chunk)
} 

it would be still possible for marked to still only be sync but it would be up to the developer to either use for or for await depending on if something where to return a promise that resolves to a string

[UziTech]

The biggest problem with making marked asynchronous in any of these ways is that most users won't use it but it will still slow it down for them. Even #2405 is about 5% slower because it needs to load more files. And not everything is async yet.

The only way that I can think to do this with keeping markeds speed for synchronous users is to have a separate entry point. Something like import {marked} from "marked/async".

The biggest problem with that approach is that most of the code for marked will have to be duplicated and at that point it would almost be easier to just create a separate package that imports marked just for the parts that aren't duplicated (probably just the rules).

[jimmywarting]

I don't think we necessary have to make the marked library async, if we could experimenting with creating a sync iterable
that yields either a string or a promise, then it's up to the developer to have to wait for the promise to complete - dos the marked library don't need to really support any async stuff and don't necessery have to pay for the performences either...

if we have markdown like

# title
description

```js
some example code (that need async transformation)
``

# footer title

then it would have to yield [string, promise<string>, string]
so that the developer would have to do:

for (const chunk of parser) {
  await chunk
}

// another form of possible solution could be:
const chunks = [...parse(markdown)] // [string, promise<string>, string]
const result = await Promise.all(chunks)
const html = result.join('')

[UziTech]

@jimmywarting if you want to create a PR with your proposal I would be interested to see how that will work. I don't know if that would solve the issue of not having duplicate code since most users would still just want to do const html = parse(markdown).

[jimmywarting]

hmm, haven't touched the underlying codebase or investigated the underlying code of this package or anything - just coming up with proposals 😄
But I can give it a test. see if it is any well written / document
likely going to change the function of markdown to return a value instead of using a callback... but will see what i can cook up. maybe it's even still possible to keep the callback approche

[github-actions]

🎉 This issue has been resolved in version 4.1.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀