importing prism-php package breaks other packages #2769

michtio · 2021-02-17T21:21:43Z

Information:

Prism version: 1.23.0
Plugins: line-numbers
Environment: webpack

Description

Currently importing scripts in webpack ( due to the failure of the babel plugin ) as follows:

// settings
module.exports = {
    alias: {
        'vue$': 'vue/dist/vue.esm.js'
    },
    copyright: '©2020 Percipio.London',
    entry: {
        'app': [
            'prismjs',
            'prismjs/components/prism-bash', 
            'prismjs/components/prism-css', 
            'prismjs/components/prism-docker', 
            'prismjs/components/prism-git', 
            'prismjs/components/prism-graphql', 
            'prismjs/components/prism-javascript', 
            'prismjs/components/prism-json', 
            'prismjs/components/prism-markdown', 
            'prismjs/components/prism-markup', 
            'prismjs/components/prism-sql', 
            'prismjs/components/prism-twig', 
            'prismjs/components/prism-typescript', 
            'prismjs/components/prism-yaml',
            //'prismjs/components/prism-php', 
            'prismjs/plugins/line-numbers/prism-line-numbers',
            '../src/fonts/open-sans-regular.woff',
            '../src/fonts/open-sans-regular.woff2',
            '../src/fonts/open-sans-italic.woff',
            '../src/fonts/open-sans-italic.woff2',
            '../src/fonts/open-sans-600.woff',
            '../src/fonts/open-sans-600.woff2',
            '../src/fonts/open-sans-700.woff',
            '../src/fonts/open-sans-700.woff2',
            '../src/fonts/open-sans-800.woff',
            '../src/fonts/open-sans-800.woff2',
            '../src/fonts/source-code-pro-regular.woff',
            '../src/fonts/source-code-pro-regular.woff2',
            '../src/js/app.ts',
            '../src/js/assets/icons.js',
            '../src/css/tailwind-base.pcss',
            '../src/css/tailwind-components.pcss',
            '../src/css/tailwind-utilities.pcss',
            '../src/css/app-components.pcss',
            '../src/css/app-utilities.pcss',
            '../src/css/vendor.pcss',
        ],
        'lazysizes-wrapper': '../src/js/utils/lazysizes-wrapper.ts',
    },
    extensions: ['.ts', '.js', '.vue', '.json'],
    name: 'percipio.london',
    paths: {
        dist: '../../cms/web/dist/',
    },
    urls: {
        criticalCss: 'https://sandbox.percipio.london/',
        publicPath: () => process.env.PUBLIC_PATH || '/dist/',
    },
};

You see the imports, and prism-php being commented out.
Prism works fine ( loading with Prism.highlightAll() ) - inside of a Vue app:

mounted() {
            window.addEventListener('load', () => {
                Prism.highlightAll();
            })
        }

Code highlighting works as expected, line numbers show up.

Now the odd case: The moment prism-php is uncommented and brought into play, the code highlighting breaks completely. No elements added inside of the code block ( can't find any spans or classes with tokens or whatsoever )

If I use Twig/CSS/Javascript/Docker/Git/GraphQL ... they all work perfect, they all break once prism-php is added into play.

Without the php package

With the php package

I can't find any errors in the console or in the buildchain whatsoever which could explain what is going on.

The text was updated successfully, but these errors were encountered:

michtio · 2021-02-17T21:28:23Z

after only loading the php language file ( and using a PHP snippet with language-php ) I get the following error:

prism-php.js?8e9b:311 Uncaught TypeError: Cannot read property 'tokenizePlaceholders' of undefined
    at eval (prism-php.js?8e9b:311)
    at Object.run (prism.js?c197:705)
    at Object.highlight (prism.js?c197:614)
    at Object.highlightElement (prism.js?c197:582)
    at Object.highlightAllUnder (prism.js?c197:486)
    at Object.highlightAll (prism.js?c197:454)
    at eval (app.ts?53a0:57)
    ```
    
So definitely something going on here.

RunDevelopment · 2021-02-17T21:32:32Z

Prism languages have dependencies and the one loading Prism files is responsible for resolving them. This is a non-trivial task which is why we provide a few ways for doing so. Please use one of the officially supported methods to load languages. They are guaranteed to work and will continue to work in the future.

If I use Twig/CSS/Javascript/Docker/Git/GraphQL ... they all work perfect

That is because the default entry file of the prismjs package includes the Markup, CSS, JS, and C-like languages. All other languages you manually specified either only depend on those languages or happen to be in the correct load order.

The reason PHP fails is that it also depends on our Markup templating component. (Markup Templating exports the tokenizePlaceholders function you know from the error message.)

michtio · 2021-02-17T21:53:32Z

This is a quick close to be honest @RunDevelopment as you can see I am importing the markup templating` component as well, still this error persists, the entry points don't do more than simply importing the files.

If I put :

import Prism from 'prismjs';
import 'prismjs/components/prism-bash'; 
import 'prismjs/components/prism-css'; 
import 'prismjs/components/prism-docker'; 
import 'prismjs/components/prism-git'; 
import 'prismjs/components/prism-graphql'; 
import 'prismjs/components/prism-javascript'; 
import 'prismjs/components/prism-json'; 
import 'prismjs/components/prism-markdown'; 
import 'prismjs/components/prism-markup'; 
import 'prismjs/components/prism-sql'; 
import 'prismjs/components/prism-twig'; 
import 'prismjs/components/prism-typescript'; 
import 'prismjs/components/prism-yaml';
import 'prismjs/components/prism-php'; 
import 'prismjs/plugins/line-numbers/prism-line-numbers';

Inside my main app.ts the error persists.

michtio · 2021-02-17T22:10:17Z

Also using the babel config as written in the documentation:

module.exports = (type = 'modern', settings) => {
    // common config
    const common = (browserList) => ({
        module: {
            rules: [
                {
                    test: /\.js$/,
                    exclude: settings.exclude,
                    use: {
                        loader: 'babel-loader',
                        options: {
                            cacheDirectory: true,
                            presets: [
                                [
                                    '@babel/preset-env', {
                                    modules: type === 'legacy' ? 'auto' : false,
                                    corejs: {
                                        version: 3,
                                        proposals: true
                                    },
                                    debug: false,
                                    useBuiltIns: 'usage',
                                    targets: {
                                        browsers: browserList,
                                    },
                                }
                                ],
                                [
                                    '@babel/preset-typescript', {
                                    'allExtensions': true,
                                    'isTSX': false,
                                }
                                ],
                            ],
                            plugins: [
                                '@babel/plugin-syntax-dynamic-import',
                                '@babel/plugin-transform-runtime',
                                '@babel/plugin-proposal-class-properties',
                                '@babel/plugin-proposal-object-rest-spread',
                                '@babel/plugin-proposal-nullish-coalescing-operator',
                                '@babel/plugin-proposal-optional-chaining',
                                [
                                    'prismjs', {
                                        'languages': [
                                            'bash', 
                                            'css', 
                                            'docker', 
                                            'git', 
                                            'graphql',
                                            'javascript', 
                                            'json', 
                                            'markdown', 
                                            'markup',
                                            'php',
                                            'sql', 
                                            'twig', 
                                            'typescript', 
                                            'yaml'
                                        ],
                                        'plugins': ['line-numbers'],
                                        'theme': 'okaida',
                                        'css': false
                                    }
                                ]
                            ],
                            sourceType: 'unambiguous',
                        },
                    },
                },
            ],
        },
    });
    // configs
    const configs = {
        // development config
        development: {
            // legacy development config
            legacy: {
            },
            // modern development config
            modern: {
                ...common(pkg.browserslist.modernBrowsers),
            },
        },
        // production config
        production: {
            // legacy production config
            legacy: {
                ...common(pkg.browserslist.legacyBrowsers),
            },
            // modern production config
            modern: {
                ...common(pkg.browserslist.modernBrowsers),
            },
        }
    };

    return configs[process.env.NODE_ENV][type];
}

simply using:

import Prism from 'prismjs';
Prism.highlightAll();

still doesn't give me anything.

See screenshots: The plugin loads but seems to be broken / not triggering - code highlighting is not applied, since no components seem to be loaded:

michtio · 2021-02-17T22:13:19Z

That is because the default entry file of the prismjs package includes the Markup, CSS, JS, and C-like languages. All other languages you manually specified either only depend on those languages or happen to be in the correct load order.

On it's turn is what I meant, all packages work ... only the PHP one doesn't, and does break all the rest when it is being added.
Load all the components --> every component will get code highlighting, working as expected. Add the php component in the mix, none of the languages highlight anymore, remove the component again, and everything magically works.

This wasn't a remark based on "missing dependencies"

RunDevelopment · 2021-02-17T22:22:03Z

Yeah, we get this kind of issue like once every two months, so I'm quick to close...

This wasn't a remark based on "missing dependencies"

I am very sure it was. The error message you provided is a clear indication that Markup Templating isn't loaded.

If I put :

import Prism from 'prismjs';
import 'prismjs/components/prism-bash'; 
import 'prismjs/components/prism-css'; 
import 'prismjs/components/prism-docker'; 
import 'prismjs/components/prism-git'; 
import 'prismjs/components/prism-graphql'; 
import 'prismjs/components/prism-javascript'; 
import 'prismjs/components/prism-json'; 
import 'prismjs/components/prism-markdown'; 
import 'prismjs/components/prism-markup'; 
import 'prismjs/components/prism-sql'; 
import 'prismjs/components/prism-twig'; 
import 'prismjs/components/prism-typescript'; 
import 'prismjs/components/prism-yaml';
import 'prismjs/components/prism-php'; 
import 'prismjs/plugins/line-numbers/prism-line-numbers';

Inside my main app.ts the error persists.

This doesn't include prismjs/components/prism-markup-templating.

Also using the babel config as written in the documentation:

@mAAdhaTTah Could you please check this? It seems right to me.

Just to make sure: @michtio, the plugin is installed via npm, right? (It's the only thing I can't verify from the information you've given.)

RunDevelopment · 2021-02-17T22:28:28Z

@michtio Could you please show us the transformed output file that contains the import Prism from 'prismjs'? Just the imports are enough. The babel plugin is supposed to generate a bunch of import statements for you, so I want to verify that it does so correctly.

mAAdhaTTah · 2021-02-17T22:47:10Z

Off the top, if you're using the babel plugin, you shouldn't also import all of the dependencies manually. Use import Prism from 'prismjs' only + the babel plugin (which looks correctly configured) and maybe that'll fix it.

michtio · 2021-02-18T10:00:45Z

@mAAdhaTTah it was the one or the other, I tried it without importing all the rest, still no success whatsoever.

I will try @RunDevelopment his suggestion with importing prism-markup-templating without the babel setup and report back.

michtio · 2021-02-18T10:06:46Z

I can already confirm:

    entry: {
        'app': [
            'prismjs',
            'prismjs/components/prism-markup-templating', 
            'prismjs/components/prism-bash', 
            'prismjs/components/prism-css', 
            'prismjs/components/prism-docker', 
            'prismjs/components/prism-git', 
            'prismjs/components/prism-graphql', 
            'prismjs/components/prism-javascript', 
            'prismjs/components/prism-json', 
            'prismjs/components/prism-markdown', 
            'prismjs/components/prism-markup', 
            'prismjs/components/prism-sql', 
            'prismjs/components/prism-twig', 
            'prismjs/components/prism-typescript', 
            'prismjs/components/prism-yaml',
            'prismjs/components/prism-php', 
            'prismjs/plugins/line-numbers/prism-line-numbers',
            '../src/fonts/open-sans-regular.woff',
            '../src/fonts/open-sans-regular.woff2',
            '../src/fonts/open-sans-italic.woff',
            '../src/fonts/open-sans-italic.woff2',
            '../src/fonts/open-sans-600.woff',
            '../src/fonts/open-sans-600.woff2',
            '../src/fonts/open-sans-700.woff',
            '../src/fonts/open-sans-700.woff2',
            '../src/fonts/open-sans-800.woff',
            '../src/fonts/open-sans-800.woff2',
            '../src/fonts/source-code-pro-regular.woff',
            '../src/fonts/source-code-pro-regular.woff2',
            '../src/js/app.ts',
            '../src/js/assets/icons.js',
            '../src/css/tailwind-base.pcss',
            '../src/css/tailwind-components.pcss',
            '../src/css/tailwind-utilities.pcss',
            '../src/css/app-components.pcss',
            '../src/css/app-utilities.pcss',
            '../src/css/vendor.pcss',
        ],
        'lazysizes-wrapper': '../src/js/utils/lazysizes-wrapper.ts',
    },

All these entry points ( with prism-markup-templating added, do work for php now )

not using those entrypoints and relying on babel:

import { createStore } from './stores/store.js';

import Prism from 'prismjs';
Prism.highlightAll();

// App main
const site = async () => {
    // Async load the vue module
    const [ Vue ] = await Promise.all([
        import(/* webpackChunkName: "vue" */ 'vue'),
    ]).then(arr => arr.map(({ default: defaults }) => defaults));

    const store = await createStore(Vue);

    const vm = new Vue({

        el: '#page-container',
        store,
        components: {
            'navigation--main': () => import(/* webpackChunkName: "navigation--main" */ '../vue/organisms/navigations/navigation--main.vue'),
            'notification--cookie': () => import(/* webpackChunkName: "notification--cookie" */ '../vue/molecules/notifications/notification--cookie.vue'),
        },

        data: () => ({}),

        methods: {

            // Pre-render pages when the user mouses over a link
            // Usage: <a href="" @mouseover="prerenderLink">
            prerenderLink: function (e : Event) {
                const head = document.getElementsByTagName("head")[0];
                const refs = head.childNodes;
                const ref = refs[refs.length - 1];

                const elements = head.getElementsByTagName("link");
                Array.prototype.forEach.call(elements, function (el, i) {
                    if (("rel" in el) && (el.rel === "prerender")) {
                        el.parentNode.removeChild(el);
                    }
                });

                if (ref.parentNode && e.currentTarget) {
                    const target : HTMLAnchorElement = <HTMLAnchorElement>e.currentTarget;
                    const prerenderTag = document.createElement("link");
                    prerenderTag.rel = "prerender";
                    prerenderTag.href = target.href;
                    ref.parentNode.insertBefore(prerenderTag, ref);
                }
            },

            printPage() {
                window.print();
            }

        },

        mounted() {
            window.addEventListener('load', () => {
                //Prism.highlightAll();
            })
        }

    })
};

// Execute async function
site().then( (value) => {
});

// Accept HMR as per: https://webpack.js.org/api/hot-module-replacement#accept
if (module.hot) {
    module.hot.accept();
}

with the above babel config, doesn't highlight the code.

RunDevelopment · 2021-02-18T13:26:57Z

with the above babel config, doesn't highlight the code.

That's very strange. Could you please provide a project to reproduce this behavior?

jazanne · 2022-02-02T15:26:57Z

This happens to me too - i'm vendoring PrismJS 1.26. It's not an issue when i use 1.1.x of this plugin, but when I update to 1.2.x it breaks

Error

Error in function eval in ./.yarn/cache/prismjs-npm-1.26.0-15be1ab06b-6de058930c.zip/node_modules/prismjs/components/prism-php.js:339
Cannot read properties of undefined (reading 'tokenizePlaceholders')

./.yarn/cache/prismjs-npm-1.26.0-15be1ab06b-6de058930c.zip/node_modules/prismjs/components/prism-php.js:339
Open in Editor
  337 |
  338 | 	Prism.hooks.add('after-tokenize', function (env) {
> 339 | 		Prism.languages['markup-templating'].tokenizePlaceholders(env, 'php');
  340 | 	});
  341 |
  342 | }(Prism));

skullface · 2023-08-03T21:57:59Z

+1 Reviving this unexpected error encountered with Prism 1.29 via prism-react-renderer 2.0.6.

  useEffect(() => {
    typeof global !== 'undefined' ? (global.Prism = Prism) : (window.Prism = Prism)
    require('prismjs/components/prism-bash')
    require('prismjs/components/prism-php')
    require('prismjs/components/prism-pug')
    require('prismjs/components/prism-python')
    require('prismjs/components/prism-ruby')
    require('prismjs/components/prism-toml')
  }, [])

I am importing several other language packages via require successfully, but the app consistently breaks when prism-php is included in the above list.

Uncaught TypeError: Cannot read properties of undefined (reading 'tokenizePlaceholders')
    at eval (webpack-internal:///(app-pages-browser)/./node_modules/.pnpm/[email protected]/node_modules/prismjs/components/prism-php.js:339:40)
    at Object.run (webpack-internal:///(app-pages-browser)/./node_modules/.pnpm/[email protected][email protected]/node_modules/prism-react-renderer/dist/index.mjs:525:15)
…

This was solved by first importing require('prismjs/components/prism-markup-templating') via suggestion here: #1400 (comment). I’m lucky it stays in alphabetical order 😅

JamieSinn · 2023-11-06T20:04:50Z

I had the exact same issue as @skullface with Docusaurus 3 and their upgraded prism-react-renderer depend. It breaks until we remove the php language option.

mAAdhaTTah · 2023-11-06T20:21:03Z

The general issue here remains a combination of doing manual dependency management and including php without including markup-templating before it. If you're having issues with the babel plugin, please file a ticket on that repository with a reproduction and I can look into it: https://github.com/mAAdhaTTah/babel-plugin-prismjs

The general problem is experienced as Cannot read properties of undefined (reading 'tokenizePlaceholders'); depending on the browser, the message may be different, but the lack of tokenizePlaceholders or something similar is how the issue appears. The solution is to add an import or require of 'prismjs/components/prism-markup-templating' before you import php or configure babel-plugin-prismjs to manage the dependencies for you.

Looking over the issues in this thread, they all seem to be covered by the above solution, so I'm going to close this, as this isn't an issue in Prismjs. If you try the above and are still experiencing issues, please provide a reproduction that I can run and investigate.

* Docusaurus 3 upgrade * Cleanup packages * Cleanup * Fix imports/package * Fix prism dependency * PHP is apparently broken? * PHP comment - confirmed broken due to PrismJS/prism#2769 * Python highlighting

Not sure why we are not using the Docusaurus file, but have our own: https://github.com/facebook/docusaurus/blob/8dc24b6a98968f585af1bb5e6f8b7a1fcae0d7d4/packages/docusaurus-theme-classic/src/theme/prism-include-languages.ts#L31 PrismJS/prism#2769 was helpful. DevCycleHQ/devcycle-docs#467 was a good reference. Signed-off-by: Gerhard Lazu <[email protected]>

RunDevelopment closed this as completed Feb 17, 2021

RunDevelopment reopened this Feb 18, 2021

mrin9 mentioned this issue Mar 7, 2022

PHP support for x-codeSamples rapi-doc/RapiDoc#680

Closed

JamieSinn added a commit to DevCycleHQ/devcycle-docs that referenced this issue Nov 6, 2023

PHP comment - confirmed broken due to PrismJS/prism#2769

8f24da2

mAAdhaTTah closed this as completed Nov 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

importing prism-php package breaks other packages #2769

importing prism-php package breaks other packages #2769

michtio commented Feb 17, 2021

michtio commented Feb 17, 2021 •

edited

Loading

RunDevelopment commented Feb 17, 2021

michtio commented Feb 17, 2021

michtio commented Feb 17, 2021

michtio commented Feb 17, 2021

RunDevelopment commented Feb 17, 2021 •

edited

Loading

RunDevelopment commented Feb 17, 2021

mAAdhaTTah commented Feb 17, 2021

michtio commented Feb 18, 2021

michtio commented Feb 18, 2021 •

edited

Loading

RunDevelopment commented Feb 18, 2021

jazanne commented Feb 2, 2022 •

edited

Loading

skullface commented Aug 3, 2023 •

edited

Loading

JamieSinn commented Nov 6, 2023

mAAdhaTTah commented Nov 6, 2023

importing prism-php package breaks other packages #2769

importing prism-php package breaks other packages #2769

Comments

michtio commented Feb 17, 2021

Without the php package

With the php package

michtio commented Feb 17, 2021 • edited Loading

RunDevelopment commented Feb 17, 2021

michtio commented Feb 17, 2021

michtio commented Feb 17, 2021

michtio commented Feb 17, 2021

RunDevelopment commented Feb 17, 2021 • edited Loading

RunDevelopment commented Feb 17, 2021

mAAdhaTTah commented Feb 17, 2021

michtio commented Feb 18, 2021

michtio commented Feb 18, 2021 • edited Loading

RunDevelopment commented Feb 18, 2021

jazanne commented Feb 2, 2022 • edited Loading

skullface commented Aug 3, 2023 • edited Loading

JamieSinn commented Nov 6, 2023

mAAdhaTTah commented Nov 6, 2023

michtio commented Feb 17, 2021 •

edited

Loading

RunDevelopment commented Feb 17, 2021 •

edited

Loading

michtio commented Feb 18, 2021 •

edited

Loading

jazanne commented Feb 2, 2022 •

edited

Loading

skullface commented Aug 3, 2023 •

edited

Loading