Proposal: create a user-facing benchmarking extension

[krassowski]

Problem

Profiling of the UI performance issues is difficult due to differences in:

• browsers and browser versions
• sets of extensions users have installed, some of which are not even public
• notebooks, cell outputs and other main area widgets used
• complexity of workspaces

Some browser-specific performance regressions are caused by CSS rules which affect the number of elements marked for style recalculation; existence of a single such a rule in an extension will cause JupyterLab to slow down substantially unless browser implements clever optimisation strategies. Such issues by definition cannot be detected on CI.

Proposed Solution

I would like to propose a creation of user-facing extension enabling users and managers of JupyterLab deployments to benchmark the UI of JupyterLab locally, providing information on performance impact of installed extensions in a specific browser of choice for a set or pre-specified actions. The extension would ideally be forward compatible with upcoming Jupyter Notebook v7.

I would like to enable users/workspace administrators to:

• run benchmarks for pre-specified UI actions such as (see Should we add more test cases: menu responsiveness, latex and moving tabs around #84):
  • opening context menu
  • switching menus in the menu bar
  • switching tabs
  • opening notebooks
  • opening completer
• parameterise the benchmarks (e.g. using my_notebook.ipynb for tab switching)
• generate results to share with developers (of extensions or of JupyterLab)
  • automatic or semi-automatic uploading of reports is a non-goal as that would require some kind of privacy compliance checks, we don't really need that
  • the result should optionally allow to include (allowing to deselect each of the options separately):
    • count of HTML nodes on the page, broken down by type
    • basic, anonymised workspace information (how many notebooks, how many cells, how many files were opened etc)
    • basic information about the browser/OS versions
• run the benchmarks in any supported browser to be able to exactly capture the performance issue the users are facing
• the benchmarks would cover:
  • timing with specific stylesheets disabled to narrow down the plausible source of the problem
  • timing specific style rules disabled
  • JavaScript self-profiling for browsers which support it (currently only Chromium-based browsers) to understand if the problem comes from JS code, or any specific browser operation like layout, style or painting;
    • self-profiling would also require a server extension to enable self-profiling, see Enabling javascript self-profiling jupyter-server/jupyter_server#1029

I don't know how to name it, but for now we could name the npm package @jupyterlab-benchmarks/ui-profiler and the PyPI jupyterlab-ui-profiler.

Some UI ideas for the UI profiler:

• it would be accessible from the Help menu bar menu with an option to disable/re-enable the menu entries from settings/command palette
• the results would be displayed in a main area widget; for simplicity that could be an HTML render of static output or a notebook

I already have a PoC for the CSS benchmarking part, and if we are happy to proceed with the idea of making it a user-facing extension, I would like to wrap it in an extension, integrate with CI (for the part which can be integrated) and open a PR.

Additional context

• Another project VS Code, according to this wiki on GitHub:
  • includes a self-profiler to record performance of extensions,
  • allows users to easily report workspace metadata etc,
  • for the renderer process it relies on the Chromium JS profiler.

[fcollonval]

@krassowski as you released https://github.com/jupyterlab/ui-profiler, can this issue be closed?