Unification of class names across languages

[isagalaev]

Problem

One of the early design principles for highlight.js was having just a few common class names in order to have universal styles that would work for any language. Unfortunately this principle wasn't strongly enforced. We now have language-specific classes, language-specific style rules and whole language-specific styles. This is a maintenance nightmare as the number of unique conditions that should be visually tested is a production of the number of unique language features and the number of styles. And since this is an impossible amount of work, we usually test only a small subset of languages and styles and rely on pure luck for the rest which is a majority. (source)

Principles

• Class names should have stronger semantics. A "variable" in PHP is hardly worth highlighting at all while a "variable" in Prolog is essential for understanding the code and is as important as, say, function definitions in other languages. This means that we'll have to invent a separate name for every case that has to be treated differently in different languages.
• The set of available class names should be rich enough for all languages and adding a new one after we convert all we currently have should require a serious reason.
• The class reference becomes a flat list of available class names with explanations of their meaning.
• All styles become language-agnostic. We may even drop generating class names for languages altogether.

Process

There is a a long lived branch where the conversion is slowly happening: https://github.com/isagalaev/highlight.js/tree/new-styles.

It should eliminate all language names from CSS files. Every current class name should also be evaluated and many will be either dropped or change their semantics. At every such change (removal of a language specific rule or redefinition of a class name) we should make sure that all styles look reasonable. This means that we'll definitely loose similarity with other highlighter engines emulated by various styles, but that's an inevitable implication of the whole task.

During the whole conversion process we will have to intelligently port new languages and styles into the new rules. Possibly encouraging contributors to do that :-)

Status

• 1c
• accesslog
• actionscript
• apache
• applescript
• asciidoc
• armasm
• aspectj
• autohotkey
• autoit
• avrasm
• axapta
• bash
• brainfuck
• cal
• capnproto
• ceylon
• clojure
• clojure-repl
• cmake
• coffeescript
• cpp
• crmsh
• crystal
• cs
• css
• d
• markdown
• dart
• delphi
• diff
• django
• dns
• dockerfile
• dos
• dust
• elixir
• elm
• erb
• erlang-repl
• erlang
• fix
• fortran
• fsharp
• gams
• gcode
• gherkin
• glsl
• go
• golo
• gradle
• groovy
• haml
• handlebars
• haskell
• haxe
• http
• inform7
• irpf90
• ini
• java
• javascript
• json
• julia
• kotlin
• lasso
• less
• lisp
• livecodeserver
• livescript
• lua
• makefile
• mathematica
• matlab
• mel
• mercury
• mizar
• mojolicious
• monkey
• nginx
• nimrod
• nix
• nsis
• objectivec
• ocaml
• openscad
• oxygene
• parser3
• perl
• pf
• php
• powershell
• processing
• profile
• prolog
• protobuf
• puppet
• python
• q
• r
• rib
• roboconf
• rsl
• ruby
• ruleslanguage
• rust
• scala
• scheme
• scilab
• scss
• smali
• smalltalk
• sml
• sqf
• sql
• stata
• step21
• stylus
• swift
• tcl
• tex
• thrift
• tp
• twig
• typescript
• vala
• vbnet
• vbscript
• vbscript-html
• verilog
• vhdl
• vim
• x86asm
• xl
• xml
• xquery
• zephir

[isagalaev]

@sourrust, @Sannis guys I've got another perfectionist idea and I'd like to hear your opinion on that… Thanks!

[sourrust]

I say go for it. A longer running beta with all the change that have happened since 7.5 is probably best.

[isagalaev]

Damn… @sourrust I missed your reply, sorry. However I ultimately decided against it as the job proved to be bigger and more complex than I thought. It just requires a lot of testing through all the languages and styles so I'd rather do it slowly without having a release hanging over.

[isagalaev]

… or I'd be happy if some of you guys could pick it up, I'm not selfish :-)

[isagalaev]

Expanding the idea:

One of the early design principles for highlight.js was having just a few common class names in order to have universal styles that would work for any language. Unfortunately this principle wasn't strongly enforced. We now have language-specific classes, language-specific style rules and whole language-specific styles. This is a maintenance nightmare as the number of unique conditions that should be visually tested is a production of the number of unique language features and the number of styles. And since this is an impossible amount of work, we usually test only a small subset of languages and styles and rely on pure luck for the rest which is a majority.

One way to deal with that is to confine class names to a very generic fixed set and force languages to use only that. Apart from reducing the amount of mess it will also enable an interesting side feature — automatically generated styles. If the semantics of class names is fixed we could intelligently group them and assign to those groups a few distinct font/color combinations provided by a user.

And this is certainly going to be completely backwards incompatible.

[sylvinus]

+1 to this!

[isagalaev]

It's begun: http://softwaremaniacs.org/blog/2015/05/06/styles-unification-first-results/en/

Everyone feeling brave is welcome to join the work on the new-styles branch! The way I do it is roughly this:

• Take a language file at random.
• For every class names being used make sure it doesn't feel wrong (like "tag" for section headers in Apache) and it is defined in the class reference. If there's no suitable name there already, invent and document a new one. This is a sensitive part of the job with no formal criteria, so I'd be happy to discuss that.
• Go through all the styles and make sure that the visual result for this particular language is good. The painful part is that many styles are quite inconsistent and you have to second-guess the author as to what things were supposed to mean. For example there's new class ".hljs-meta" that is supposed to replace all the old preprocessor directives, processing instructions, shell prompts, etc. The problem is that many styles define different colors for those and you have to try all of them to find that looks fine.
• Along the way, remove all language-specific and dead class names from this iteration from styles.

[non]

Just my 2¢ -- this is a great plan!

I initially started working on the Scala mode due to the use of highlight.js in Deckset. However, even after my changes were released, I was disappointed that Scala code still didn't highlight well in many (most?) styles, including the one Deckset seemed to be using.

I started doing what you suggested for Scala. Once I have something that seems good, shall I submit a PR against your branch?

[isagalaev]

@non yes, please do! I suggest doing it this way:

• update the language definition and the class reference (+ probably any one style as an example)
• submit the PR so we can discuss and settle the semantics of the classes
• then go ahead and update all the other styles

[non]

Sounds good.

[isagalaev]

@non just ported the rest of the styles with Scala changes. Check!

[non]

Thanks! 🎩

[Sannis]

I'll push some languages to #911.

[Sannis]

Zephir missed in list and it looks fine.

[isagalaev]

Ah, I knew I was gonna miss something :-) Thanks!

[isagalaev]

Okay… Upon re-writing the style guide this looked pretty much ready and did burn my hands a lot, so I decided to merge it into master right away. I was thinking about publishing a separate beta demo page from the branch but my prior experience show that nobody ever wants to test beta versions of small libraries :-) I believe we've got enough time before the next release date to iron out possible quirks (see #967).

P.S. Woo-hoo!

[Sannis]

Woo-hoo!