My syntax highlighter
In this article, I'm going to explain how I chose a syntax highlighting library for my blog and how I improved it because yes, I have very high expectations:
The highlighting must be performed at build time. I use Eleventy as a static site generator.
The output format must be HTML.
A Node.js library is preferable since Eleventy requires it.
The library must support at least the following programming languages: HTML, CSS, JS, TS, Makefile, JSON, Golang, Ruby, C, Rust, and Java.
Support of a dark theme is a plus.
Embedding extra JS scripts is not a possibility: As you know, energy prices are getting crazy. I bought a turtleneck sweater for my server but Github doesn't want to give me physical access to my machine so let's save some bytes.
A concise and clean HTML output is appreciated.
I'm not going to reinvent the wheel. Let's explore the different libraries available on the Internet:
codemirror: I use this library for the code vizualiser of SARD. This library is complete but too overkilled for this blog. Furthermore, Codemirror is supposed to be run in a browser.
rainbow: Supported languages don't match my needs.
highlightjs: Highlight looks great. It's also very well integrated with Eleventy.
prism.js: Another library for the browser.
torchlight: There's no way I pay for highlighting my shitty code.
shiki: This one looks great too.
After comparing highlightjs and shiki, I decided to choose shiki because the colors look great and sharp (inspired by VScode themes) and it's a way for me to test the flexibility and modularity of Eleventy. Let's run Shiki for the first time to see what the output looks like:
Not too bad but few problems here:
Colors are hardcoded, meaning there is no way to support different color themes, especially the dark theme.
The HTML output is verbose due to the
style
attribute of each token.A lot of colors are duplicated multiple times. For a small number of lines of code, it's not a big deal but when it comes to sharing big chunks of code, it increases the size of the HTML.
In my opinion, a better approach would look like this:
There are a lot of benefits here:
If I want to change the theme one day, I'll just need to update the CSS files.
I can define as many themes as I want by defining some extra CSS variables.
The HTML is way cleaner than the default output.
There's a special theme css-variables
that uses CSS variables instead of hardcoded values. Using this theme would allow me to support dark theme. After testing this feature, it turns out the result is not as great as choosing a normal theme like github-light
. So I'm going to bake some homemade code.
Shiki's API is well-designed. It exposes 2 functions: codeToHtml
which returns the HTML output from a string of code and codeToThemedTokens
which returns an intermediate representation of the highlighted code. I'm going to use codeToThemedTokens
so I can have control over the HTML rendering. The function returns an object of type IThemedToken[][]
, the first dimension is for the lines and the second dimension is for the different tokens, nothing fancy here.
highlight.ts
is the main module. the functionhighlight
will be used in Eleventy.generate-theme.ts
is a program that must be run once to generate the CSS rules andtheme-mapping.json
.theme-mapping.json
maps an HTML class with a color of a theme (heregithub-light.json
).
import { encode } from 'html-entities'
import { Highlighter, IThemedToken } from 'shiki'
// eslint-disable-next-line @typescript-eslint/no-var-requires, unicorn/prefer-module
const mapping = require('./theme-mapping.json')
class Highlight {
constructor(private readonly h: Highlighter) {}
/**
* Highlight code with colors.
* @param {string} code
* @param {string} language
* @returns {string} HTML content
*/
highlight(code: string, language: string): string {
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
const tokens = this.h.codeToThemedTokens(code, language)
return this.renderToHTML(tokens)
}
/**
* Build the HTML output from tokens, https://github.com/shikijs/shiki/blob/main/packages/shiki/src/renderer.ts#L24
* @param tokens tokens from the shiki library, see `codeToThemedTokens`
* @returns {string} HTML content
*/
renderToHTML(tokens: IThemedToken[][]): string {
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
const theme = this.h.getTheme()
let html = ''
for (const line of tokens) {
html += `<span>`
for (const token of line) {
const className = token.color ? mapping[token.color.toUpperCase()] : theme.fg
html += this.generateSpanTag(token, className)
}
html += `</span>\n`
}
return html.trimEnd()
}
generateSpanTag(token: IThemedToken, className) {
if (!token.content.trim()) {
return `<span>${this.processToken(token)}</span>`
}
return `<span class="${className}">${this.processToken(token)}</span>`
}
processToken(token: IThemedToken) {
const HTTP_URL = /(https?:\/\/.[\w#%+.:=@~-]{2,256}\.[a-z]{2,6}\b[\w#%&+./:=?@~-]*)/g
const t = token.content.match(HTTP_URL)
if (t) {
return token.content.replaceAll(t[0], `<a href="${t[0]}">${encode(t[0])}</a>`)
}
return encode(token.content)
}
}
export { Highlight }
import { promises as fs } from 'node:fs'
import path from 'node:path'
import { loadTheme } from 'shiki'
interface Colors {
light: string
dark: string
}
// https://github.com/shikijs/shiki/tree/main/packages/shiki/themes
const DEFAULT_THEME = 'github'
function generateClass(index) {
return `h${String(index).padStart(2, '0')}`
}
function generateCSS(colors: Array<Colors>) {
let cssOutput = ''
const variables = colors.map((colors, index) => ` --${generateClass(index)}: ${colors.light};`)
const darkVariables = colors.map((colors, index) => ` --${generateClass(index)}: ${colors.dark};`)
for (let index = 0; index < colors.length; index++) {
const paddedIndex = generateClass(index)
cssOutput = `${cssOutput}.${paddedIndex} { color: var(--${paddedIndex}) }\n`
}
return `.code-block-inner {
${variables.join('\n')}
}
#dark .code-block-inner {
${darkVariables.join('\n')}
}
${cssOutput}`
}
function generateMappingFile(colors: Array<Colors>) {
const mapping = new Map<string, string>()
for (const [index, color] of colors.entries()) {
mapping.set(color.light.toUpperCase(), generateClass(index))
}
// eslint-disable-next-line unicorn/prefer-module
return fs.writeFile(path.join(__dirname, 'theme-mapping.json'), JSON.stringify(Object.fromEntries(mapping), undefined, 2))
}
async function main(themeName: string) {
const colors = new Map<string, string>()
const theme = await loadTheme(`themes/${themeName}-light.json`).then((t) => t.settings)
const darkTheme = await loadTheme(`themes/${themeName}-dark.json`).then((t) => t.settings)
for (const index in theme) {
const key = theme[index].settings.foreground || undefined
if (key) {
if (colors.get(key) === darkTheme[index].settings.foreground) {
console.error(`Color ${key} already used for ${theme[index].scope}`)
continue
}
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
colors.set(key, darkTheme[index].settings.foreground!)
}
}
const sortedColors = [...colors]
.map(([light, dark]) => ({ light, dark }))
.sort((a, b) => {
return a.light.localeCompare(b.dark)
})
await generateMappingFile(sortedColors)
console.log(generateCSS(sortedColors))
}
// node_modules/.bin/ts-node modules/eleventy/generate-theme.ts
// eslint-disable-next-line unicorn/prefer-top-level-await
main(process.argv[2] || DEFAULT_THEME).catch(console.error)
{
"#032F62": "h00",
"#22863A": "h01",
"#6F42C1": "h02",
"#005CC5": "h03",
"#6A737D": "h04",
"#F6F8FA": "h05",
"#586069": "h06",
"#B31D28": "h07",
"#E36209": "h08",
"#D73A49": "h09",
"#24292E": "h10",
"#FAFBFC": "h11"
}
<span>
<span class="h11">const</span>
<span> </span>
<span class="h03">shiki</span>
<span> </span>
<span class="h11">=</span>
<span> </span>
<span class="h05">require</span>
<span class="h08">(</span>
<span class="h01">'shiki'</span>
<span class="h08">)</span>
</span>