Blog Recipes Links

My syntax highlighter

In this article, I'm going to explain how I chose a syntax highlighting library for my blog and how I improved it because yes, I have very high expectations:

Choosing a library

I'm not going to reinvent the wheel. Let's explore the different libraries available on the Internet:

After comparing highlightjs and shiki, I decided to choose shiki because the colors look great and sharp (inspired by VScode themes) and it's a way for me to test the flexibility and modularity of Eleventy. Let's run Shiki for the first time to see what the output looks like:

HTML
<!-- codeToHtml(`const shiki = require('shiki')`, { lang: 'js' }) -->
<pre class="shiki">
  <code>
    <span class="line">
      <span style="color: #81A1C1">const</span>
      <span style="color: #D8DEE9FF"> </span>
      <span style="color: #D8DEE9">shiki</span>
      <span style="color: #D8DEE9FF"> </span>
      <span style="color: #81A1C1">=</span>
      <span style="color: #D8DEE9FF"> </span>
      <span style="color: #ECEFF4">'</span>
      <span style="color: #A3BE8C">shiki</span>
      <span style="color: #ECEFF4">'</span>
    </span>
  </code>
</pre>

Not too bad but few problems here:

In my opinion, a better approach would look like this:

HTML
<!-- classes are prefixed by 'h' for 'highlight' to make every class unique -->
<!-- codeToHtml(`const shiki = require('shiki')`, { lang: 'js' }) -->
<pre class="shiki">
  <code>
    <span class="line">
      <span class="h1">const</span>
      <span class="h2"> </span>
      <span class="h3">shiki</span>
      <span class="h2"> </span>
      <span class="h1">=</span>
      <span class="h2"> </span>
      <span class="h4">'</span>
      <span class="h5">shiki</span>
      <span class="h4">'</span>
    </span>
  </code>
</pre>

<!-- The following CSS rules will be embedded in a CSS file for caching concerns -->
<style>
  .shiki {
    --h1: #81a1c1;
    --h2: #d8dee9ff;
    /* ... */
  }

  @media (prefers-color-scheme: dark) {
    .shiki {
      --h1: #0550ae;
      --h2: #a5d6ff;
      /* ... */
    }
  }

  .h1 {
    color: var(--h1);
  }
  .h2 {
    color: var(--h2);
  }
  /* ... */
</style>

There are a lot of benefits here:

Enhancing shiki's output

There's a special theme css-variables that uses CSS variables instead of hardcoded values. Using this theme would allow me to support dark theme. After testing this feature, it turns out the result is not as great as choosing a normal theme like github-light. So I'm going to bake some homemade code.

Shiki's API is well-designed. It exposes 2 functions: codeToHtml which returns the HTML output from a string of code and codeToThemedTokens which returns an intermediate representation of the highlighted code. I'm going to use codeToThemedTokens so I can have control over the HTML rendering. The function returns an object of type IThemedToken[][], the first dimension is for the lines and the second dimension is for the different tokens, nothing fancy here.

import { encode } from 'html-entities'
import { Highlighter, IThemedToken } from 'shiki'

// eslint-disable-next-line @typescript-eslint/no-var-requires, unicorn/prefer-module
const mapping = require('./theme-mapping.json')

class Highlight {
  constructor(private readonly h: Highlighter) {}

  /**
   * Highlight code with colors.
   * @param {string} code
   * @param {string} language
   * @returns {string} HTML content
   */
  highlight(code: string, language: string): string {
    // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
    const tokens = this.h.codeToThemedTokens(code, language)
    return this.renderToHTML(tokens)
  }

  /**
   * Build the HTML output from tokens, https://github.com/shikijs/shiki/blob/main/packages/shiki/src/renderer.ts#L24
   * @param tokens tokens from the shiki library, see `codeToThemedTokens`
   * @returns {string} HTML content
   */
  renderToHTML(tokens: IThemedToken[][]): string {
    // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
    const theme = this.h.getTheme()

    let html = ''
    for (const line of tokens) {
      html += `<span>`
      for (const token of line) {
        const className = token.color ? mapping[token.color.toUpperCase()] : theme.fg
        html += this.generateSpanTag(token, className)
      }
      html += `</span>\n`
    }
    return html.trimEnd()
  }

  generateSpanTag(token: IThemedToken, className) {
    if (!token.content.trim()) {
      return `<span>${this.processToken(token)}</span>`
    }
    return `<span class="${className}">${this.processToken(token)}</span>`
  }

  processToken(token: IThemedToken) {
    const HTTP_URL = /(https?:\/\/.[\w#%+.:=@~-]{2,256}\.[a-z]{2,6}\b[\w#%&+./:=?@~-]*)/g
    const t = token.content.match(HTTP_URL)
    if (t) {
      return token.content.replaceAll(t[0], `<a href="${t[0]}">${encode(t[0])}</a>`)
    }
    return encode(token.content)
  }
}

export { Highlight }
import { promises as fs } from 'node:fs'
import path from 'node:path'
import { loadTheme } from 'shiki'

interface Colors {
  light: string
  dark: string
}

// https://github.com/shikijs/shiki/tree/main/packages/shiki/themes
const DEFAULT_THEME = 'github'

function generateClass(index) {
  return `h${String(index).padStart(2, '0')}`
}

function generateCSS(colors: Array<Colors>) {
  let cssOutput = ''
  const variables = colors.map((colors, index) => `    --${generateClass(index)}: ${colors.light};`)
  const darkVariables = colors.map((colors, index) => `    --${generateClass(index)}: ${colors.dark};`)
  for (let index = 0; index < colors.length; index++) {
    const paddedIndex = generateClass(index)
    cssOutput = `${cssOutput}.${paddedIndex} { color: var(--${paddedIndex}) }\n`
  }
  return `.code-block-inner {
${variables.join('\n')}
}

#dark .code-block-inner {
${darkVariables.join('\n')}
}

${cssOutput}`
}

function generateMappingFile(colors: Array<Colors>) {
  const mapping = new Map<string, string>()
  for (const [index, color] of colors.entries()) {
    mapping.set(color.light.toUpperCase(), generateClass(index))
  }
  // eslint-disable-next-line unicorn/prefer-module
  return fs.writeFile(path.join(__dirname, 'theme-mapping.json'), JSON.stringify(Object.fromEntries(mapping), undefined, 2))
}

async function main(themeName: string) {
  const colors = new Map<string, string>()
  const theme = await loadTheme(`themes/${themeName}-light.json`).then((t) => t.settings)
  const darkTheme = await loadTheme(`themes/${themeName}-dark.json`).then((t) => t.settings)
  for (const index in theme) {
    const key = theme[index].settings.foreground || undefined
    if (key) {
      if (colors.get(key) === darkTheme[index].settings.foreground) {
        console.error(`Color ${key} already used for ${theme[index].scope}`)
        continue
      }
      // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
      colors.set(key, darkTheme[index].settings.foreground!)
    }
  }

  const sortedColors = [...colors]
    .map(([light, dark]) => ({ light, dark }))
    .sort((a, b) => {
      return a.light.localeCompare(b.dark)
    })

  await generateMappingFile(sortedColors)
  console.log(generateCSS(sortedColors))
}

// node_modules/.bin/ts-node modules/eleventy/generate-theme.ts
// eslint-disable-next-line unicorn/prefer-top-level-await
main(process.argv[2] || DEFAULT_THEME).catch(console.error)
{
  "#032F62": "h00",
  "#22863A": "h01",
  "#6F42C1": "h02",
  "#005CC5": "h03",
  "#6A737D": "h04",
  "#F6F8FA": "h05",
  "#586069": "h06",
  "#B31D28": "h07",
  "#E36209": "h08",
  "#D73A49": "h09",
  "#24292E": "h10",
  "#FAFBFC": "h11"
}
<span>
  <span class="h11">const</span>
  <span> </span>
  <span class="h03">shiki</span>
  <span> </span>
  <span class="h11">=</span>
  <span> </span>
  <span class="h05">require</span>
  <span class="h08">(</span>
  <span class="h01">'shiki'</span>
  <span class="h08">)</span>
</span>