shikiTokenizer

function · packages/richtext-code-shiki/src/shiki.ts:58

function shikiTokenizer(
  highlighter: HighlighterCore,
  language: string,
): CodeTokenizer

The tokenizer for one language the highlighter has loaded. It throws here, when it is made, if the language or any theme is missing, rather than on the first render. Adjacent pieces of the same kind become one token. Results are remembered by text, so an unchanged block is not tokenized again.