How do I create a compiler?

[nicholas-wilcox]

I would like to create a custom compiler. After combing through docs and code, I gather that I am supposed to create a transformer-like method compile(node, file) (or a constructor that creates an object with such a method), whose output ends up in file.result. By using utilities such as unist-util-visit, I can produce the text via different callbacks for the different nodes in the tree.

For context, I am trying to compile Markdown into LaTeX. That is, my processor will look like:

const markdownLatex = unified()
  .use(remarkParse)
  .use(remarkMath)
  .use(myCompiler)
  .freeze();

My use case (not to mention my lack of bandwidth and expertise) is rather slim and idiosyncratic, which is why I am not:

• creating a LaTeX AST and a from-markdown transformer, or
• copying and modify the existing but abandoned remark-latex package.

I am using remark-latex as a reference, though, as the only other compilers I am familiar with, remark-stringify and rehype-stringify, hand off control to their respective syntax-tree utility serializers.

So,

1. Is there anything else I'm missing, or should be on the lookout for in terms of common design patterns for unified compilers?
2. Are there other resources I am not aware of that would be helpful in learning how to create custom compilers?
3. Is this something that would be worth compiling my own notes about?

[wooorm]

Heya!

rebber is also somewhat what you’re looking for: https://github.com/zestedesavoir/zmarkdown/tree/master/packages/rebber.

whose output ends up in file.result.

Typically the return value of a compiler is a string, which will end up on file.value. Non-strings are set at file.result.

By using utilities such as unist-util-visit

unist-util-visit is more typically a utility used in transforms. It could be used in a compiler. But compilers more typically handle each node type and return a string value for them.

[...] as the only other compilers I am familiar with, remark-stringify and rehype-stringify, hand off control to their respective syntax-tree utility serializers.

I would also recommend, at least architecturally, to go that same route.

---

Your original question, “how do I create a compiler”, can be answered with:

function myPlugin() {
  this.Compiler = myCompiler
}

…so it’s a plugin that attaches a compiler. Where myCompiler is either a compiler function or a compiler class. It looks like you already found the docs for those. Here’s an example of a compiler function and a compiler class.

---

For all the other points, it sounds like you have enough information, now it needs to be build!

[nicholas-wilcox]

Thanks for your quick reply. I was not aware of rebber, and I will most likely be fine with using that instead of creating my own compiler.

For what its worth, I'm still a bit unsure of how I might create a compiler with given features. Creating an object of "handlers" that maps node types to stringify functions (some of which are recursive to stringify a node's children) is certainly a natural approach. Like you say, it's enough for me to get started and finished.

However, if I wanted to add settings or contextual behavior to my compiler, I wouldn't see any options other than to design these mechanisms from scratch or to crib code from other compilers. I only say this because existing resources and guides seem to ignore the possibility that someone is looking to learn how to make a compiler. If the official stance is simply that a compiler is (a class with) a function whose inputs are a syntax tree and a vfile, and whose output is a string, then that's perfectly fine.

I'll go ahead and mark your response as an answer. If anyone has any links to guides or writeups about common design practices involved in creating compilers, feel free to post them below.

[wooorm]

However, if I wanted to add settings or contextual behavior to my compiler, I wouldn't see any options other than to design these mechanisms from scratch or to crib code from other compilers.

Note that plugins are an interface that accept options. Configuration happens at that level of abstraction, not compilers. Both of the linked examples I shared accept options. The docs for plugins go into detail on how that works.

I only say this because existing resources and guides seem to ignore the possibility that someone is looking to learn how to make a compiler. If the official stance is simply that a compiler is (a class with) a function whose inputs are a syntax tree and a vfile, and whose output is a string, then that's perfectly fine.

What are you suggesting, which is compiler-specific, that could be documented? We do have a lot of docs on plugins and ASTs, and I don’t think there’s much else to compilers?

[nicholas-wilcox]

I was unclear in my OP. I already knew how to write a plugin that attaches a compilation function by setting this.Compiler. I was trying to ask for advice on the implementation of a compilation function. I'm specifically asking for guides about how various objectives are achieved during the stringify phase, not documentation.

[wooorm]

I’ve written several compilers, and I do sort of “fork” from one another. Though I’ve definitely enjoyed a more functional rather than class based approach in recent years.

xast-util-to-xml is probably my most recent one.
mdast-util-to-markdown the most complex?
You can see that they all kinda follow a similar pattern. A handle function that takes any node and defers to specific handlers per node type. And a context to track some state.

But the language defines a lot. I think a lot of your question depends on that and can’t really be documented in a guide. Markdown is complex. XML relatively simple.
Maybe I’m too close to the topic. Perhaps @ChristianMurphy has some good advice?

[ChristianMurphy]

As @wooorm notes the topic is incredibly broad, I don't think docs could capture the breadth and depth of topics without writing a book (or book series).
There are some good books on the topic, the GCC community for example has collected recommendations on a few https://gcc.gnu.org/wiki/ListOfCompilerBooks

A more general recommendation when it comes to parsers and compilers.
They are complex and time consuming to manage, lean on existing parsers and compilers (several for latex noted in syntax-tree/unist#31) where possible, and write a light wrapper that translates the tools representation into something compatible with Unist https://github.com/syntax-tree/unist.