Building a compiler – basic setup

Siddharth - Jun 6 '21 - - Dev Community

Building a compiler – basic setup

Be sure to check out the previous part to learn the basics

Now that you have a general overview on a compiler, let's setup a few files so you can start creating one!

I'm gonna code this using nodejs, but you can choose any environment you like, as there is nothing nodejs specific in here.

You only have to create a single JS file, and you can safely skip this part of this series if you like. But I recommend doing this.


You still here? Alright let's start!

The files

I like to divide everything up, so I am going to create

  • The main index file
  • The tokenizer
  • The parser
  • The traverser
  • The transformer
  • The code generator

Plus a few more files

  • A simple cli which parses input and logs it to the console, just made for testing purposes
  • A test file which compiles some code and checks if the code is compiled correctly

So I am going to put all the boilerplate code here, which you can just copy and paste. Also, I am using .mjs because I prefer the import/export statements more than require.

index.mjs

import {tokenizer} from './components/tokenizer';
import {parser} from './components/parser';
import {transformer} from './components/transformer';
import {codeGenerator} from './components/code-generator';


export function compiler(input) {
    const tokens = tokenizer(input);
    const oldAST = parser(tokens);
    const newAST = transformer(oldAST);
    const output = codeGenerator(newAST);

    return output;
}
Enter fullscreen mode Exit fullscreen mode

components/tokenizer.mjs

export function tokenizer(input) {

}
Enter fullscreen mode Exit fullscreen mode

components/parser.mjs

export function parser(tokens) {

}
Enter fullscreen mode Exit fullscreen mode

components/traverser.mjs

export function traverser(ast, visitor) {

}
Enter fullscreen mode Exit fullscreen mode

components/transformer.mjs

export function transformer(ast) {

}
Enter fullscreen mode Exit fullscreen mode

components/code-generator.mjs

export function codeGenerator(node) {

}
Enter fullscreen mode Exit fullscreen mode

cli.mjs

import {compiler} from "./index";

const argv = process.argv.slice(2);

argv.forEach(codeBlock => {
    console.log(`\`${codeBlock}\` compiles to\n\n${compiler(codeBlock)}\n`)
});
Enter fullscreen mode Exit fullscreen mode

tests.mjs

import assert from 'assert';
import {compiler} from './compiler';

function itCompiles(codeBlock, expectedOutput) {
    assert.strictEqual(
        compiler(codeBlock),
        expectedOutput
    )
}

// No tests now
Enter fullscreen mode Exit fullscreen mode

That's all, and see you in the next part!

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .