I made a new programming language! (((Note: right now it can only add number literals to each other and export constants, but it runs!!))

[Part 1] Introduction

Name: The Making of a Programming Language: Slate [Part 1]
Rating: 1.3 (3065 reviews)
Author: nektro

Slate!

In the recent time, I've been working on making a brand new programming language and it's called Slate! This is going to be a series, more or less start to "finish" as I document my progress making the compiler, standard library, and maybe even some programs in Slate.

Slate is the (first?) programming language that compiles directly from source code to WebAssembly. Yes, that's why I've been asking about WASM for so long :). The syntax is largely inspired by JavaScript ES2015+ with other influences from Java, Kotlin, and more.

So what can it do right now? A new language? Why? How is it different? Can I use it?

Right now this is about all it does.

/**
 * https://github.com/nektro/slate/blob/master/tests/02.grammar/01.operators/001.slate
 */
//
export const expected = 80;
//
export function main(): i32 {
    return 48 + 32;
}

What do?

You can export integer constant literals and export a function that adds integer literals together. That's about it. But slate.js can fully parse this and export a WASM Module that does the same, albeit very literal for now.

Why make?

I really love JavaScript. This love stems from a broader love of the Web as a platform as a whole and JS is all we get. Until now! WebAssembly is the answer to the old question "is the Web getting any other languages other than JavaScript?". With WASM gains access ALL THE LANGUAGES[1]. So in part of a love for JS, part for a desire to make my own language because, and to try to implement features I've never seen before, I set out to make a language specifically for the Web through WASM.

[1]: provided aforementioned language has the proper toolchain

And through this series I'm going to document more or less the entire journey.

How different?

Slate is strongly typed. So that's one thing that's different. But I also want to add things like operator overloading, object extensions(Like adding onto <Object>.prototype but in a statically typed lang), and more.

Can I use it?

Technically yes! If you'd like to compile the program above and run it in your very own WebAssembly-supporting browser then you can do the following:

import * as slate from "https://rawgit.com/nektro/slate/master/src/slate.js"

const slate_program = `
/**
 *
 */
//
export const expected = 80;
//
export function main(): i32 {
    return 48 + 32;
}
`;

Promise.resolve(slate_program)
.then(x => slate.parse(x))
.then(x => x.instance)
.then(x => x.exports)
.then(x => {
    // `x` == { expected: 80, main: func() { [native code] } }
});

How did you do it?

Like any other language, there are a number of steps that are similar between making a compiler and all of them take place in Slate as well.

Lexical Analysis
Parser
Semantic Analyzer
Code Generation
Linker

1. Lexical Analysis

This step was made easy because the majority of code used for this part was already written when I made an HTML preprocessor and added to my basalt javascript library. The code for Slate's lexer can be found here.

Our lexer will take the text of our program and do some really handy things for us. It has to remove the comments, as well as convert the code into a list of tokens with data that we can then later pass onto the parser.

So with the lexer set up properly, basalt will turn our test program into something like the code below.

[
    Key("export"),
    Key("const"),
    Id("expected"),
    Symbol(=),
    Int(80),
    Symbol(;),
    Key("export"),
    Key("function"),
    Id("main"),
    Symbol("("),
    Symbol(")"),
    Symbol(":"),
    Id("i32"),
    Symbol("{"),
    Key("return"),
    Int(48),
    Symbol("+"),
    Int(32),
    Symbol(";"),
    Symbol("}")
]

2. Parser

This part of the process was also very involved when I made my HTML preprocessor so parsing is also a module in basalt. Basalt helps us build a parser but we still have to add all the magic. Slate's parser is here. Those familiar with the computer science here, we are attempting to create a formal language by means of a pseudo-context-free grammar. ANTLR is another big project in this space of creating a lexer/parser in a format much more similar to Backus–Naur form.

Simply put, we have to come up with a series of patters that can take our token list from before and compress it down into a single express that we can then analyze later to create our program.

After that process, our test program looks more like this:

Note: I'm skipping the code demo part because the output from the parser is very verbose and the next step we're going to condense it down a bit to show the same information but in a lot more useful format

3. Semantic Analyzer

This part is done in Slate by the "converter" which takes the very verbose output from the parser, verifies it, and generates the AST. The source for the Slate converter can be found here.

So now what does our program look like?

File(
    Export(
        Const(
            "expected"
            80
        )
    )
    Export(
        Function(
            "main"
            Parameters(
            )
            "i32"
            Block(
                Return(
                    Add(
                        48
                        32
                    )
                )
            )
        )
    )
)

4. Code Generation

Whew! Almost there! At this point we have a nice AST but now need to compile to WebAssembly so that it's able to run by WebAssembly.instantiateStreaming(), etc. Since I wanted to make this a little easier on myself, I decided to have my compiler generate WASM in the text format as opposed to the binary format and then to use wabt to convert the text to binary WASM. Trust me, I love WebAssembly and what it stands for, but even trying to figure out the text format has been difficult. There is very little docs on the formats currently and most of what I've going off is the WASM platform spec tests and output from various WASM playgrounds.

The code for generating WAST from our AST is actually attached to the objects sent out of the converter, so that code is here. After generation of said WAST we shoudld get the following:

(module
    (memory $0 1)
    (global i32 (i32.const 80) )
    (export "expected" (global 0) )
    (func (export "main") (result i32)
        (i32.add (i32.const 48) (i32.const 32) )
    )
)

Hooray! 🙌

5. Linker

For now we're actually done. Imports are not currently implemented and there is no standard library yet, so this phase will have to come later.

Thanks for reading! If you liked this, let me know what you'd like to see in the future of Slate and stay tuned for more!

Coming up in future installments:

Design ideas and long term goals
More operators
Type inference to add support for floating point numbers and objects
Variables
if, while, for, etc
Strings
More functions
Classes
Metaprogramming

Links to save you a scroll:

Follow Slate on GitHub! https://github.com/nektro/slate
Follow me on Twitter! https://twitter.com/nektro
Follow me on Dev! https://dev.to/nektro