Lisp for the web - 5

This is a post about using Common Lisp for web development. Lisp is one of the oldest programming languages that has been still used today. It has been dubbed as the programmable programming language for it's great extensible characteristics. It pioneered many ideas in computer science, including tree data structures, automatic storage management, dynamic typing, conditionals, higher-order functions, recursion, the self-hosting compiler and the read–eval–print loop.

There has been many great posts about using Lisp for Web development in the past. This is my shot to bring Lisp to the ultra-modern web and especially how to deploy your Common Lisp web apps to the cloud.

Being a Front-end developer, I will focus mainly on the following aspects of Web development for this post.

Frameworks
IDE & Tooling
Development environment setup
Deployment

This is an introductory level post to get you started with the tools and practices to develop and deploy your Lisp web apps. I am planning to do more deep dive articles on each of the tools and practices at a later point of time.

These are the most popular posts which dealt with the topic before my post. Please feel free to check them out if you want to have more perspective on web development with Lisp.

How do I get started?

To get started with Lisp for Web development, you basically need two things. A Lisp implementation installed on your machine and an editor with IDE like capabilities to develop web applications with lisp. I am using SBCL as my Lisp implementation and Emacs as my editor. You are free to choose any editor for the task but nothing comes close to Emacs for editing Lisp code. It is the perfect tool for the job at hand.

SBCL

Steel Bank Common Lisp (SBCL) is a high performance Common Lisp compiler. It is open source / free software, with a permissive license. In addition to the compiler and runtime system for ANSI Common Lisp, it provides an interactive environment including a debugger, a statistical profiler, a code coverage tool, and many other extensions.

SBCL runs on a number of POSIX platforms, and experimentally on Windows. See the download page for supported platforms, and getting started guide for additional help.

Installing SBCL on Ubuntu

sudo apt-get install sbcl

Installing SBCL on Mac using Homebrew

brew install sbcl

Once you have installed SBCL in your machine, you can start playing around in the REPL by invoking it from the command line.

sbcl

You can type in any Lisp expression in the REPL and get the result immediately.

CL-USER> (+ 1 2)
3

Quicklisp

Once you installed SBCL , the next step is to setup Quicklisp. Quicklisp is a library manager for Common Lisp. It works with your existing Common Lisp implementation to download, install, and load any of over 1,500 libraries with a few simple commands. It's like npm for Node.js, cargo for Rust and similar to other package managers for other language implementations. Go to their home page and refer to the installation instructions for how to install and setup quicklisp for your Lisp implementation.

Emacs

Next comes our editor which is Emacs. Emacs is an extensible, customizable and free text editor, which is, at its core, an interpreter for Emacs Lisp. So what this basically means is that, it has got built in support for editing Lisp code. You can go to their website and follow the instructions for installing on various operating systems and platforms. Emacs has also got a ton of awesome stuff like org-mode, a built-in file explorer called dired and so on. It is an operating system within itself.

SLIME

SLIME is a Emacs mode for Common Lisp development. It is an environment for hacking Common Lisp. It has got a Common Lisp debugger, REPL (Read-Eval-Print-Loop) which is written in Emacs Lisp for tighter integration with Emacs and an interactive object-inspector.

So this is a must have addon for Emacs if you are interested in doing serious Lisp. Once you installed Emacs, you can install slime with M-x package-install and then type slime and press Enter.
You can also refer to the Quick setup instructions on their github README to quickly configure SLIME.

You can start SLIME with M-x slime which is Alt or Meta key + x and then type slime and enter. It will start the default Lisp program you configured (SBCL in our case) before, within Emacs itself. You can play around in the interpreter just like you did with SBCL earlier.

Caveman

Once you are ready with your IDE setup and the required tools installed in your machine, it's time to do the real job. Creating a web application project with Lisp.

In the older posts I mentioned above, there were really no frameworks used for setting up and developing web applications for Lisp. They were just hand-made, plain and simple.

But if you want to have scalable and maintainable code base for your web applications you need to have some sort of conventions on how you organize your Lisp code for building websites and applications. Having used frameworks like Ruby on Rails and Express.js, I am naturally inclined to bootstrap our project with a framework. Because conventions make you productive in the long run, it save you from making arbitrary decisions about how you organize your code and make your codebase maintainable in the future.

Hence I chose Caveman for this project. After having been played around with and without Caveman for building web applications in Common Lisp, I found that it is the best framework out there for developing web apps in Lisp. Caveman is a lightweight web application framework created by Eitaro Fukamachi for Common lisp. Fukamachi has got some serious tools for doing web development in Lisp. Please feel free to check out his Github profile for more useful tools.

Caveman is available on Quicklisp, so you can install it with:

(ql:quickload :caveman2)

And this is how you can generate a project skeleton with Caveman:

(caveman2:make-project #P"~/quicklisp/local-projects/hello-caveman" :author "Rajasegar")

This will generate a project folder in "~/quicklisp/local-projects", which is the default location from where you can load your projects into Quicklisp. So which means you can load our newly created app in quicklisp like below:

(ql:quickload :hello-caveman)

Your application has functions named start and stop to start/stop your web application.
So you can start the development web server like this:

(hello-caveman:start :port 3000)

Open your browser and visit the url http://localhost:3000. You will get a blank web page saying Welcome to Caveman2.

Adding new routes

You can add new routes to your web application by editing the web.lisp file situated under src folder of your newly created project.
Say for example if you want to add a new route like http://localhost:3000/about to your app, you can define a new route under the default route like below.

(defroute "/about" ()
  (render #P"about.html"))

src/web.lisp

        (in-package :cl-user)
        (defpackage super-rentals.web
          (:use :cl
            :caveman2
            :super-rentals.config
            :super-rentals.view
            :super-rentals.db
            :datafly
            :sxql)
          (:export :*web*))
        (in-package :super-rentals.web)

        ;; for @route annotation
        (syntax:use-syntax :annot)

        ;;
        ;; Application

        (defclass <web> (<app>) ())
        (defvar *web* (make-instance '<web>))
        (clear-routing-rules *web*)

        ;;
        ;; Routing rules

        (defroute "/" ()
          (render #P"index.html"))

        (defroute "/about" ()
          (render #P"about.html"))

        ;;
        ;; Error pages

        (defmethod on-exception ((app <web>) (code (eql 404)))
          (declare (ignore app))
          (merge-pathnames #P"_errors/404.html"
                   *template-directory*))

Adding new template for your route

You can create new templates for your routes in the templates folder situated at the root of your project. Just create a new file about.html and some HTML to it.

Djula Templates

A good framework provides solid support for templates in your web apps. Because it helps you to express your markup with dynamic information from the server.

Caveman uses Djula templating for the job. Djula is a port of Python's Django template engine to Common Lisp.
Djula’s template language is designed to strike a balance between power and ease. It’s designed to feel comfortable to those used to working with HTML.

<h1>About page</h1>
<p>This is the about page</p>

Now go to your web.lisp file, reload your changes to reflect the newly created routes to take effect. In Emacs, you can just do C-c C-l which is Ctrl-c and then Ctrl-l to compile your file and load it into the SLIME environment. Then go to your browser and visit the url http://localhost:3000/about to view your new page. The beauty of using a framework like Caveman is whatever changes you make inside your templates, the HTML files, will be automatically reflected in the browser. All you need to do is to just refresh the page.

Adding styling

There is a default css file called main.css under the static/css folder in the root of your project. You can add any custom css properties and declarations to style your web pages.

/* static/css/main.css */
h1 {
 color: red;
}

The above code will make all your h1 tags to have red color text. Now we have played around enough to understand our project structure and developed a very simple web app, it's time to deploy our app in the cloud so that we can share it with others.

What is the point in creating a cool web app, if you are not able to share it to the outside world and get some feedback. This is one of the important aspect which has not been dealt with the previous posts about Common Lisp and web development.

Modify your home page to include links for the about and home page. For that you need to make the following changes to templates/index.html file. This is the home page template for our app.

    {% extends "layouts/default.html" %}
    {% block title %}Lisp for the web{% endblock %}
    {% block content %}
    <div id="main">
      <h1>Lisp for the Web</h1>
      <nav>
        <ul>
          <li><a href="/">Home</a></li>
          <li><a href="/about">About</a></li>
        </ul>
      </nav>
      <p>Source Code: <a href="https://github.com/rajasegar/lisp-for-the-web">Github</a></p>
    </div>
    {% endblock %}

Deployment

Now we will see how to deploy our Common Lisp web app to the cloud. I am using Heroku for it. Heroku is a powerful PaaS (Platform as a Service) provider which has got a ton of features for building and deploying web applications. It has got a cool CLI tool with which you can deploy your apps in just minutes, you can view the logs from your command line , use custom buildpacks and so much more. I have been using Heroku for a decade now, and I have a lot of demos created in it. So for our app, we are going to use the same.

First we need to heroku CLI tool to be installed in our machine. You can visit the heroku-cli official page for instructions on how to install in various platforms. Once you installed the heroku CLI, for creating a new app all you need to do is just fire this command from the project folder.

heroku apps:create my-awesome-app

But for our Lisp application, this is not enough, because we don't have inbuilt support for Lisp language within Heroku platform, so we are going to use something called buildpacks.

Heroku Buildpacks

Buildpacks are the scripts that power app builds on Heroku. They are composed of a set of scripts, and depending on the programming language, the scripts will retrieve dependencies, output generated assets or compiled code, and more.

This output is assembled into a slug by the slug compiler. Heroku’s support for Ruby, Python, Java, Clojure, Node.js, Scala, Go and PHP is implemented via a set of open source buildpacks.

Heroku maintains a collection of officially supported buildpacks that are available by default to all Heroku apps.

So for our Common Lisp web application to be properly deployed into Heroku, we need a custom buildpack. Fortunately there are so many Common Lisp buildpacks created by some awesome people for deploying Lisp web apps into the Heroku platform. We are going to use one of them, called heroku-buildpack-roswell created by gos-k. So how we are going to use this buildpack. We need to pass a flag --buildpack with the url of the buildpack to our heroku command like this.

heroku apps:create my-awesome-app --buildpack https://github.com/gos-k/heroku-buildpack-roswell

This tells Heroku to use the custom buildpack for building our Common Lisp web apps and deploy them. But before creating the app we need to do some setup in our project for the buildpack to work. The buildpack is based on roswell.

roswell

Roswell is a Lisp implementation installer/manager, launcher, and much more. It started out as a command-line tool with the aim to make installing and managing Common Lisp implementations really simple and easy. It has now evolved into a full-stack environment for Common Lisp development, and has many features that makes it easy to test, share, and distribute your Lisp applications. With Roswell, we can push the Common Lisp community to a whole new level of productivity.

So in order for our custom buildpack to work we need to create four new files at the root of the project.

.roswell-install-list

This file tells roswell to install these packages as dependencies for your application.

    clack

.roswell-use

This file tells roswell to use the default sbcl binary for our applications.

    sbcl-bin

.roswell-load-system-list

This file tells roswell to load these systems as part of our project dependencies.

    clack

Procfile

This file tells Heroku the startup command to start our Web server using Clack's clackup command.

web: clackup --port $PORT --address 0.0.0.0 app.lisp

Clack is a web server abstraction layer for Common Lisp inspired by Python's WSGI and Ruby's Rack.
Clack provides a script to start a web server. It's useful when you deploy to production environment.

We supply the --port flag to use the env variable PORT assigned by Heroku for your project and use the --address flag to use the 0.0.0.0 address instead of localhost. This flag is very important otherwise your app will throw a startup error saying that the application is not bound to the port within the specified timeout. And finally we supply our main file app.lisp for clack.

After creating all these files, we still need to do one last change within our code to make the buildpack work. This is to the tell the asdf system to pick the project system definition from the /app folder in Heroku once the build is completed and the project is ready to load. We will add this line to the top, in our app.lisp file in the root of the project folder.

What is asdf?

ASDF stands for Another System Definition Facility. It is what Common Lisp hackers use to build and load software.
ASDF is the de facto standard build facility for Common Lisp. Your Lisp implementation probably contains a copy of ASDF, which you can load using (require "asdf").

It is a tool to describe how Lisp source code is organized in systems, and how to build and load these systems. Every lisp project have a system definition file that ends with .asd extension where you can specify some details about your project. It is like a project manifest file similar to package.json for Node.js and cargo.toml for Rust.

;; app.lisp
(push #p"/app/" asdf:*central-registry*)

Once you add all the above changes, commit it and push it to heroku branch and automatically your app will be built and deployed with the new buildpack.

git push heroku master

That's it you are ready to view your web app in action in the url https://lisp-for-the-web.herokuapp.com The source code for this post is hosted in Github.

Hope you enjoyed the post and learned how to create web applications in Common Lisp and how to deploy them to a cloud platform like Heroku.

Please feel to to throw your feedback/queries in the comments section, will be glad to answer and help.

Here are some of the other demos I built with Common Lisp:

Lisp for the Web - 5