Answers to some commonly asked questions about how to organise Common Lisp code, and use or create libraries. Covering systems, packages, ASDF, Quicklisp, and more.

“Libraries” are a tricky concept in Common Lisp, probably as the language has been around for ages and the preferred way of doing things has evolved. This article aims to give you some insight into how things are done these days, and hopefully clear up some confusion.

We’ll start with an overview of some fundamental concepts and definitions, then look at source code organisation styles, and finally talk about using open source libraries. Read on for the info, or go straight to the examples.

Definitions

For the purposes of this article, a library is a collection of code for doing something specific, designed to be used as part of a larger library or application. This is a pretty vague definition. Common Lisp has a few ways of grouping code together to create libraries and applications. Here is a brief overview of the main concepts you’ll need.

For more detailed info, go and read http://weitz.de/packages.html, a fantastic and detailed description of how all this stuff works. Also useful is the PCL chapter on packages.

Packages

“Packages” are defined by the Common Lisp standard as containers for symbols, similar to C++ namespaces (see Symbols for more info about what these magical “symbol” things are). Key point: one does not “install” a package; that’d be silly, it’s (effectively) just a group of names. Instead, the code that defines the package is loaded (more details), and then the symbols in the package can be accessed. We’ll see how to do that later.

When you start your Lisp REPL, you probably end up with a prompt that contains something like CL-USER. This indicates that your REPL input is evaluated in the context of the CL-USER package, which is the default package intended for general experimentation, and has all the standard Common Lisp symbols available. See PCL for more details.

Normally when writing an application or library, it is advisable to create a package to contain the application symbols, rather than definiing everything in the CL-USER package.

Systems

“Systems” are not defined in the CL standard, but they have been around as a concept for decades. Essentially they are groups of code (usually including one or more package definitions), and any information required to build and run the code (dependencies, unit tests, etc). They typically also include ‘extra’ information about the code (author, license, system name, etc).

Loading Code

Developing and running Common Lisp code is different from most languages in use today. We develop applications by building upon a Lisp “image” (blank canvas), and we use source code files to record the building steps. Sort of.

Anyway, the thing to remember is that when you start a new Lisp instance (for example, by running sbcl at the command line), it comes up blank, with no user code. In order to run things, you must load definitions into it (or define code directly from the REPL, of course). This is typically done by either using the Common Lisp LOAD function, or using ASDF to load a system (see below).

Using Packages

Remember that Common Lisp packages are similar to namespaces, or python modules. Knowing this, you will almost always want to define a package to hold code you write.

When your code is spread across multiple files, the preferred organisational style is to use one single package.lisp file at the top level of your project which contains the package definitions. Then each source file simply declares which package its symbols lives in. The other option is to use a single package definition per source code file. This means packages never span multiple files, and dependencies are more clearly indicated. However it can feel a little cumbersome as the number of files and therefore packages grows. Using a single file to define all packages often feels easier to organise, and also provides a basic but useful “API” and dependency tree for all the files.

Package Definitions

Packages are defined with DEFPACKAGE, and usually contain at least:

  • the package name / identifier (an ASCII string)
  • a list of symbols the package exports
  • a list of other symbols imported into the package

Packages must be named with a so-called “string-designator”, which can be quite confusing, as it results in several different styles of working with packages. These three lines all define a package with the same name (in most lisps with “normal” read-table configurations):

(defpackage :sneaky  (:use #:cl))
(defpackage #:sneaky (:use #:cl))
(defpackage "SNEAKY" (:use #:cl))

However, the first version has the side effect of also interning the :sneaky symbol in the keywords package. The second version does not have this side effect (the #: reader macro results in an uninterned symbol), so a small amount of memory is saved. In either case, the package name is the name of the symbol (which is usually uppercase, unless the *readtable* has been modified).

Most people tend to use either the first or second styles.

Accessing Symbols in Other Packages

There are three commonly used and seen ways to refer to symbols:

  1. bar
  2. foo:bar
  3. foo::bar

In the first case, the Lisp reader will search for the symbol bar in the current package.

In the second case, the symbol will be searched for in the list of symbols exported by the foo package. If the package foo cannot be found, or the symbol bar is not external in foo, an error is signalled.

In the third case, the symbol is evaluated in the context of foo, ignoring the exported symbols list. This allows “unlimited” access to the symbols in foo, but should rarely (if ever) be used in code you write. If a symbol is not exported by a package, it probably is not meant to be used. However it is common to see symbols printed in this format.

Note: when the Lisp reader reads a symbol, it internalises the symbol, which essentially means it creates a symbol of the given name in the current package (if one does not already exist). This can result in conflicts later if a symbol of the same name is inherited (via use-package) from another package. See Symbols for more details.

ASDF

ASDF (Another System Definition Facility) is a kind of build tool on steroids. You can use it to do lots of things, but it’s commonly used to define how to build and load Common Lisp libraries or applications.

ASDF is included in most popular CL implementations, including SBCL. It is the de-facto system definition tool, and most open source projects will come with an ASDF system definition. This definition, contained in a .asd file, probably includes a number of things, such as:

  • The project author, description, license, and other similar data
  • A list of files in the system
  • Any dependencies between files
  • A description of how to run unit tests

Developing with ASDF

Process:

  • create the system definition file (.asd extension)
  • add the containing folder to asdf’s load path. The easiest way is to do (push *default-pathname-defaults* asdf:*central-registry*)
  • then you can do (asdf:oos 'asdf:load-op 'system-name) to “load” the system

The Slime interactivity with ASDF isn’t amazing. The current path has to be pushed onto the central repo, and load op has to be called manually. There should be a slime-load-project command…

Quicklisp

Experimenting with and using open-source libraries is so easy in <favourite-language-here>. I want that in Common Lisp!

Most languages deal with this stuff quite differently. Python has pip, which is fairly well established as the Python package manager. If you want to share some code, you put it on PyPI, and people can find it with pip search. Node.js programmers use NPM (or yarn, if they’re cool), and the NPM package registry. C and C++ programmers must rely on their system’s package manager to provide development libraries, or just download them manually.

Common lisp has traditionally been more like C/C++ in this respect - if you wanted to use someone’s libary you downloaded it and put it somewhere your lisp implementation could find it, typically on the ASDF load path somewhere.

These days we have Quicklisp!

Examples

How to create and use Packages, define Systems, and install stuff with Quicklisp!

We will build a very small web application using ningle to demonstrate how everything hangs together in Common Lisp. This web app will do just one thing: generate us a (probably not cryptographically secure) random character on every page refresh.

We shall call our the application… “bobbio”. You’ll need to create a few files:

  • bobbio.asd – the bobbio system definition
  • package.lisp – the package definitions
  • bobbio.lisp – the main lisp source file
  • web.lisp – the web application code

Packages

We’ll split up our little web app into two packages – one to hold the web stuff, and one to hold the backend functionality. Two packages is overkill for something this small, but lets go with it anyway.

Put the following into package.lisp:

;;;; package.lisp
(defpackage :bobbio
  (:use #:cl)
  (:export
   :get-message))

(defpackage :bobbio.web
  (:use #:cl)
  (:export
   :start-server)
  (:import-from :bobbio :get-message))

This defines our two packages – they are both quite minimal. The first form defines a package named :bobbio using the defpackage macro. One symbol, get-message, is exported by the package – this is the only symbol that will be “visible” to other packages. The :bobbio package also uses all of the symbols from the :cl package (a nickname for :common-lisp), which means all standard Common Lisp symbols (i.e. defun, setf, etc) will be available.

The :bobbio.web package also uses cl (most packages will), exports the start-server symbol, and also imports the get-message symbol from bobbio. This means that code in the :bobbio.web will be able to use get-message as if it was defined in the :bobbio.web package.

bobbio.lisp

Next, put the following into bobbio.lisp:

;;;; bobbio.lisp
(in-package :bobbio)

(defconstant alphabet-length 26)
(defconstant ascii-value-A 65)

(defun random-char ()
  "Return a random character"
  (code-char (+ ascii-value-A
                (random alphabet-length))))

(defun get-message ()
  (format nil "Your random letter is: ~a" (random-char)))

The call to (in-package)[clhs-defpackage] changes the current “namespace” to :bobbio. After that, the file defines a couple of functions we’ll use later on.

web.lisp

Put the following into web.lisp:

;;;; web.lisp
(in-package :bobbio.web)

(defvar *app* (make-instance 'ningle:<app>))

(setf (ningle:route *app* "/")
      (get-message))

(defun start-server ()
  (clack:clackup *app*))

This creates an instance of the ningle:<app> class, which is a very minimal framework for writing web applications. get-message is set as the handler for the “/” route, and the server is started using clack, an HTTP server abstraction.

At this point, if you started a lisp image and evaluated the code in all of these files, you would be able to evaluate (bobbio.web:start-server) and your shiny new web application would spring into existence. Assuming you had the required dependencies (ningle, clack) installed.

bobbio.asd

Put the following into bobbio.asd:

;;;; bobbio.asd
(in-package :cl-user)

(asdf:defsystem #:bobbio
  :version "0.1"
  :description "Something epic"
  :author "You"
  :depends-on (#:ningle
               #:clack)
  :serial t
  :components ((:file "package")
               (:file "bobbio")
               (:file "web")))

This defines the bobbio system and tells ASDF a few things about it

  • which other systems it depends on
  • which components (files) are required to build it
  • build order – each component depends on the previous one (the :serial t line)

Now, from a fresh lisp image, you can evaluate (asdf:oos 'asdf:load-op 'bobbio), and ASDF will use the above definition to load all dependencies and build the source files.

Quicklisp

If you don’t have the two dependencies required for bobbio, they can be easily installed with Quicklisp. Either install them manually:

(ql:quickload :ningle)
(ql:quickload :clack)

Or get Quicklisp to load all dependencies automatically! Quicklisp has a “local projects” directory, in which it will search for ASDF system definitions, letting you load your projects as if they were in the official Quicklisp system repository. This includes loading their dependencies.

You have two options – either do all of your development in the Quicklisp local projects directory (~/quicklisp/local-projects by default), or symlink your projects into it manually. Once there, you can evaluate:

(ql:quickload :bobbio)