3.1 System building

3.1.1 Compiling with ECL

In this section we will introduce topics on compiling Lisp programs. ECL is especially powerful on combining lisp programs with C programs. You can embed ECL as a lisp engine in C programs, or call C functions via Foreign Function Interface. We explain file types generated by some compilation approaches. For the examples, a GNU/Linux system and gcc as a development environment are assumed.

You can generate the following files with ECL:

1. Portable FASL file (.fasc)
2. Native FASL file (.fas, .fasb)
3. Object file (.o)
4. Static library
5. Shared library
6. Executable file

Relations among them are depicted below:

Figure 3.1: Build file types

3.1.1.1 Portable FASL

ECL provides two compilers (bytecodes compiler, and C/C++ compiler). Portable FASL files are built from source lisp files by the bytecodes compiler. Generally FASC files are portable across architectures and operating systems providing a convenient way of shipping portable modules. Portable FASL files may be concatenated, what leads to bundles. FASC files are faster to compile, but generally slower to run.

;; install bytecodes compiler
(ext:install-bytecodes-compiler)

;; compile hello.lisp file to hello.fasc
(compile-file "hello1.lisp")
(compile-file "hello2.lisp")

;; reinitialize C/C++ compiler back
(ext:install-c-compiler)

;; FASC file may be loaded dynamically from lisp program
(load "hello1.fasc")

;; ... concatenated into a bundle with other FASC
(with-open-file (output "hello.fasc"
                        :direction :output
                        :if-exists :supersede)
  (ext:run-program
   "cat" '("hello1.fasc" "hello2.fasc")  :output output))

;; ... and loaded dynamically from lisp program
(load "hello.fasc")

3.1.1.2 Native FASL

If you want to make a library which is loaded dynamically from a lisp program, you should choose the fasl file format. Under the hood native fasls are just shared library files.

This means you can load fasl files with dlopen and initialize it by calling a init function from C programs, but this is not an intended usage. The recommended usage is to load fasl files by calling the load lisp function. To work with Native FASL files ECL has to be compiled with --enable-shared configure option (enabled by default).

Creating a fasl file from one lisp file is very easy.

(compile-file "hello.lisp")

To create a fasl file from more lisp files, firstly you have to compile each lisp file into an object file, and then combine them with c:build-fasl.

;; generates hello.o
(compile-file "hello.lisp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lisp" :system-p t)

;; generates hello-goodbye.fas
(c:build-fasl "hello-goodbye"
              :lisp-files '("hello.o" "goodbye.o"))

;; fasls may be built from mix of objects and libraries (both shared and
;; static)
(c:build-fasl "mixed-bundle"
              :lisp-files '("hello1.o" "hello2.a" "hello3.so"))

3.1.1.3 Object file

Object files work as an intermediate file format. If you want to compile more than two lisp files, you might better to compile with a :system-p t option, which generates object files (instead of a fasl).

On linux systems, ECL invokes gcc -c to generate object files.

An object file consists of some functions in C:

• Functions corresponding to Lisp functions
• The initialization function which registers defined functions on the lisp environment

Consider the example below.

(defun say-hello ()
  (print "Hello, world"))

During compilation, this simple lisp program is translated into the C program, and then compiled into the object file. The C program contains two functions:

• static cl_object L1say_hello: ’say-hello’ function
• ECL_DLLEXPORT void _eclwm2nNauJEfEnD_CLSxi0z(cl_object flag): initialization function

In order to use these object files from your C program, you have to call initialization functions before using lisp functions (such as say-hello). However the name of an init function is seemed to be randomized and not user-friendly. This is because object files are not intended to be used directly.

ECL provides other user-friendly ways to generate compiled lisp programs (as static/shared libraries or executables), and in each approach, object files act as intermediate files.

3.1.1.4 Static library

ECL can compile lisp programs to static libraries, which can be linked with C programs. A static library is created by c:build-static-library with some compiled object files.

;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates libhello-goodbye.a
(c:build-static-library "hello-goodbye"
                        :lisp-files '("hello.o" "goodbye.o")
                        :init-name "init_hello_goodbye")

When you use a static/shared library, you have to call its init function. The name of this function is specified by the :init-name option. In this example, it is then init_hello_goodbye. The usage of this function is shown below:

#include <ecl/ecl.h>
extern void init_hello_goodbye(cl_object cblock);

int
main(int argc, char **argv)
{
    /* setup the lisp runtime */
    cl_boot(argc, argv);

    /* call the init function via ecl_init_module */
    ecl_init_module(NULL, init_hello_goodbye);

    /* ... */

    /* shutdown the lisp runtime */
    cl_shutdown();

    return 0;
}

Because the program itself does not know the type of the init function, a prototype declaration is inserted. After booting up the lisp environment, it invokes init_hello_goodbye via ecl_init_module. init_hello_goodbye takes an argument, and ecl_init_module supplies an appropriate one. Now that the initialization is finished, we can use functions and other stuff defined in the library.

DEPRECATED read_VV - equivalent to ecl_init_module

3.1.1.5 Shared library

Almost the same as with a static library. The user has to use c:build-shared-library:

;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates libhello-goodbye.so
(c:build-shared-library "hello-goodbye"
                        :lisp-files '("hello.o" "goodbye.o")
                        :init-name "init_hello_goodbye")

3.1.1.6 Executable

ECL supports the generation of executable files. To create a standalone executable from a lisp program, compile all lisp files to object files. After that, calling c:build-program creates the executable:

;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates hello-goodbye
(c:build-program "hello-goodbye"
                 :lisp-files '("hello.o" "goodbye.o"))

Like with native FASL, the program may be built also from libraries.

3.1.1.7 Summary

In this section, some file types that can be compiled with ECL were introduced. Each file type has an adequate purpose:

• Object file: intermediate file format for others
• Fasl files: loaded dynamically via the load lisp function
• Static library: linked with and used from C programs
• Shared library: loaded dynamically and used from C programs
• Executable: standalone executable

ECL provides a high-level interface c:build-* for each native format. In case of Portable FASL the bytecodes compiler is needed.

3.1.2 Compiling with ASDF

First, let’s disregard the simple situation in which we write Lisp without depending on any other Lisp libraries. A more practical example is to build a library that depends on other asdf, systems. ECL provides a useful extension for asdf called asdf:make-build, which offers an abstraction for building libraries directly from system definitions.

To download dependencies you may use Quicklisp to load your system (with dependencies defined). Make sure you can successfully load and run your library in the ECL REPL (or *slime-repl*). Don’t worry about other libraries loaded in your image – ECL will only build and pack libraries your project depends on (that is, all dependencies you put in your .asd file, and their dependencies - nothing more, despite the fact that other libraries may be loaded).

3.1.2.1 Example code to build

We use a simple project that depends on alexandria to demonstrate the interface. The example consists of example-with-dep.asd, package.lisp and example.lisp (included in the examples/asdf_with_dependence/ directory in the ECL source tree). Before any kind of build you need to push the full path of this directory to asdf:*central-registry* (or link it in a location already recognized by ASDF).

3.1.2.2 Build it as an single executable

Use this in the REPL to make an executable:

(asdf:make-build :example-with-dep
                 :type :program
                 :move-here #P"./"
                 :epilogue-code '(progn (example:test-function 5)
                                        (si:exit)))

Here the :epilogue-code is executed after loading our library; we can use arbitrary Lisp forms here. You can also put this code in your Lisp files and directly build them without this :epilogue-code option to achieve the same result. Running the program in a console will display the following and exit:

Factorial of 5 is: 120

3.1.2.3 Build it as shared library and use in C

Use this in the REPL to make a shared library:

(asdf:make-build :example-with-dep
                 :type :shared-library
                 :move-here #P"./"
                 :monolithic t
                 :init-name "init_example")

Here :monolithic t means that ECL will compile the library and all its dependencies into a single library named example-with-dep--all-systems.so. The :move-here parameter is self-explanatory. :init-name sets the name of the initialization function. Each library linked from C/C++ code must be initialized, and this is a mechanism to specify the initialization function’s name.

To use it, we write a simple C program:

/* test.c */
#include <ecl/ecl.h>
extern void init_dll_example(cl_object);

int main (int argc, char **argv) {
  
  cl_boot(argc, argv);
  ecl_init_module(NULL, init_dll_example);

  /* do things with the Lisp library */
  cl_eval(c_string_to_object("(example:test-function 5)"));

  cl_shutdown();
  return 0;
}

Compile the file using a standard C compiler (note we’re linking to libecl.so with -lecl, which provides the lisp runtime²):

gcc test.c example-with-dep--all-systems.so -o test -lecl

If ECL is installed in a non-standard location you may need to provide flags for the compiler and the linker. You may read them with:

ecl-config --cflags
ecl-config --libs

Since our shared object is not in the standard location, you need to provide LD_LIBRARY_PATH pointing to the current directory to run the application:

LD_LIBRARY_PATH=`pwd` ./test

This will show:

Factorial of 5 is: 120

You can also build all dependent libraries separately as a few .so files and link them together. For example, if you are building a library called complex-example, that depends on alexandria and cl-fad, you can do the following (in the REPL):

(asdf:make-build :complex-example
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_example")

(asdf:make-build :alexandria
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_alexandria")

(asdf:make-build :cl-fad
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_fad")

(asdf:make-build :bordeaux-threads
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_bt")

Note that we haven’t specified :monolithic t, so we need to build bordeaux-threads as well because cl-fad depends on it. The building sequence doesn’t matter and the resultant .so files can also be used in your future programs if these libraries are not modified.

We need to initialize all these modules using ecl_init_module in the correct order. (bordeaux-threads must be initialized before cl-fad; cl-fad and alexandria must be initialized before complex-ecample.)

Here is a code snippet (not a full program):

extern void init_fad(cl_object);
extern void init_alexandria(cl_object);
extern void init_bt(cl_object);
extern void init_example(cl_object);

/* call these *after* cl_boot(argc, argv); 
   if B depends on A, you should first init A then B. */
ecl_init_module(NULL, init_bt);
ecl_init_module(NULL, init_fad);
ecl_init_module(NULL, init_alexandria);
ecl_init_module(NULL, init_example);

3.1.2.4 Build it as static library and use in C

To build a static library, use:

(asdf:make-build :example-with-dep
                 :type :static-library
                 :move-here #P"./"
                 :monolithic t
                 :init-name "init_example")

This will generate example-with-dep--all-systems.a in the current directory which we need to initialize with the init_example function. Compile it using:

gcc test.c example-with-dep--all-systems.a -o test-static -lecl

Then run it:

./test-static

This will show:

Factorial of 5 is: 120

Note we don’t need to pass the current path in LD_LIBRARY_PATH here, since our Lisp library is statically bundled with the executable. The result is the same as the shared library example above. You can also build all dependent libraries separately as static libraries.

3.1.3 C compiler configuration

ECL provides some global variables to customize which C compiler and compiler options to use:

3.1.3.1 Compiler flags

It is not required to surround the compiler flags with quotes or use slashes before special characters.

Variable: string c:*user-cc-flags*

Flags and options to be passed to the C compiler when building FASL, shared libraries and standalone programs.

Variable: string c:*user-ld-flags*

Flags and options to be passed to the linker when building FASL, shared libraries and standalone programs.

Variable: string c:*cc-optimize*

Optimize options to be passed to the C compiler.

3.1.3.2 Compiler & Linker programs

Variable: string c::*cc*

This variable controls how the C compiler is invoked by ECL. One can set the variable appropriately adding for instance flags which the C compiler may need to exploit special hardware features (e.g. a floating point coprocessor).

Variable: string c::*ld*

This variable controls the linker which is used by ECL.

Variable: string c::*ranlib*

Name of the ‘ranlib’ program on the hosting platform.

Variable: string c::*ar*

Name of the ‘ar’ program on the hosting platform.

Variable: string c::*ecl-include-directory*

Directory where the ECL header files for the target platform are located.

Variable: string c::*ecl-library-directory*

Directory where the ECL library files for the target platform are located.