Importing Common Lisp Files in Medley with TextModules

February 18, 2024

Medley is a residential environment for Interlisp and Common Lisp development.

With some effort it's possible to use Medley as a traditional file based Common Lisp environment. But in specific cases a better approach is to bring in Medley's residential environment Common Lisp sources created in file based environments.

In this post I explain the latter, i.e. how to use TextModules to import Common Lisp files into the residential environment. I go over the steps for converting an example program, the database of CD music tracks in Chapter 3 Practical: A Simple Database of Peter Seibel's book Practical Common Lisp.

Motivation

Using the tools and facilities of the residential environment, such as the File Manager, is the normal way of developing new Lisp programs. To run existing Common Lisp code you don't plan to change often, you can also use Medley as a traditional file based environment.

For existing code written in file based environments you want to use and further develop in Medley, a better option is to import the code into the residential environment and continue working from there. This is what TextModules helps to do.

What is TextModules

TextModules is a Medley tool for bringing Common Lisp sources into the residential environment and place them under the control of the File Manager. It can also do the reverse, i.e. export the File Manager descriptions and metadata to Common Lisp sources accessible from file based environments.

Using TextModules is a one time process. You run the tool once to import the code, then use and modify it with the tools and facilities of the residential environment.

The documentation of textModules starts from page 305 (page 335 of the PDF) of the Lisp Library Modules manual.

Preparing the Common Lisp files

This example involves two of the source files of Seibel's book, packages.lisp and simple-database.lisp in directory practicals-1.0.3/Chapter03 of the code archive.

Unlike CL:LOAD on Medley, TextModules doesn't require any special formatting of Common Lisp source files. For example, they don't need to begin with a semicolon character.

However, the Common Lisp implementation of Medley is incomplete and not ANSI compliant, so be sure to remove or adapt any unsupported forms. This is the case of the database example: packages.lisp makes current the package CL-USER which is missing from Medley. The fix is to substitute xcl-user for cl-user in the file, as XCL-USER is the Medley equivalent of CL-USER.

Another source of incompatibility is the LOOP macro. In Medley it's only a stub that runs an infinite loop no matter what clauses a call specifies.

To import with TextModules code that contains LOOP it would normally be necessary to replace any calls with equivalent expressions. Since this post focuses on TextModules I just use the LOOP calls intended to run an infinite loop, and ignore the others.

Running TextModules

As noted, importing with TextModules is the one time process of running the tool for every source file. Once in the residential environment, you save and manipulate the code as any other code under the File Manager.

First off, load TextModules by evaluating (FILESLOAD TEXTMODULES) at an Interlisp Exec. All its exported symbols are in package TM. Next, call the function TM:LOAD-TEXTMODULE for every Common Lisp file, which is similar to CL:LOAD with some additional processing.

Most Common Lisp programs comprise a file packages.lisp with package definitions, and a number of additional .lisp files that contain the bulk of the code. This dependency requires passing the files to TM:LOAD-TEXTMODULE in the proper order.

The file packages.lisp of the database defines the package for simple-database.lisp, so start with the former. At a Xerox Common Lisp (XCL) Exec with prompt > evaluate:

> (tm:load-textmodule "packages.lisp" :module "SIMPLEDB" :package (find-package "XCL-USER") :install t)
IL:SIMPLEDB

The only required argument is the input file packages.lisp. However, by default TM:LOAD-TEXTMODULE uses the same input file name as the name of the program for the File Manager. It wouldn't make much sense to call a database PACKAGES.LISP. A better choice is to pass the :module parameter with the more descriptive name SIMPLEDB.

The code in packages.lisp begins with an in-package form. To make sure the in-package symbol is accessible without qualifier, it should be read in a package such as XCL-USER that imports the standard Common Lisp symbols. Hence the argument :package (find-package "XCL-USER") in the call.

The argument :install t installs the definitions in the running system. Although not strictly necessary, it's useful for diagnostic purposes and because you likely want to continue working on the imported code.

Next, process simple-database.lisp by evaluating at a XCL Exec:

> (tm:load-textmodule "simple-database.lisp" :module "SIMPLEDB" :package (find-package "XCL-USER") :install t)
IL:SIMPLEDB

Again, the file begins with in-package and the reason for passing the :package argument is the same as for packages.lisp.

Saving the imported code

At this point the imported code is in the running Lisp image and the File Manager is ready to manipulate it. You can check the File Manager noticed the imported definitions by calling FILES? at an Interlisp Exec with prompt ←:

← (FILES?)
To be dumped:
SIMPLEDB ...changes to VARS: SIMPLEDBCOMS
                       VARIABLES: COM.GIGAMONKEYS.SIMPLE-DB::*DB*
                       FUNCTIONS: COM.GIGAMONKEYS.SIMPLE-DB::MAKE-CD, 
         COM.GIGAMONKEYS.SIMPLE-DB::ADD-RECORD, 
         COM.GIGAMONKEYS.SIMPLE-DB::DUMP-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::PROMPT-READ, 
         COM.GIGAMONKEYS.SIMPLE-DB::PROMPT-FOR-CD, 
         COM.GIGAMONKEYS.SIMPLE-DB::ADD-CDS, 
         COM.GIGAMONKEYS.SIMPLE-DB::SAVE-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::LOAD-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::CLEAR-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::SELECT, 
         COM.GIGAMONKEYS.SIMPLE-DB::WHERE, 
         COM.GIGAMONKEYS.SIMPLE-DB::MAKE-COMPARISONS-LIST, 
         COM.GIGAMONKEYS.SIMPLE-DB::MAKE-COMPARISON-EXPR, 
         COM.GIGAMONKEYS.SIMPLE-DB::UPDATE, 
         COM.GIGAMONKEYS.SIMPLE-DB::DELETE-ROWS

To save the definitions to the symbolic file SIMPLEDB call MAKEFILE from an Interlisp Exec:

← (MAKEFILE 'SIMPLEDB)
{DSK}<home>medley>il>SIMPLEDB.;1

Calling TM:LOAD-TEXTMODULE for every source file, and saving the result to a symbolic file with MAKEFILE, completes the import process.

You may terminate the session and resume later. When you're ready to proceed you can load, run, and modify the imported program as any other code under File Manager control.

Loading and running the imported code

In a new Medley session evaluate (FILESLOAD TEXTMODULES) at an Interlisp Exec. The tool must be in memory whenever you work with imported code, as TextModules sets up a special file environment and readtable the code needs to be read in.

Next, load the symbolic file of the database program by evaluating at a XCL Exec:

> (load "SIMPLEDB")

; Loading {DSK}<home>medley>il>SIMPLEDB.;1
; File created 14-Feb-2024 02:44:46
; IL:SIMPLEDBCOMS
IL:|{DSK}<home>medley>il>SIMPLEDB.;1|

One of the main entry points of the program is the function add-cds to add new records to the database, one record for each music track of a CD. A typical run from a XCL Exec looks like this:

> (setf *package* (find-package "COM.GIGAMONKEYS.SIMPLE-DB"))
#<Package COM.GIGAMONKEYS.SIMPLE-DB>
> (add-cds)
Title: Punch My Cards
Artist: The Fortrans
Rating: 6
Ripped [y/n]: n
Another? [y/n]: y
Title: Lisp n Roll
Artist: The Garbage Collectors
Rating: 8
Ripped [y/n]: y
Another? [y/n]: y
Title: Cdr Care Less
Artist: The Garbage Collectors
Rating: 7
Ripped [y/n]: y
Another? [y/n]: n
NIL

Since all the symbols of the program are in the package COM.GIGAMONKEYS.SIMPLE-DB, and none are exported, for convenience make the package current by setfing *package* as above.

I intentionally didn't rename the package or create nicknames. This is to show that Common Lisp code may be imported and used with minimal or no changes.

The program provides select and where to query the database. But where uses CL:LOOP features Medley doesn't support. To stay close to the original code, instead of modifying where you can use another function in Seibel's book. Define and call the specialized query function select-by-artist to search the database by artist:

> (defun select-by-artist (artist)
    (remove-if-not
     #'(lambda (cd) (equal (getf cd :artist) artist))
     *db*))
SELECT-BY-ARTIST
> (select-by-artist "The Garbage Collectors")
((:TITLE "Cdr Care Less" :ARTIST "The Garbage Collectors" :RATING 7 :RIPPED T) (:TITLE "Lisp n Roll" :ARTIST "The Garbage Collectors" :RATING 8 :RIPPED T))

Continuing the development

The imported code works. Now you can load, compile, run, edit, and save it as any other program developed in the residential environment under the File Manager.

You no longer need the original files packages.lisp and simple-database.lisp because you work only with SIMPLEDB. But remember to load TextModules with (FILESLOAD TEXTMODULES) in every session in which you use SIMPLEDB. It's a minor inconvenience but, for automating the task, you may add a loading command to the INIT initialization file or to a script that loads the program.

#CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org