A single package Common Lisp workflow for Medley
My exploration of Medley as a Common Lisp development environment proceeds with setting up a workflow for writing and saving code.
The workflow consists of a series of steps in a specific order using appropriate Lisp REPLs and tools. It supports writing the simplest type of Common Lisp software, i.e. programs or libraries in a single package that exports some symbols. I'll eventually extend the workflow to more complex cases such as programs with more than one package.
Since the steps are not intuitive, especially for Medley novices, in this post I'll describe the workflow in detail. But first there are a few concepts to introduce.
Why do you need a workflow in the first place? Because the differences between Medley and modern environments constrain how and in what order Common Lisp code may be written and managed. Before examining the constraints let's describe the differences.
The residential environment of Medley
Writing single package programs is straightforward in modern file based Common Lisp environments. In a new file you just define the package with DEFPACKAGE, then in the rest of the file or at the top of a new one you place a matching IN-PACKAGE followed by the code.
Although it's technically possible to do the same in Medley this doesn't take advantage of its facilities, and you may actually need to fight the system to accomplish what you want. Indeed, Medley is not an ordinary environment. Not only it predates current Common Lisp implementations, it supports a different development process.
In file based Common Lisps you directly edit source files and evaluate or load the code into the running Lisp image.
Medley instead is a “residential environment” in which you edit and evaluate Lisp objects that reside in the image — hence “residential”. Then you save the code to files that are more like code databases than traditional source files.
You don't edit the code databases, which Medley calls “symbolic files”. Rather, you use the SEdit structure editor to modify the code in memory and the “File Manager” to save the code to disk. The beginning of a symbolic file defines metadata, the “file environment”, which describes the Common Lisp package and readtable associated with the code.
The File Manager, also known as “File Package” (not to be confused with Common Lisp packages), is a facility that coordinates the development tools and code management tasks. It notices the changes to Lisp objects edited with SEdit or manipulated in memory, tracks what changed functions and objects need to be saved to symbolic files, and carries out the actions for building programs such as compiling or listing them. The File Manager has some of the functionality of Unix Make.
Figuring what changed is easy in modern Common Lisp environments, as you know which files you edited and need action like saving or compiling. System definition tools like ASDF can track this for you. In Medley it's the File Manager which tracks the changes that take place in the running image and need to be synchronized to disk.
Motivation
How do the peculiarities of Medley constrain writing Common Lisp code and require a tailored workflow?
The functions and objects of Interlisp programs usually live in the same namespace of Interlisp and its tools. As a consequence, functions and Lisp objects may be mostly defined in any order and accessed without package qualifiers. No special handling is necessary with the File Manager either.
With Common Lisp code, however, a subtle complication arises due to packages and exported symbols.
A good explanation of why things are different, and how the File Manager and file environment interact, is in the documentation of TextModules, a tool for importing Common Lisp code created outside of Medley. Although in the context of TextModules, these remarks give an overview of the same issues the File Manager faces with other Common Lisp code. The TextModules chapter of the Lisp Library Modules manual says on page 311 (page 341 of the PDF):
It is important to separate the environment of the file from its contents because the File Manager (not TextModules) first reads all the forms in the file, and then evaluates them. Text based source files sometimes change the package as needed. This cannot work for the File Manager since the file's forms are all read and then executed, i.e. the package changes would not occur until after the entire file had been read, and forms after any IN-PACKAGE form would have been read incorrectly.
In other words, unless you define packages and access symbols in the proper order, you'll get subtle errors. The solution is a workflow that avoids such errors.
More information on dealing with packages in Medley is in the sources referenced in section “Documentation” of my post on using Common Lisp on Medley.
The workflow
How does the workflow order the development tasks to achieve its goal?
At any one time the workflow accesses only defined symbols and ensures the running Lisp image stays synchronized with the symbolic file. It's not the only or the best possible workflow, just one that works. I put it together after extensively reading the documentation and experimenting.
As I said in the overview of Medley as a Common Lisp environment, when coding in Common Lisp I use two Executives (Lisp REPLs), a Xerox Common Lisp (XCL) Exec and an Interlisp one.
The former is for testing, running, and evaluating Common Lisp code. I use the Interlisp Exec for running system tools and interacting with the File Manager. Since the tools and File Manager facilities are in the Interlisp package, referencing them from a Common Lisp Exec would require qualifying all symbols with the IL: package.
What follows assume you're familiar with basic Interlisp and File Manager features such as SEdit, file coms, FILES?, and MAKEFILE. If not I recommend reading the Medley primer, particulary Chapter 7 “Editing and Saving”. Also, unless otherwise noted, you should carry out the steps in sequence in the same Medley session.
Let's start.
Defining the file environment and a minimal package
Suppose you want to write a Common Lisp program or library stored in the file SINGLEPKG. The package SINGLEPKG, nicknamed SP, will export the two functions FUN1 and FUN2.
From now on, ← denotes the prompt of an Interlisp Exec and > that of a Xerox Common Lisp (XCL) Exec. Sometimes I'll tell you in which Exec to evaluate expressions.
The first step is to define the file environment. At an Interlisp Exec evaluate:
← (XCL:DEFINE-FILE-ENVIRONMENT SINGLEPKG :PACKAGE (DEFPACKAGE "SINGLEPKG" (:USE "LISP" "XCL")) :READTABLE "XCL")
For now don't use other packages or export any symbols, just enter the form as is.
Although the file environment references package SINGLEPKG, the package doesn't exist yet in the running image. To synchronize the image with the file environment evaluate the definition of a minimal package from an Interlisp Exec:
← (DEFPACKAGE "SINGLEPKG" (:USE "LISP" "XCL"))
Next, from an Interlisp Exec evaluate (FILES?) and, when asked where the SINGLEPKG file info should go, respond yes, enter SINGLEPKG as the file name, and confirm the creation of the file.
Defining the first function
Everything is ready to define the first function FUN1. At an Interlisp Exec call SEdit with (ED 'SINGLEPKG::FUN1 '(FUNCTIONS :DONTWAIT)) and select DEFUN from the menu. Enter the code of FUN1:
(DEFUN FUN1 ()
(FORMAT T "Hello from FUN1."))
Save and exit with Ctrl-Meta-X and test the function at an XCL Exec (an Interlisp Exec will do too):
> (SINGLEPKG::FUN1)
Hello from FUN1.
NIL
It works, so at an Interlisp Exec evaluate (FILES?) to associate FUN1 with the file SINGLEPKG.
Completing the package definition
The Lisp image contains the new symbol FUN1 in package SINGLEPKG but there's no symbolic file yet, let alone an exported symbol in the file. Therefore, to keep things in sync you need to update the package definition by exporting the function name. This is also an opportunity for adding the SP nickname to the package.
At an Interlisp Exec evaluate (DC SINGLEPKG) to open the file coms in SEdit. Just after the XCL:FILE-ENVIRONMENTS form enter:
(P (DEFPACKAGE "SINGLEPKG"
(:USE "LISP" "XCL")
(:NICKNAMES "SP")
(:EXPORT SINGLEPKG::FUN1)))
Both colon characters : are required in the function name. The P File Manager command tells the system to execute the following Lisp expressions at load time, so loading SINGLEPKG will define the package and export the symbol.
Save and exit with Ctrl-Meta-X and, at an Interlisp Exec, save the file with (MAKEFILE 'SINGLEPKG) to reflect the updated definitions. This creates the file SINGLEPKG on disk.
Before doing anything else it's better to make sure the package is properly defined and the function exported.
The most reliable way is to exit the Medley session with (IL:LOGOUT), start a new session and, from an Interlisp Exec, evaluate (LOAD 'SINGLEPKG). If there are errors you will have to go back and check whether you went through all the steps correctly. You may need to delete the latest or all versions of the file SINGLEPKG.
Assuming there are no errors the package is properly defined and the function exported. To double check, from an XCL Exec call the exported function like this:
> (SINGLEPKG:FUN1)
Hello from FUN1.
NIL
Since it works you may proceed development by editing the existing function or defining new ones. In the former case you employ SEdit, FILES?, and MAKEFILE as usual. Just make sure to pass to ED the package-qualified function name like (ED 'SINGLEPKG:FUN1 :DONTWAIT).
Things change if you want to define new functions.
Defining more functions
As planned you proceed to define a second function FUN2 exported from package SINGLEPKG. If you're still in the Medley session in which you've just loaded the file SINGLEPKG, continue from there. Otherwise start a new session and load the file with (LOAD 'SINGLEPKG) from any Exec.
At an Interlisp Exec define the new function with (ED 'SINGLEPKG::FUN2 '(FUNCTIONS :DONTWAIT)) and select DEFUN from the menu:
(DEFUN FUN2 ()
(FORMAT T "Hello from FUN2."))
Save and exit with Ctrl-Meta-X and test the function at an XCL Exec:
> (SINGLEPKG::FUN2)
Hello from FUN2.
NIL
Call FILES? to associated FUN2 with file SINGLEPKG.
As already done for FUN1 you need to modify the package definition to export the new function. At an Interlisp Exec edit the file coms with (DC SINGLEPKG) and add the symbol FUN2 to the export clause, which will now look like this:
(P (DEFPACKAGE "SINGLEPKG"
(:USE "LISP" "XCL")
(:NICKNAMES "SP")
(:EXPORT SINGLEPKG:FUN1 SINGLEPKG::FUN2)))
SEdit replaced the double colon of SINGLEPKG::FUN1 with a single colon as in SINGLEPKG:FUN1. But you still have to type :: for FUN2 because FUN2 hasn't been exported yet.
Save and exit with Ctrl-Meta-X and, at an Interlisp Exec, save the file with (MAKEFILE 'SINGLEPKG).
To synchronize the Lisp image with the symbolic file, which will allow to call the exported function as (SINGLEPKG:FUN2), either load the file SINGLEPKG from a fresh session or, just after editing the file coms, evaluate a revised package definition at an Interlisp Exec:
← (DEFPACKAGE "SINGLEPKG"
(:USE "LISP" "XCL")
(:NICKNAMES "SP")
(:EXPORT SINGLEPKG::FUN1 SINGLEPKG::FUN2))
Either way, to check that everything works evaluate at an XCL Exec:
> (SINGLEPKG:FUN2)
Hello from FUN2.
NIL
Success!
For every new function or Lisp object you want to export from the package, go through the steps of this section again, making sure the Lisp image and the symbolic file stay synchronized. The steps are, in order:
- edit the new function
- call
FILES?to tell the File Manager about the function - edit the file coms to update the package
- save the file with
MAKEFILE - evaluate the revised package definition
That's all, you're finally ready to develop single package programs. This workflow may seem convoluted at first but things will come more natural as you gain experience with Medley.
Discuss... Email | Reply @amoroso@fosstodon.org