Paolo Amoroso's Journal

CommonLisp

My exploration of Medley as a Common Lisp development environment proceeds with setting up a workflow for writing and saving code.

The workflow consists of a series of steps in a specific order using appropriate Lisp REPLs and tools. It supports writing the simplest type of Common Lisp software, i.e. programs or libraries in a single package that exports some symbols. I'll eventually extend the workflow to more complex cases such as programs with more than one package.

Since the steps are not intuitive, especially for Medley novices, in this post I'll describe the workflow in detail. But first there are a few concepts to introduce.

Why do you need a workflow in the first place? Because the differences between Medley and modern environments constrain how and in what order Common Lisp code may be written and managed. Before examining the constraints let's describe the differences.

The residential environment of Medley

Writing single package programs is straightforward in modern file based Common Lisp environments. In a new file you just define the package with DEFPACKAGE, then in the rest of the file or at the top of a new one you place a matching IN-PACKAGE followed by the code.

Although it's technically possible to do the same in Medley this doesn't take advantage of its facilities, and you may actually need to fight the system to accomplish what you want. Indeed, Medley is not an ordinary environment. Not only it predates current Common Lisp implementations, it supports a different development process.

In file based Common Lisps you directly edit source files and evaluate or load the code into the running Lisp image.

Medley instead is a “residential environment” in which you edit and evaluate Lisp objects that reside in the image — hence “residential”. Then you save the code to files that are more like code databases than traditional source files.

You don't edit the code databases, which Medley calls “symbolic files”. Rather, you use the SEdit structure editor to modify the code in memory and the “File Manager” to save the code to disk. The beginning of a symbolic file defines metadata, the “file environment”, which describes the Common Lisp package and readtable associated with the code.

The File Manager, also known as “File Package” (not to be confused with Common Lisp packages), is a facility that coordinates the development tools and code management tasks. It notices the changes to Lisp objects edited with SEdit or manipulated in memory, tracks what changed functions and objects need to be saved to symbolic files, and carries out the actions for building programs such as compiling or listing them. The File Manager has some of the functionality of Unix Make.

Figuring what changed is easy in modern Common Lisp environments, as you know which files you edited and need action like saving or compiling. System definition tools like ASDF can track this for you. In Medley it's the File Manager which tracks the changes that take place in the running image and need to be synchronized to disk.

Motivations

How do the peculiarities of Medley constrain writing Common Lisp code and require a tailored workflow?

The functions and objects of Interlisp programs usually live in the same namespace of Interlisp and its tools. As a consequence, functions and Lisp objects may be mostly defined in any order and accessed without package qualifiers. No special handling is necessary with the File Manager either.

With Common Lisp code, however, a subtle complication arises due to packages and exported symbols.

A good explanation of why things are different, and how the File Manager and file environment interact, is in the documentation of TextModules, a tool for importing Common Lisp code created outside of Medley. Although in the context of TextModules, these remarks give an overview of the same issues the File Manager faces with other Common Lisp code. The TextModules chapter of the Lisp Library Modules manual says on page 311 (page 341 of the PDF):

It is important to separate the environment of the file from its contents because the File Manager (not TextModules) first reads all the forms in the file, and then evaluates them. Text based source files sometimes change the package as needed. This cannot work for the File Manager since the file's forms are all read and then executed, i.e. the package changes would not occur until after the entire file had been read, and forms after any IN-PACKAGE form would have been read incorrectly.

In other words, unless you define packages and access symbols in the proper order, you'll get subtle errors. The solution is a workflow that avoids such errors.

More information on dealing with packages in Medley is in the sources referenced in section “Documentation” of my post on using Common Lisp on Medley.

The workflow

How does the workflow order the development tasks to achieve its goal?

At any one time the workflow accesses only defined symbols and ensures the running Lisp image stays synchronized with the symbolic file. It's not the only or the best possible workflow, just one that works. I put it together after extensively reading the documentation and experimenting.

As I said in the overview of Medley as a Common Lisp environment, when coding in Common Lisp I use two Executives (Lisp REPLs), a Xerox Common Lisp (XCL) Exec and an Interlisp one.

The former is for testing, running, and evaluating Common Lisp code. I use the Interlisp Exec for running system tools and interacting with the File Manager. Since the tools and File Manager facilities are in the Interlisp package, referencing them from a Common Lisp Exec would require qualifying all symbols with the IL: package.

What follows assume you're familiar with basic Interlisp and File Manager features such as SEdit, file coms, FILES?, and MAKEFILE. If not I recommend reading the Medley primer, particulary Chapter 7 “Editing and Saving”. Also, unless otherwise noted, you should carry out the steps in sequence in the same Medley session.

Let's start.

Defining the file environment and a minimal package

Suppose you want to write a Common Lisp program or library stored in the file SINGLEPKG. The package SINGLEPKG, nicknamed SP, will export the two functions FUN1 and FUN2.

From now on, denotes the prompt of an Interlisp Exec and > that of a Xerox Common Lisp (XCL) Exec. Sometimes I'll tell you in which Exec to evaluate expressions.

The first step is to define the file environment. At an Interlisp Exec evaluate:

← (XCL:DEFINE-FILE-ENVIRONMENT SINGLEPKG :PACKAGE (DEFPACKAGE "SINGLEPKG" (:USE "LISP" "XCL")) :READTABLE "XCL")

For now don't use other packages or export any symbols, just enter the form as is.

Although the file environment references package SINGLEPKG, the package doesn't exist yet in the running image. To synchronize the image with the file environment evaluate the definition of a minimal package from an XCL Exec:

← (DEFPACKAGE "SINGLEPKG" (:USE "LISP" "XCL"))

Next, from an Interlisp Exec evaluate (FILES?) and, when asked where the SINGLEPKG file info should go, respond yes, enter SINGLEPKG as the file name, and confirm the creation of the file.

Defining the first function

Everything is ready to define the first function FUN1. At an Interlisp Exec call SEdit with (ED 'SINGLEPKG::FUN1 '(FUNCTIONS :DONTWAIT)) and select DEFUN from the menu. Enter the code of FUN1:

(DEFUN FUN1 ()
  (FORMAT T "Hello from FUN1."))

Save and exit with Ctrl-Meta-X and test the function at an XCL Exec (an Interlisp Exec will do too):

> (SINGLEPKG::FUN1)
Hello from FUN1.
NIL

It works, so at an Interlisp Exec evaluate (FILES?) to associate FUN1 with the file SINGLEPKG.

Completing the package definition

The Lisp image contains the new symbol FUN1 in package SINGLEPKG but there's no symbolic file yet, let alone an exported symbol in the file. Therefore, to keep things in sync you need to update the package definition by exporting the function name. This is also an opportunity for adding the SP nickname to the package.

At an Interlisp Exec evaluate (DC SINGLEPKG) to open the file coms in SEdit. Just after the XCL:FILE-ENVIRONMENTS form enter:

(P (DEFPACKAGE "SINGLEPKG"
     (:USE "LISP" "XCL")
     (:NICKNAMES "SP")
     (:EXPORT SINGLEPKG::FUN1)))

Both colon characters : are required in the function name. The P File Manager command tells the system to execute the following Lisp expressions at load time, so loading SINGLEPKG will define the package and export the symbol.

Save and exit with Ctrl-Meta-X and, at an Interlisp Exec, save the file with (MAKEFILE 'SINGLEPKG) to reflect the updated definitions. This creates the file SINGLEPKG on disk.

Before doing anything else it's better to make sure the package is properly defined and the function exported.

The most reliable way is to exit the Medley session with (IL:LOGOUT T), start a new session and, from an Interlisp Exec, evaluate (LOAD 'SINGLEPKG). If there are errors you will have to go back and check whether you went through all the steps correctly. You may need to delete the latest or all versions of the file SINGLEPKG.

Assuming there are no errors the package is properly defined and the function exported. To double check, from an XCL Exec call the exported function like this:

> (SINGLEPKG:FUN1)
Hello from FUN1.
NIL

Since it works you may proceed development by editing the existing function or defining new ones. In the former case you employ SEdit, FILES?, and MAKEFILE as usual. Just make sure to pass to ED the package-qualified function name like (ED 'SINGLEPKG:FUN1 :DONTWAIT).

Things change if you want to define new functions.

Defining more functions

As planned you proceed to define a second function FUN2 exported from package SINGLEPKG. If you're still in the Medley session in which you've just loaded the file SINGLEPKG, continue from there. Otherwise start a new session and load the file with (LOAD 'SINGLEPKG) from any Exec.

At an Interlisp Exec define the new function with (ED 'SINGLEPKG::FUN2 '(FUNCTIONS :DONTWAIT)) and select DEFUN from the menu:

(DEFUN FUN2 ()
  (FORMAT T "Hello from FUN2."))

Save and exit with Ctrl-Meta-X and test the function at an XCL Exec:

> (SINGLEPKG::FUN2)
Hello from FUN2.
NIL

Call FILES? to associate FUN2 with file SINGLEPKG.

As already done for FUN1 you need to modify the package definition to export the new function. At an Interlisp Exec edit the file coms with (DC SINGLEPKG) and add the symbol FUN2 to the export clause, which will now look like this:

(P (DEFPACKAGE "SINGLEPKG"
     (:USE "LISP" "XCL")
     (:NICKNAMES "SP")
     (:EXPORT SINGLEPKG:FUN1 SINGLEPKG::FUN2)))

SEdit replaced the double colon of SINGLEPKG::FUN1 with a single colon as in SINGLEPKG:FUN1. But you still have to type :: for FUN2 because FUN2 hasn't been exported yet.

Save and exit with Ctrl-Meta-X and, at an Interlisp Exec, save the file with (MAKEFILE 'SINGLEPKG).

To synchronize the Lisp image with the symbolic file, which will allow to call the exported function as (SINGLEPKG:FUN2), either load the file SINGLEPKG from a fresh session or, just after editing the file coms, evaluate a revised package definition at an Interlisp Exec:

← (DEFPACKAGE "SINGLEPKG"
    (:USE "LISP" "XCL")
    (:NICKNAMES "SP")
    (:EXPORT SINGLEPKG::FUN1 SINGLEPKG::FUN2))

As an alternative export the FUN2 symbol from package SINGLEPKG:

← (EXPORT (FIND-SYMBOL "FUN2" (FIND-PACKAGE "SINGLEPKG")) (FIND-PACKAGE "SINGLEPKG"))

Either way, to check that everything works evaluate at an XCL Exec:

> (SINGLEPKG:FUN2)
Hello from FUN2.
NIL

Success!

For every new function or Lisp object you want to export from the package, go through the steps of this section again, making sure the Lisp image and the symbolic file stay synchronized. The steps are, in order:

  1. edit the new function
  2. call FILES? to tell the File Manager about the function
  3. edit the file coms to update the package
  4. save the file with MAKEFILE
  5. evaluate the revised package definition

That's all, you're finally ready to develop single package programs. This workflow may seem convoluted at first but things will come more natural as you gain experience with Medley.

#CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@oldbytes.space

Now that I'm back to Lisp I'm actively exploring Interlisp as a Common Lisp environment too.

But to code in Common Lisp also on my Crostini Linux system, the Linux container of chromeOS I use on a Chromebox, I'm setting up a suitable development environment. In addition to console programs I want to write GUI applications with McCLIM.

The Common Lisp implementation I chose, SBCL, is a no brainer given its features, performance, and active maintenance. As for the development environment I won't go with the default on Linux, Emacs. Although I used Emacs for years and loved it, now it feels overkill and I'd prefer not to re-learn its intricacies.

Instead I'm using Lem, a great Emacs-like IDE written in Common Lisp with a user interface and keybindings similar to those of Emacs and SLIME. Thus my familiarity with Emacs is enough to get me up to speed. A nice side effect of Lem's implementation language is the IDE can be configured and extended in Common Lisp, which I feel more at home with than Emacs Lisp.

Despite some initial installation issues Lem works well on Crostini, is fast, and has a nice SDL2 backend. I really like the IDE.

If I need to write or run some Common Lisp code on my Raspberry Pi 400 I can easily replicate this setup there.

#CommonLisp #Lisp

Discuss... Email | Reply @amoroso@oldbytes.space

Since encountering Medley I gained considerable experience with Interlisp. Medley Interlisp is a project for preserving, reviving, and modernizing the Interlisp-D software development environment of the Lisp Machines Xerox created at PARC.

Nine months later I know enough to find my way around and confidently use most of the major system tools and features.

I read all the available documentation, books, and publications, so I know where to look for information. And I undertook Interlisp programming projects such as Stringscope, Braincons, Sysrama, and Femtounit.

Now I'm ready to explore Medley as a Common Lisp development environment.

Although most of the system, facilities, and tools are written in and designed around Interlisp, the companies that maintained and marketed Medley over time partially implemented Common Lisp and integrated it with the environment. The completion level of the implementation is somewhere between CLtL1 and CLtL2, plus CLOS via Portable Common Loops (PCL).

Motivation

I want to widen this experience to Common Lisp.

I'll leverage the more advanced Lisp dialect and interface with Interlisp's facilities as an application platform that comprises a rich set of libraries and tools such a window system, graphics primitives, menu facilities, and GUI controls for building applications. Each world can interoperate with the other, so Common Lisp functions can call Interlisp ones and the other way around.

Developing Common Lisp programs with Medley is both my goal and a way of achieving it through practice. Medley is an ideal self-contained computing universe for my personal projects and Common Lisp greatly enchances its toolbox.

Tools

The main tools for developing Common Lisp code are the same as for Interlisp: the SEdit structure editor for writing code; the File Manager, a make-like tool for tracking changes to Lisp objects in the running image and saving them to files; and the Executive (or Exec), the Lisp listener.

However, the workflow is subtly different.

In some cases taking advantage of the integration with Medley involves different steps for Common Lisp code. For example, defining and changing packages so that the File Manager notices and tracks them needs to be done in a certain order. And there are Medley extensions to the package forms.

When working with Common Lisp I open at least two Execs, a Common Lisp and an Interlisp one. The former is for testing, running, and evaluating Common Lisp code.

The Interlisp Exec is for launching system tools and interacting with the File Manager. Since all the symbols of SEdit, the File Manager, and other system tools are in the IL Interlisp package, in an Interlisp Exec it's not necessary to add package qualifiers to symbols all the time.

Exec commands such as DIR and CD work the same in both Execs.

Documentation

Medley's Common Lisp features aren't documented in the Interlisp Reference Manual, the main information source about the system. The reason is the companies that distributed and maintained the product ceased operations before the work on implementing and documenting Common Lisp was completed.

I found only a couple of good sources on Common Lisp under Medley.

The implementation notes and the release notes of Lyric, the music-themed codename of one of Interlisp-D's versions, provide an overview of the integration between Common Lisp and Medley. The release notes of Medley 1.0, a later version, expand on this. Issue 5 of HOTLINE!, a newsletter Xerox published for its Lisp customers, has useful step by step examples of creating and managing Common Lisp packages the Medley way.

Some of the system code of Medley is written in Common Lisp and may be a source of usage examples and idioms. I'm also writing Common Lisp code snippets to test my understanding of the integration with Medley.

#CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@oldbytes.space