Paolo Amoroso's Journal

Lisp

I'm examining in depth NoteCards, the hypermedia environment of Medley Interlisp well described in the 1987 paper NoteCards in a Nutshell.

To experiment with the NoteCards API I set out to explore the space of possible answers to the question: what if Lisp programs were represented as NoteCards hypertexts? Leveraging NoteCards to play with the idea seemed like a natural extension to Sysrama, my Interlisp documentation tool for presenting information on the objects of Lisp programs.

The rich environment of NoteCards allowed me to quickly put together what I wanted as the system provides an extensive, well designed, well documented, and easy to use API.

Generating hypertexts with NoteCards and Sysrama

As a start I wrote a Sysrama function that produces a hypertext representation of a program by querying the File Manager. The function, appropriately named CODECARDS, creates a notefile and populates it with cards and fileboxes.

In NoteCards, a “card” is the unit of information the nodes of a hypertext network store and present. A card can hold text, media, and more. A “filebox” is a container of cards and fileboxes. The fileboxes and cards of a hypertext are stored on disk in a “notefile”.

CODECARDS, which is passed the file name of a program as the only argument, sets up a filebox for every non empty File Manager type of the program. It further creates one card for every Lisp object of the type, fills the card with information on the object, and files the card under the type's filebox. The fileboxes are filed under the Table of Contents filebox. A side effect of filing the cards and fileboxes is the creation of a series of system links to construct and navigate the hierarchical structure.

The generated cards and fileboxes can be manipulated with NoteCards' suite of hypertext editing and navigation tools such as the graph browser.

After evaluating (CODECARDS 'SYSRAMA) to process Sysrama's own code, opening some fileboxes and cards, and arranging them on the desktop the Medley screen may look like this:

Some fileboxes and cards of a NoteCards hypertext on Medley Interlisp.

The Table of Context filebox in the top row of windows, between the NoteCards icon and the Medley logo, contains links to the fileboxes of the File Manager types. The links are the rectangular outlines with an icon at the left and a textual description in the rest of the outline.

Clicking on the VARS (variables), FNS (functions), and EXPRESSIONS (expressions) links of the Table of Contents opens the corresponding fileboxes of the middle row of windows. Section FILE BOXES has no entries because the fileboxes contain only cards.

The leftmost in the bottom row of three windows is the filebox of the type FNS (functions). Every one of its links leads to the card of the Lisp function indicated by the link name. The two rightmost windows of the bottom row are the cards of functions CODECARDS and DESCRIBEFNS with the shown signatures ((CODECARDS FILE) and (DESCRIBEFNS FNS TYPE)) and type (EXPR LAMBDA-spread).

The window at the bottom left of the screen is a browser, a NoteCards editing and navigation tool for visualizing and manipulating a hypertext as a graph. The trees in the browser represent the hierarchical structures of fileboxes and cards CODECARDS produces.

Note the VARS, FNS, and FILEVARS fileboxes at the roots of the trees in the browser. The leaves are cards.

Sysrama and the File Manager

The Medley File Manager should not be confused with the similarly named tool of desktop operating systems for navigating and manipulating files and folders. As I wrote in a post on using Common Lisp in the residential environment:

The File Manager, also known as “File Package” (not to be confused with Common Lisp packages), is a facility that coordinates the development tools and code management tasks. It notices the changes to Lisp objects edited with SEdit or manipulated in memory, tracks what changed functions and objects need to be saved to symbolic files, and carries out the actions for building programs such as compiling or listing them. The File Manager has some of the functionality of Unix Make.

The File Manager stores Lisp sources in “symbolic files”, which are code and metadata databases rather than ordinary source files.

Sysrama and CODECARDS query the File Manager metadata associated with a program to present information about its Lisp objects, such as functions and variables, and build a notefile.

Next steps

So far CODECARDS doesn't do much as it's an early proof of concept to explore the idea of representing programs as hypertexts. The evolution of the feature will point to where the idea can lead.

The hypertext functionality of Sysrama relies only on system links and doesn't define user links, i.e. user defined connections that establish typed relationships between the end points. Also, I'm thinking about whether and how to update the notefile with the changes the associated program undergoes during development.

The next tasks I'll likely work on are fleshing out the information in the Lisp object cards and defining some user links.

#Interlisp #sysrama #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

Although online access to Medley Interlisp is convenient and prebuilt binaries are available for all major operating system, it's also possible to compile it from source.

I first built Medley from source when I wanted to run it on my Raspberry Pi 400 and no ARM64 binaries were available. I later built Medley on Linux to try other options such as SDL2 support instead of X11. Since the build process is simple and fast, even on low-end devices such as the Pi 400, now I often do it on Linux.

There are some compilation instructions and notes for Maiko and Medley (see also the scripts README) but not a full build guide, so I fleshed out the details with the help of the project team.

I initially didn't write down the process as I relied on memory, which sometimes made me take wrong turns and waste time. This post finally puts together all the required steps and information for my own reference, and I hope it will be useful to you too.

Building Medley involves downloading the sources, configuring the environment, compiling the Maiko Virtual Machine, and generating the Medley system image.

What to build

Medley provides a number of compilation options, such as SDL2 support as an alternative to X11, with configurations that range from a minimal system to a complete build that includes all the Lisp applications like NoteCards.

These instructions cover building the complete system, which Medley veterans frequently refer to as the “apps sysout” as a sysout is a Lisp image.

Assumptions

The process I describe assumes you'll build the system on Linux using Clang. If you prefer Gcc the Maiko README file explains what to do. In addition I assume you're familiar with the Unix C toolchain and the shell.

Although the instructions may work on similar platforms, if you're not on Linux you'll need to adapt them to your setup.

The setup

The tools to compile Medley are the same as for C development on Linux, with a few additional dependencies. Make sure the following packages are installed on your system:

  • Clang: clang
  • Make: make
  • X11 development libraries: libx11-dev
  • SDL2 development libraries (optional): libsdl2-dev

libsdl2-dev is needed only if you want SDL2 instead of X11.

I store the Medley sources under ~/interlisp and the build instructions assume this location. Change to the ~/interlisp directory and download the required files.

The Medley Interlisp project automatically publishes weekly binary builds whereas the project repositories hold the source tree. Since there is really no stable source release per se, these instructions assume you use git to download the sources of Maiko, Medley, and NoteCards:

$ cd ~/interlisp
$ git clone https://github.com/Interlisp/maiko.git
$ git clone https://github.com/Interlisp/medley.git
$ git clone https://github.com/Interlisp/notecards.git

The repos are pretty large, so to save space and bandwidth you may want to run git with the --depth 1 option.

After the download the source directory will contain the following subdirectories:

.
├── maiko
├── medley
└── notecards

The build steps

The Medley system consists of two layers, Maiko and the Medley environment.

The bottom layer is the Maiko Virtual Machine, written in C, which implements the evolution of the microcode of the Xerox workstations that ran Interlisp-D in the 1980s. The rest of the system in the top layer, the actual Medley environment the user interacts with, is written in Lisp.

The build process reflects this layered organization. You first compile Maiko with the C toolchain, then run the scripts that drive the Lisp environment to compile the Lisp sources and produce a loadable image, i.e. a sysout.

Building Maiko

Start by changing to the maiko/bin directory of your source tree. To build Maiko for X11 clean up any leftovers from prior invocations and run the build:

$ cd ~/interlisp/maiko/bin
$ ./makeright x cleanup
$ ./makeright x

Next, build the ldeinit Maiko tool required for producing the apps sysout:

$ ./makeright init

These steps create the executable files lde, ldex, and ldeinit under ../ostype.cputype, e.g. ../linux.x86_64 on typical Linux systems. Move the lde* programs to or symlink them from somewhere in your executable path.

If you want the SDL2 version instead of X11 substitute sdl for x in the above build commands:

$ ./makeright sdl cleanup
$ ./makeright sdl
$ ./makeright init

On my Linux system these compilation steps collectively take a dozen seconds of runtime.

Doing the loadup

The final step is to make the apps loadup.

A “load up”, or just “loadup”, is a platform independent Lisp image generated by the Lisp side of the build process. Historically, building the Medley sysout was called “doing a loadup”. Along with an image, doing the loadup produces a set of compiled Lisp and supporting files.

To do the loadup change to the Medley directory ~/interlisp/medley, remove the loadups subdirectory to clean up any leftovers from prior builds, and launch the driver script loadup-all.sh in the scripts subdirectory:

$ cd ~/interlisp/medley
$ ./scripts/loadup-all.sh -apps

The Medley graphical environment will open several times and print log messages on the files it is compiling or other status information.

During this step it is amazing to watch the Medley window and log messages zip by in seconds in a process that took many hours back in the day. On my Linux system this step completes in about 3 minutes and 10 seconds.

Testing the build

Your environment is already set up for building Medley and no additional steps are required for installing and running it.

When the build ends, test Medley by launching the medley command. For example, execute medley with the -a option to load the applications, -e to start an Interlisp Exec, and -n to remove the screen scroll bars that are not much useful:

$ cd ~/interlisp/medley
$ ./medley -a -e -n &

Execute ./medley --help to get a brief help message with a list of the command line options, and ./medley -z to read the manual page.

#Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

I implemented the last features originally planned for Insphex, my hex dump tool in Common Lisp for Medley Interlisp.

The first new feature is an Exec command for invoking the program. The command HD works the same way as the function INSPHEX:HEXDUMP and accepts the same arguments, a file name and an optional boolean flag to indicate whether the output should go to a separate window:

← HD FILENAME [NEWIN-P]

The other feature is the addition to the File Browser menu of the Hexdump command, which shows the hex dump of the selected files in as many separate windows:

Hexdump File Browser command of the Insphex hex dump tool on Medley Interlisp.

For other commands that produce output in windows the File Browser lets the user view one window at a time, with menu options for skipping through the windows. Insphex doesn't do anything so elaborate though.

Implementing the features was easy as the relevant Interlisp APIs are well documented and I have experience with adding an Exec command to Stringscope.

The Medley Lisp library modules manual covers the File Browser API from page 115 of the PDF, with the explanation of how to add commands on page 118. It's as simple as registering a callback function the command invokes, INSPHEX::FB-HEXDUMP for Insphex.

An issue I bumped into is that instead of 4 arguments as the manual says, the callback actually requires 5. The last, undocumented argument was likely introduced since the publication of the manual.

#insphex #CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

I added a GUI to Insphex, the hex dump tool I'm writing in Common Lisp on the Medley Interlisp environment.

The initial version printed the hex dump only to the standard output, now optionally to a separate TEdit window with a command menu. The menu has items for displaying the next page of output, redisplaying from the beginning of the file, and exiting the program.

Window and command menu of the Insphex hex dump tool for Medley Interlisp.

Most window, menu, and other Medley GUI facilities, like the TEdit rich text editor, provide Interlisp APIs in the IL package that Common Lisp programs such as Insphex can access. However, since the APIs usually rely on Interlisp records, from Common Lisp it's often necessary to write quite a few package qualifiers like this example to create a menu record:

(IL:CREATE IL:MENU
           IL:ITEMS IL:← '(ITEM1 ITEM2 ITEM3)
           IL:MENUFONT IL:← '(IL:MODERN 12)
           IL:TITLE IL:← "Menu"
           IL:CENTERFLG IL:← T)

The XCL:DEFINE-RECORD macro helps reduce package qualifiers by wrapping Interlisp records in equivalent Common Lisp structures with ordinary structure accessors, setters, predicates, and constructors. The structures can be in any package, not just IL like Interlisp symbols. XCL:DEFINE-RECORD is described on page 7-3 (page 143 of the PDF) of the Medley 1.0 release notes.

This way Common Lisp blends well with Interlisp and reduces verbosity. For example, this is the Insphex Common Lisp function that creates the output window:

(DEFUN CREATE-HEX-WINDOW (FILE)
   "Create and return a window to display the hex dump of FILE."
   (LET* ((IN (OPEN FILE :DIRECTION :INPUT :ELEMENT-TYPE '(UNSIGNED-BYTE 8)))
          (COMMANDS (IL:MENUWINDOW (MAKE-MENU :ITEMS '(("Next" :NEXT "Show the next page.")
                                                       ("Reread" :REREAD "Reread the input file.")
                                                       ("Exit" :EXIT "Quit the program."))
                                          :MENUFONT
                                          '(IL:MODERN 12)
                                          :TITLE "Commands" :CENTERFLG T :WHENSELECTEDFN 
                                          #'HANDLE-MENU)))
          (OUT (IL:OPENTEXTSTREAM))
          (TEDIT-PROC (IL:TEDIT OUT))
          (WINDOW (IL:WFROMDS OUT)))
     (IL:ATTACHWINDOW COMMANDS WINDOW 'IL:TOP 'IL:LEFT)
     (IL:WINDOWPROP WINDOW 'INSTREAM IN)
     (IL:WINDOWPROP WINDOW 'OUTSTREAM OUT)
     (IL:WINDOWPROP WINDOW 'BLOCK-OFFSET 0)
     (IL:WINDOWPROP WINDOW 'IL:TITLE (FORMAT NIL "Insphex ~A" FILE))
     (NEXT-HEX-PAGE WINDOW)
     WINDOW))

The INSPHEX::MAKE-MENU constructor creates a Common Lisp INSPHEX::MENU structure that wraps the Interlisp IL:MENU record.

Most of the Insphex GUI functionality is in place but I need to work on a couple of tweaks.

First, the Insphex window should be read-only whereas now the user can type into the editor buffer. Next, I need to clean up all the allocated resources when the user quits the program via various interaction flows, such as closing the window instead of clicking the Exit menu item.

#insphex #CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

I'm developing the new program Insphex (inspect hex), a hex dump tool that is created with and runs on the Medley Interlisp environment.

Similarly to the Linux command hexdump, it shows the contents of files as hexadecimal values and the corresponding ASCII characters. An early version of the program prints the hex dump to the standard output like this.

Output of the Insphex hex dump tool for Medley Interlisp.

I plan to enhance Insphex to optionally display the dump in a separate window one page at a time. An attached menu will have options for showing the next page and exiting. I'll also provide an Exec command for running the program.

The code is in Common Lisp but will include some Interlisp to access the required system functionality.

Although Insphex is useful in itself, I have three main goals for it. First, I want a real project to practice the process for writing Common Lisp with the residential environment of Medley. This is the native way of coding on Medley and takes full advantage of its development environment and features such as the File Manager and the SEdit editor.

Most Medley tools and facilities are written in Interlisp or expose Interlisp APIs through which the functionality can be invoked. So another goal is to interface with Interlisp from Common Lisp to access the functionality I need like windows and menus.

My third goal is to experiment with displaying textual output in TEdit, the Medley word processor where the hex dump will optionally go.

Although the Interlisp API of TEdit supports advanced editing and formatting, Insphex does only basic text output. The primary feature I want is TEdit's ability to automatically handle repainting the window after it's resized or a hidden portion is exposed. This is handy as by default Interlisp windows mostly don't handle the repaint.

Now that the basic functionality of Insphex is in place I will implement displaying the hex dump in a TEdit window.

#insphex #CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

When the Medley Interlisp Project began reviving the system around 2020, its Common Lisp implementation was in the state it had when commercial development petered off in the 1990s, mostly prior to the ANSI standard.

Back then Medley Common Lisp mostly supported CLtL1 plus CLOS and the condition system. Some patches submitted several years later to bring the language closer to CLtL2 needed review and integration.

Aside from these general areas there was no detailed information on what Medley missed or differed from ANSI Common Lisp.

In late 2021 Larry Masinter proposed to evaluate the ANSI compatibility of Medley Common Lisp by running the code of popular Common Lisp books and documenting any divergences. In March of 2024 I set to work to test the code of the book Practical Common Lisp by Peter Seibel.

I went over the book chapter by chapter and completed a first pass, documenting the effort in a GitHub issue and a series of discussion posts. In addition I updated a running list of divergences from ANSI Common Lisp.

Methodology

Part of the code of the book is contained in the examples in the text and the rest in the downloadable source files, which constitute some more substantial projects.

To test the code on Medley I evaluated the definitions and expressions at a Xerox Common Lisp Exec, noting any errors or differences from the expected outcomes. When relevant source files were available I loaded them prior to evaluating the test expressions so that any required definitions and dependencies were present. ASDF hasn't been ported to Medley, so I loaded the files manually.

Adapting the code

Before running the code I had to apply a number of changes. I filled in any missing function and class definitions the book leaves out as incidental to the exposition. This also involved adding appropriate function calls and object instantiations to exercise the definitions or produce the expected output.

The source files of the book needed adaptation too due to the way Medley handles pure Common Lisp files.

Skipped code

The text and source files contain also code I couldn't run because some features are known to be missing from Medley, or key dependencies can't be fulfilled. For example, a few chapters rely on the AllegroServe HTTP server which doesn't run on Medley. Although Medley does have a XNS network stack, providing the TCP/IP network functions AllegroServe assumes would be a major project.

Some chapters depend on code in earlier chapters that uses features not available in Medley Common Lisp, so I had to skip those too.

Findings

Having completed the first pass over Practical Common Lisp, my initial impression is Medley's implementation of Common Lisp is capable and extensive. It can run with minor or no changes code that uses most basic and intermediate Common Lisp features.

The majority of the code I tried ran as expected. However, this work did reveal significant gaps and divergences from ANSI.

To account for the residential environment and other peculiarities of Medley, packages need to be defined in a specific way. For example, some common defpackage keyword arguments differ from ANSI. Also, uppercase strings seem to work better than keywords as package designators.

As for the gaps the loop iteration macro, symbol-macrolet, the #p reader macro, and other features turned out to be missing or not work.

While the incompatibilities with ANSI Common Lisp are relativaly easy to address or work around, what new users may find more difficult is understanding and using the residential environment of Medley.

Bringing Medley closer to ANSI Common Lisp

To plug the gaps this project uncovered Larry ported or implemented some of the missing features and fixed a few issues.

He ported a loop implementation which he's enhancing to add missing functionality like iterating over hash tables. Iterating over packages, which loop lacks at this time, is trickier. More work went into adding #p and an experimental symbol-macrolet.

Reviewing and merging the CLtL2 patches is still an open issue, a major project that involves substantial effort.

Future work and conclusion

When the new features are ready I'll do a second pass to check if more of the skipped code runs. Another outcome of the work may be the beginning of a test suite for Medley Common Lisp.

Regardless of the limitations, what the project highlighted is Medley is ready as a development environment for writing new Common Lisp code, or porting libraries and applications of small to medium complexity.

#CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

I finally fixed a longstanding duplicate definitions issue with Femtounit, my Interlisp unit test framework.

Femtounit creates a new File Manager type for unit tests, TESTS. DEFTEST, which defines a test, adds an entry of type TESTS and expands into an internal function that carries out the test when called. The problem was the File Manager noticed and tracked two objects for each DEFTEST, the TESTS entry and the function. But the function is an implementation detail that shouldn't be tracked by the File Manager or seen directly by the user.

To fix the issue, in the DEFDEFINER that creates the new type I added a call to UNMARKASCHANGED immediately after the definition of the internal function. UNMARKASCHANGED undoes the association with the File Manager the creation of the internal function establishes. I had tried DELDEF but it removes both the association and the internal function.

I also refactored the definition of the internal function to use DEFINEQ instead of assigning a LAMBDA to the function cell of the symbol naming the test, which is less clear and obscures the intent.

#femtounit #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

I got a cheap used copy of the book A Programmer's Guide to COMMON LISP by Deborah G. Tatar, Digital Press, 1987.

The book A Programmer's Guide to COMMON LISP by Deborah G.

Why did I read such an old book, published a few years after CLtL1 and well before ANSI finalized the Common Lisp standard?

I'm always looking for good Lisp books. Since Medley is my primary Lisp environment, I'm particularly interested in books published when the system was originally developed and used. These works are relevant because they cover a set of features close to the state of the Common Lisp implementation of Medley, and present a programming style typical of Lisp development in those years.

Two old reviews got me curious about A Programmer's Guide to COMMON LISP, one by Daniel Weinreb and the other by Richard Caruana.

Both reviews point out the book is different from most contemporary introductory Lisp books which focus on AI. Although Tatar's does contain some AI code, such as an interesting and complete toy expert system, the sample code spans a wider range of domains like a text formatter similar to nroff.

What sets A Programmer's Guide to COMMON LISP apart from other Lisp books is its environment independent discussion of the interactive Lisp programming process. Writing code in the editor, evaluating expressions from the editor, interacting with the REPL for testing expressions and exploring, and so on.

I've never seen the process expressed so clearly in any book, past of present. I'm familiar with it but the material is particulary helpful for complete beginners.

Although the short chapter on macros presents some interesting examples like a simplified version of defstruct, it doesn't discuss gensym and variable capture. This is unusual. But it's only one of a few issues and the book is a valuable addition to my Lisp library.

#CommonLisp #books #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

Medley is a residential environment for Interlisp and Common Lisp development.

With some effort it's possible to use Medley as a traditional file based Common Lisp environment. But in specific cases a better approach is to bring in Medley's residential environment Common Lisp sources created in file based environments.

In this post I explain the latter, i.e. how to use TextModules to import Common Lisp files into the residential environment. I go over the steps for converting an example program, the database of CD music tracks in Chapter 3 Practical: A Simple Database of Peter Seibel's book Practical Common Lisp.

Motivation

Using the tools and facilities of the residential environment, such as the File Manager, is the normal way of developing new Lisp programs. To run existing Common Lisp code you don't plan to change often, you can also use Medley as a traditional file based environment.

For existing code written in file based environments you want to use and further develop in Medley, a better option is to import the code into the residential environment and continue working from there. This is what TextModules helps to do.

What is TextModules

TextModules is a Medley tool for bringing Common Lisp sources into the residential environment and place them under the control of the File Manager. It can also do the reverse, i.e. export the File Manager descriptions and metadata to Common Lisp sources accessible from file based environments.

Using TextModules is a one time process. You run the tool once to import the code, then use and modify it with the tools and facilities of the residential environment.

The documentation of textModules starts from page 305 (page 335 of the PDF) of the Lisp Library Modules manual.

Preparing the Common Lisp files

This example involves two of the source files of Seibel's book, packages.lisp and simple-database.lisp in directory practicals-1.0.3/Chapter03 of the code archive.

Unlike CL:LOAD on Medley, TextModules doesn't require any special formatting of Common Lisp source files. For example, they don't need to begin with a semicolon character.

However, the Common Lisp implementation of Medley is incomplete and not ANSI compliant, so be sure to remove or adapt any unsupported forms. This is the case of the database example: packages.lisp makes current the package CL-USER which is missing from Medley. The fix is to substitute xcl-user for cl-user in the file, as XCL-USER is the Medley equivalent of CL-USER.

Another source of incompatibility is the LOOP macro. In Medley it's only a stub that runs an infinite loop no matter what clauses a call specifies.

To import with TextModules code that contains LOOP it would normally be necessary to replace any calls with equivalent expressions. Since this post focuses on TextModules I just use the LOOP calls intended to run an infinite loop, and ignore the others.

Running TextModules

As noted, importing with TextModules is the one time process of running the tool for every source file. Once in the residential environment, you save and manipulate the code as any other code under the File Manager.

First off, load TextModules by evaluating (FILESLOAD TEXTMODULES) at an Interlisp Exec. All its exported symbols are in package TM. Next, call the function TM:LOAD-TEXTMODULE for every Common Lisp file, which is similar to CL:LOAD with some additional processing.

Most Common Lisp programs comprise a file packages.lisp with package definitions, and a number of additional .lisp files that contain the bulk of the code. This dependency requires passing the files to TM:LOAD-TEXTMODULE in the proper order.

The file packages.lisp of the database defines the package for simple-database.lisp, so start with the former. At a Xerox Common Lisp (XCL) Exec with prompt > evaluate:

> (tm:load-textmodule "packages.lisp" :module "SIMPLEDB" :package (find-package "XCL-USER") :install t)
IL:SIMPLEDB

The only required argument is the input file packages.lisp. However, by default TM:LOAD-TEXTMODULE uses the same input file name as the name of the program for the File Manager. It wouldn't make much sense to call a database PACKAGES.LISP. A better choice is to pass the :module parameter with the more descriptive name SIMPLEDB.

The code in packages.lisp begins with an in-package form. To make sure the in-package symbol is accessible without qualifier, it should be read in a package such as XCL-USER that imports the standard Common Lisp symbols. Hence the argument :package (find-package "XCL-USER") in the call.

The argument :install t installs the definitions in the running system. Although not strictly necessary, it's useful for diagnostic purposes and because you likely want to continue working on the imported code.

Next, process simple-database.lisp by evaluating at a XCL Exec:

> (tm:load-textmodule "simple-database.lisp" :module "SIMPLEDB" :package (find-package "XCL-USER") :install t)
IL:SIMPLEDB

Again, the file begins with in-package and the reason for passing the :package argument is the same as for packages.lisp.

Saving the imported code

At this point the imported code is in the running Lisp image and the File Manager is ready to manipulate it. You can check the File Manager noticed the imported definitions by calling FILES? at an Interlisp Exec with prompt ←:

← (FILES?)
To be dumped:
SIMPLEDB ...changes to VARS: SIMPLEDBCOMS
                       VARIABLES: COM.GIGAMONKEYS.SIMPLE-DB::*DB*
                       FUNCTIONS: COM.GIGAMONKEYS.SIMPLE-DB::MAKE-CD, 
         COM.GIGAMONKEYS.SIMPLE-DB::ADD-RECORD, 
         COM.GIGAMONKEYS.SIMPLE-DB::DUMP-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::PROMPT-READ, 
         COM.GIGAMONKEYS.SIMPLE-DB::PROMPT-FOR-CD, 
         COM.GIGAMONKEYS.SIMPLE-DB::ADD-CDS, 
         COM.GIGAMONKEYS.SIMPLE-DB::SAVE-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::LOAD-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::CLEAR-DB, 
         COM.GIGAMONKEYS.SIMPLE-DB::SELECT, 
         COM.GIGAMONKEYS.SIMPLE-DB::WHERE, 
         COM.GIGAMONKEYS.SIMPLE-DB::MAKE-COMPARISONS-LIST, 
         COM.GIGAMONKEYS.SIMPLE-DB::MAKE-COMPARISON-EXPR, 
         COM.GIGAMONKEYS.SIMPLE-DB::UPDATE, 
         COM.GIGAMONKEYS.SIMPLE-DB::DELETE-ROWS

To save the definitions to the symbolic file SIMPLEDB call MAKEFILE from an Interlisp Exec:

← (MAKEFILE 'SIMPLEDB)
{DSK}<home>medley>il>SIMPLEDB.;1

Calling TM:LOAD-TEXTMODULE for every source file, and saving the result to a symbolic file with MAKEFILE, completes the import process.

You may terminate the session and resume later. When you're ready to proceed you can load, run, and modify the imported program as any other code under File Manager control.

Loading and running the imported code

In a new Medley session evaluate (FILESLOAD TEXTMODULES) at an Interlisp Exec. The tool must be in memory whenever you work with imported code, as TextModules sets up a special file environment and readtable the code needs to be read in.

Next, load the symbolic file of the database program by evaluating at a XCL Exec:

> (load "SIMPLEDB")

; Loading {DSK}<home>medley>il>SIMPLEDB.;1
; File created 14-Feb-2024 02:44:46
; IL:SIMPLEDBCOMS
IL:|{DSK}<home>medley>il>SIMPLEDB.;1|

One of the main entry points of the program is the function add-cds to add new records to the database, one record for each music track of a CD. A typical run from a XCL Exec looks like this:

> (setf *package* (find-package "COM.GIGAMONKEYS.SIMPLE-DB"))
#<Package COM.GIGAMONKEYS.SIMPLE-DB>
> (add-cds)
Title: Punch My Cards
Artist: The Fortrans
Rating: 6
Ripped [y/n]: n
Another? [y/n]: y
Title: Lisp n Roll
Artist: The Garbage Collectors
Rating: 8
Ripped [y/n]: y
Another? [y/n]: y
Title: Cdr Care Less
Artist: The Garbage Collectors
Rating: 7
Ripped [y/n]: y
Another? [y/n]: n
NIL

Since all the symbols of the program are in the package COM.GIGAMONKEYS.SIMPLE-DB, and none are exported, for convenience make the package current by setfing *package* as above.

I intentionally didn't rename the package or create nicknames. This is to show that Common Lisp code may be imported and used with minimal or no changes.

The program provides select and where to query the database. But where uses CL:LOOP features Medley doesn't support. To stay close to the original code, instead of modifying where you can use another function in Seibel's book. Define and call the specialized query function select-by-artist to search the database by artist:

> (defun select-by-artist (artist)
    (remove-if-not
     #'(lambda (cd) (equal (getf cd :artist) artist))
     *db*))
SELECT-BY-ARTIST
> (select-by-artist "The Garbage Collectors")
((:TITLE "Cdr Care Less" :ARTIST "The Garbage Collectors" :RATING 7 :RIPPED T) (:TITLE "Lisp n Roll" :ARTIST "The Garbage Collectors" :RATING 8 :RIPPED T))

Continuing the development

The imported code works. Now you can load, compile, run, edit, and save it as any other program developed in the residential environment under the File Manager.

You no longer need the original files packages.lisp and simple-database.lisp because you work only with SIMPLEDB. But remember to load TextModules with (FILESLOAD TEXTMODULES) in every session in which you use SIMPLEDB. It's a minor inconvenience but, for automating the task, you may add a loading command to the INIT initialization file or to a script that loads the program.

#CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

Managing Lisp code in the residential environment of Medley differs from similar tasks in traditional file based Common Lisp systems.

In a previous post I explained how the residential environment of Medley works, discussed some of its facilities and tools, and introduced a workflow for managing Common Lisp code under the residential environment. This leverages the way Medley is designed to work and minimizes friction.

In this post I explain how to use Medley as a file based environment, albeit with some friction and reduced functionality.

I show how to edit, organize, and load pure Common Lisp files, i.e. source files not under the control of the File Manager. Pure files are ordinary text files that contain only Common Lisp code, with no metadata or font control commands like the code databases the File Manager maintains.

Motivation

Letting the residential environment track and control code is the most convenient and productive way of using Medley for Lisp development.

But you can still manually manage pure Common Lisp files. For example, you may want to use from Medley some code you mainly intend to load into the environment and don't change often. This is the case of external libraries and programs developed elsewhere, which you call from new code or run in Medley.

Using Common Lisp code not developed for the pre ANSI implementation of Medley may need adaptation. So let's see how to edit source files.

Editing source files

To create or modify pure Common Lisp files use the TEdit word processor of Medley, not the SEdit structure editor. SEdit doesn't directly manipulate the arbitrary, unstructured text of pure files.

TEdit can read and write ordinary text files, but not by default. To save a file as ASCII execute the command Put > Plain-Text. Execute Get > Unformatted Get to load such a file.

The main downside of using TEdit for Common Lisp is the program is not Lisp aware. It doesn't support automatic indentation, prettyprinting, perenthesis matching, and code evaluation. Another major problem is cursor keys don't work in Medley, which severely limits text editing with TEdit. The Medley team is well aware of the issue.

An alternative is to use your favorite Lisp editor to edit the Common Lisp sources outside of Medley and then load the files from Medley.

Format of Pure Common Lisp files

Although Medley can load pure Common Lisp sources, these files do have some minimal formatting requirements. A pure Common Lisp file must begin with a semicolon ; character, otherwise the system assumes it's an Interlisp file.

Other than that a pure file can contain any Common Lisp feature Medley supports, in any order the language allows.

Loading source files

Once a pure Lisp file is available, load it in Medley with the usual file loading forms such as CL:LOAD. The way symbols are interned depends on whether the file defines or makes current any Common Lisp packages.

Let's go over the main cases: the code has no packages; a single file contains both package and function definitions; the code is split between a file for the package and another for the function definitions.

No package definition

The simplest type of pure file contains no package forms, just function definitions and other Common Lisp expressions.

In this case Medley interns symbols in the USER package without exporting them. The Medley implementation of Common Lisp is closer to CLtLx than ANSI, so it provides the standardized packages LISP (nicknames: COMMON-LISP and CL) and USER instead of COMMON-LISP and COMMON-LISP-USER.

For example, suppose this file SQUARE.LISP defines a function calc-square to compute the square of its argument:

;;; SQUARE.LISP

(defun calc-square (x)
  "Return the square of X."
  (* x x))

After evaluating (load "SQUARE.LISP"), at the > prompt of a Common Lisp Exec you call the function using the proper package qualifier:

> (user::square 3)
9

If the USER package isn't adequate for your code, on Medley CL:LOAD also accepts the keyword parameter :package to supply a package to which *package* is bound when reading the file. For example, at the > prompt of a Xerox Common Lisp (XCL) Exec you can pass the package XCL-USER with a form like:

> (load "SQUARE.LISP" :package (find-package "XCL-USER"))

You must pass an actual package object to :package, not a package designator like :xcl-user.

Since XCL-USER is the default package of XCL Execs no qualifier is required for calling the function:

> (square 3)
9

Package and definitions in one file

Not using packages may be adequate for very small files. However, typical Common Lisp programs do define their own packages.

This example file PKGDEMO.LISP defines the package PKGDEMO with nickname PD that exports the function FUN, which just prints a message:

;;; PKGDEMO.LISP


(in-package "XCL-USER")

(defpackage "PKGDEMO"
  (:use "LISP" "XCL")
  (:nicknames "PD")
  (:export "FUN"))


(in-package "PKGDEMO")

(defun fun ()
  (format t "Hello from PKGDEMO:FUN."))

The file holds both the package definition and the rest of the code. It's important that the first form in the file be in-package to make current a known package like XCL-USER. XCL-USER is the Medley equivalent of COMMON-LISP-USER and uses the LISP package with the standard Common Lisp symbols.

After loading PKGDEMO.LISP from an XCL Exec with (load "PKGDEMO.LISP") you can call the exported function like this:

> (pkgdemo:fun)
Hello from PKGDEMO:FUN.
NIL

or via the package nickname:

> (pd:fun)
Hello from PKGDEMO:FUN.
NIL

Separate package and definition files

Unlike PKGDEMO.LISP in the previous example, most Common Lisp programs split the code between at least two files, one holding the package definition and the other the function definitions. Let's rewrite PKGDEMO.LISP by storing the package in the file PKGSPLIT-PKG.LISP:

;;; PKGSPLIT-PKG.LISP


(in-package "XCL-USER")

(defpackage "PKGDEMO"
  (:use "LISP" "XCL")
  (:nicknames "PD")
  (:export "FUN"))

The function definition goes into the file PKGSPLIT-FUN.LISP that begins with an appropriate in-package followed by the rest of the code:

;;; PKGSPLIT-FUN.LISP


(in-package "PKGDEMO")

(defun fun ()
  (format t "Hello from PKGDEMO:FUN."))

Finally, make sure to load the package first with (load "PKGSPLIT-PKG.LISP"), then the function with (load "PKGSPLIT-FUN.LISP"). Loading the files creates the same objects as the single file example PKGDEMO.LISP and you can call the function the same way:

> (pkgdemo:fun)
Hello from PKGDEMO:FUN.
NIL
> (pd:fun)
Hello from PKGDEMO:FUN.
NIL

#CommonLisp #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org