Paolo Amoroso's Journal

I moved the Stringscope project infrastructure from a GitHub gist to a full repo. The gist is now unmaintained and links to the repo.

Announcing my new Interlisp project Braincons I motivated setting up its infrastructure as a repo with a number of reasons boiling down to convenience. Code is still readble despite the formatting Interlisp encodes, source files no longer need conversion, and a full repo is more flexible. I moved Stringscope to a repo for the same reasons.

#stringscope #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

After Stringscope I've been working on Braincons, an Interlisp implementation of the esoteric programming language Brainfuck. The project goals are to further learn the Medley Interlisp environment I'm developing Braincons with, as well as experiment with the compilation and implementation of a very simple language.

Status

Braincons is in an early stage of development and largely incomplete, so I don't have much to show yet.

There's no program, output window, or Executive command to run. But preliminary versions of some of the main functions and data structures are in place, which can be called and inspected from Lisp.

So far the system can only parse Brainfuck program sources, check for errors, and produce an intermediate representation a virtual machine will eventually execute.

Installation

To try out Braincons download the file BRAINCONS from the project repo, copy it to a file system your Medley Interlisp installation has access to, and evaluate the following expression from the Lisp Executive:

(FILESLOAD BRAINCONS)

Usage

At this point you can call the parser function BRC.PARSE and pass it a Brainfuck source string:

(SETQ PARSED.PROGRAM (BRC.PARSE "<>[.,]+-"))

BRC.PARSE returns the intermidiate representation of the source, then assigned to the variable PARSED.PROGRAM which you can inspect as a BRC.PROGRAM record:

(INSPECT PARSED.PROGRAM)

The inspected record and its fields should look similar to the above screenshot. This is all I have for now.

Design

I initially focused on designing the main data structures and functions upon which to build the rest of Braincons.

First, I assigned descriptive mnemonics to the Brainfuck commands as in this table:

Command	Mnemonic
>	NEXT
<	PREV
+	INC
-	DEC
.	OUT
,	IN
[	JZ
]	JNZ

Braincons represents the source of a Brainfuck program as a string. Two data structures implemented as Interlisp records hold the intermediate representation the parser produces.

The record BRC.PROGRAM consists of an array of instructions, the index of the last instruction in the array, a list of any branch instructions in the source, and a list of any errors encountered while parsing the program. Each array entry holds a Lisp symbol of the mnemonic of the corresponding instruction, except for the branch instructions JZ (jump if zero) and JNZ (jump if not zero).

Branches are records with fields for the mnemonic, the source address in the instruction array, the destination address, and the index of the branch command in the source string.

So far Braincons has only one entry point, function BRC.PARSE, which takes a Brainfuck source string as the only argument and returns the parsed intermediate representation record BRC.PROGRAM. BRC.PARSE calls directly or indirectly some helper functions to parse individual instructions, create branch records, and check for errors.

Sharing

Along with ASCII characters, Interlisp source files contain special characters such as ← and control codes that encode font changes. Visualizing these files in editors and tools that expect ASCII thus shows spurious characters.

To work around this I didn't directly share the raw code of Stringscope. Instead I stripped the control and formatting codes from the source exported from Medley Interlisp, replaced the special characters with more readable versions, and published it to a GitHub gist. The code in the gist is nicely formatted for reading but Medley Interlisp can't load it as is, so I have to provide the raw source separately.

Since Interlisp programmers are used to the formatting, and cleaning it up involves additional steps, I decided to store the raw source of Braincons in a GitHub repo. This also provides the additional flexibility of a full repo.

Further development

I'm actively changing and enhancing the code of the only working feature, the parser. Although the basic design of the main data structures is stabilizing, everything else will likely need a lot of refactoring and cleanup.

Next, I'll implement the virtual machine that processes and executes the parsed program.

In the long term I'd like to build a minimal Brainfuck development environment on top of the parsing and program execution functionality. The environment will comprise a source editor based on TEdit and an interactive visualization with controls for loading and starting programs, showing the state of the virtual machine and the executing program, reading user input, and displaying the output.

#braincons #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

Reporting on the implementation of new commands for Stringscope I noted my confusion on why File Manager building commands such as (CLEANUP 'STRINGSCOPE) sometimes don't write to a file the compiled Lisp code.

It turns out this occurred when loading Stringscope with FILESLOAD, which loads a compiled file if available. So, even after modifying the Lisp source, the File Manager somehow skipped the compilation step as it assumed the file was up to date. Loading Stringscope with LOAD instead, which pulls the source, is usually enough to make CLEANUP write the compiled file.

It's a step forward. But I still don't understand why (CLEANUP 'STRINGSCOPE) or (MAKEFILE 'STRINGSCOPE 'C) don't always compile to a file when the source is edited in memory.

#stringscope #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

One year ago today I published the first post of this blog hosted at Write.as, ending a long quest for a better publishing platform.

I had grown frustrated with the clunkyness and limitations of Blogger, where I settled for the prior threee years. So I set out to research a lightweight blogging platform with good support for technical writing that could replace Blogger, and finally went with Write.as. One year and 225 posts later, I'd say Write.as was the right choice despite a major missing feature, post previews. I've never been so productive, neither with Blogger nor any other publishing platform.

Another fortunate decision, intentionally against the conventional wisdom of SEO, was to publish a personal blog that focuses on me rather than remaining confined in a specific niche. This let me write about new interests or experiences, adapting to the wandering and widening of my interests and activities.

The first year on Write.as was eventful. Two of my posts went viral on Hacker News, one on why I use a Chromebox and the other on my encounter with Medley Interlisp, and brought tens of thousands of views.

That was exciting. But over the past year I also joined the Fediverse, and sharing some of my blog posts there generated more interesting, meaninfgul, and lasting interactions than the drive-by traffic of viral posts.

Aside from the rewards and validation of such success metrics, an unexpected benefit of the first year of blogging at Write.as has been writing for an audience of one: me.

I'm using the blog more and more as a sort of lab journal for my hobby programming and tech projects. Since I work on a number of projects, and put them aside for some time before resuming, the blog has proven invaluable in getting back up to speed with projects or track useful resources by reviewing and referencing past posts.

So, even if nobody read my blog, I would still gain the most benefit from this personal space and online archive.

#blogging

Discuss... Email | Reply @amoroso@fosstodon.org

In addition to blogging about Medley Interlisp I also post about it on my Mastodon profile with the hashtag #interlisp. I share links, screenshots, project updates, videos, and other short content.

#Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

I continued working on Stringscope, the string listing tool I'm developing with Medley Interlisp. I added the function STRINGS to print the strings to the primary output stream rather than in a new window, as well as an Executive command that does the same.

To reflect this work I updated the Stringscope code and documentation hosted on a GitHub gist, describing also the function EXTRACT.STRINGS as part of the interface. The code in the gist is intended for publication and can't be loaded as is into Interlisp.

The functions STRINGS and STRINGSCOPE

In addition to the same arguments as STRINGSCOPE, i.e. a file name FILENAME and the minimum length MIN.LEN strings must have, STRINGS accepts a third optional boolean argument NEWWIN. By default NEWWIN is NIL and makes STRINGS print to the primary output, otherwise the function opens a new window and works exactly like STRINGSCOPE. In fact, STRINGSCOPE is now just a wrapper that calls STRINGS with NEWWIN set to T.

STRINGSCOPE and STRINGS return a window if the functions open it otherwise the primary output stream. In case the input file can not be opened or contains no strings, the functions return NIL and print a warning to the prompt window.

Interlisp functions that create or manipulate a window usually return the window itself as their value, so I adopted a similar convention.

The macro WITH.INPUT-FILE

An early version of STRINGS duplicated a lot of the code of STRINGSCOPE for opening the input stream, checking for errors, evaluating some forms with the stream bound to a variable, and closing the stream. I factored that boilerplate into the macro WITH.INPUT.FILE:

(DEFMACRO WITH.INPUT.FILE ((STREAM FILE)
                           &BODY BODY &AUX (RESULT (GENSYM))
                           (VALUE (GENSYM)))

         (* Opens an input stream to FILE and evaluates the forms in BODY with the stream 
         bound to STREAM. Returns the value of the last form in BODY, or NIL if FILE can 
         not be opened.)

   `(PROG NIL
          [SETQ ,RESULT (NLSETQ (OPENSTREAM ,FILE 'INPUT]
          (if ,RESULT
              then (SETQ ,STREAM (CAR ,RESULT))
                   (SETQ ,VALUE (PROGN ,@BODY))
                   (CLOSEF ,STREAM)
                   (RETURN ,VALUE)
            else (RETURN NIL))))

The Executive command STRINGS

The reason I implemented a function like STRINGS that prints the output to the console is I wanted to try out creating an Executive command.

Also known as a listener in other Lisp environments, an Executive is an Interlisp window that provides a read-eval-print loop. In addition to Lisp expressions, an Executive accepts commands with a non-parenthesized syntax such as DIR (the equivalent of DIR on MS-DOS and ls on Unix) or CONN directory (cd directory on MS-DOS and Unix).

So I defined the Stringscope Executive command STRINGS that works exactly like the function STRINGS but takes only the file name and the minimum length arguments:

STRINGS FILENAME MIN.LEN

The macro DEFCOMMAND defines an Executive command and is really easy to use, here's the definition of STRINGS:

(DEFCOMMAND (STRINGS :EVAL) (FILE &OPTIONAL (MIN.LENGTH SSCOPE.MIN.LEN))
   (STRINGS FILE MIN.LENGTH)
   (CL:VALUES))

The STRINGS command is just a wrapper that calls the function STRINGS with appropriate arguments.

Compiling Stringscope

I initially run Stringscope compiled but it was time to start using the File Manager commands for compiling the code, (MAKEFILE 'STRINGSCOPE 'C) and (CLEANUP 'STRINGSCOPE). However, these commands apparently compile only in memory and don't write a compiled file like (TCOMPL 'STRINGSCOPE), which I have to call manually.

The File Manager is an Interlisp subsystem conceptually similar to the Unix tool Make. It's a collection of tools to notice, keep track of, and write to files the code changes to a Lisp system under development.

Another confusing aspect of compilation is when I load Stringscope with (FILESLOAD STRINGSCOPE), I get a warning I don't understand concerning a comment in the macro WITH.INPUT.FILE: Warning: Possible comment not stripped. I actually see the compiler couldn't strip a comment from the compiled code to save space, but not why.

Further development

Some of these new features were on my initial roadmap, others emerged while coding and using Stringscope.

Since this is an exploration and learning project, going forward I'll go with the flow and implement what I need or find interesting, and consider the initial roadmap more like a source of ideas than an action plan. Which is the kind of exploratory development Lisp supports and makes enjoyable.

#stringscope #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

I'm still exploring and learning Medley Interlisp but I finally grasp the basics of structure editing.

The increasing familiarity with the SEdit structure editor is making me more productive. I begin to appreciate the efficiency and elegance of the set of mouse gestures which, combined with keypresses, allow the selection and manipulation of a wide variety of Lisp and text structures.

The core of these gestures is a series of system-wide features and conventions available also in other Interlisp tools such as the TEdit rich text editor.

To demonstrate the basic structure gestures of SEdit, the default Medley Interlisp code editor, I recorded on my Chromebox this short screencast of an Interlisp Online session.

How do structure selection and manipulation work?

The video is supposed to need no explanation. I designed a few visual techniques that hopefully make the video self explanatory while reducing production time and effort.

I recorded a single cut screencast with no audio, in which the mouse pointer moves across the Medley Interlisp desktop highlighting the sequence of steps indicated in one window and carried out in another. The first is a TEdit window showing a document with the outline of the script of the video. Next to TEdit, a SEdit window is open on a Lisp source file in which I carry out the editing tasks listed in the script. Mouse gestures are deliberately slow to allow enought time to register and interpret the changes.

If you follow the mouse pointer, the only thing that moves on the screen, it should be clear what's going to happen next and what triggers an action.

#Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org

Sometimes experimenting with the Z80-MBC2 and V20-MBC homebrew CP/M computers leaves the screen garbled or in an unusable state, usually because some program doesn't correctly handle the terminal. The Minicom terminal emulator I use for CP/M sessions has a command for clearing the screen but it may not be enough.

Since a native CP/M solution is more effective I wrote two short utilities in Assembly for properly clearing and initializing the screen.

These transient programs share the same name, CLS. One runs under CP/M-80 on the Z80-MBC2 with a Z80 processor, the other under CP/M-86 on the V20-MBC with a Nec V20 in 8088 mode. Both assume an ANSI/VT100 terminal and are launched by executing CLS at the command prompt:

A>CLS

The programs work the same way not just due to the similarity of CP/M's design across different architectures, but also because they are variations of a hello world demo that prints a text string to the console.

After defining constants for the CP/M system functions and resources they access, the CLS programs call the write string BDOS function to output a string of ANSI escape codes for clearing the screen and moving the cursor to the home position. The definition of the string ends both programs.

CLS for CP/M-80

The CLS program for CP/M-80 is written in Intel 8080 Assembly:

; Clear the screen.
;
; Runs on CP/M-80 with an ANSI/VT100 terminal.

TPA                 equ     100h
BDOS                equ     05h
WRITESTR            equ     09h             ; Write string


                    org     TPA

                    mvi     c, WRITESTR
                    lxi     d, clshome
                    call    BDOS

                    ret


; ANSI escapes:
;   clear screen      : ESC [ 2 J
;   go to screen home : ESC [ H
clshome:            db      1bh, '[2J', 1bh, '[H$'

                    end

For a short program like this that doesn't need a large stack a ret instruction is adequate to return control to CP/M.

I assembled the program with the asm80 assembler of Suite8080, my suite of 8080 Assembly cross-development tools in Python, and transferred the CLS executable to the Z80-MBC2 over the serial line.

CLS for CP/M-86

CLS for CP/M-86 is written in Intel 8086 Assembly. Aside from the different instruction set and the segmentation directives, this version calls the 00h BDOS function of int 224 to return control as CP/M-86 requires:

; Clear the screen.
;
; Runs on CP/M-86 with an ANSI/VT100 terminal.


WRITESTR            equ     09h             ; BDOS function write string
TERMCPM             equ     00h             ; BDOS function terminate program


                    cseg

                    mov     cl, WRITESTR
                    lea     dx, clshome
                    int     224

                    mov     cl, TERMCPM
                    int     224


                    dseg

                    org     100h

; ANSI escapes:
;    clear screen      : ESC [ 2 J
;    go to screen home : ESC [ H
clshome             db      1bh, '[2J', 1bh, '[H$'

                    end

I transferred the Assembly source to the V20-MBC over the serial line and assembled it with the hosted ASM86 assembler that comes with CP/M-86.

#Assembly #z80mbc2 #v20mbc

Discuss... Email | Reply @amoroso@fosstodon.org

I fulfilled the requirements of the Bring Back Blogging challenge, i.e. posting at least three times over January of 2023. I published 8 posts with not much effort as this is more or less my average output. This is my first blogging challenge and I'm glad I took it as it set a realistic goal, not the usual death march.

#blogging

Discuss... Email | Reply @amoroso@fosstodon.org

After encountering Medley Interlisp I set out to learn the system, as I'd like to contribute to the project and use Medley Interlisp as my primary development environment.

Along with reading the documentation and playing with the system, I started a programming project.

I wanted to create something small that carries out a limited but useful task. My goal was to familiarize with the Medley Interlisp development environment, tools, and process through developing, optimizing, documenting, and sharing an Interlisp program.

I hoped to build a prototype with basic functionality I could finish quickly to have something to play with and extend. Blame instant gratification, and Lisp's productivity.

So I wrote a program I called Stringscope as a nod to the Masterscope program analysis tool of Interlisp.

The program

Stringscope is a tool to display a list of the text strings contained in a binary file, i.e. the sequences of printable characters longer than a minimum threshold of 4 characters. The program is similar to the Unix tool strings.

This is the output window of Stringscope:

The above screenshot is a crop of the full Medley Interlisp desktop:

The Stringscope window shows the output of feeding into the program a word processor document as input. At the right of Stringscope is the TEdit rich text editor with the same document loaded. TEdit writes binary files that contain text and formatting commands, some of which are seen at the bottom of Stringscope window's. The largest window on the desktop shows the source code of Stringscope.

The scroll bar along the left edge of Stringscope's window is a small win I'm proud of.

The Medley Interlisp windowing environment doesn't automatically repaint program-created windows. So I had to figure out how set up Stringscope's window to repaint its content and make it scrollable. I managed to find the right code snippet in the Interlisp Reference Manual and repurposed it.

Usage

Stringscope first needs to be loaded by evaluating this Lisp expression, which assumes the file is in the current directory:

(LOAD 'STRINGSCOPE)

Next, the program is launched by evaluating:

(STRINGSCOPE FILENAME MIN.LEN)

where FILENAME is a file name and MIN.LEN the optional minimum length text strings must have. The default is 4 characters but is user-configurable by changing the global variable SSCOPE.MIN.LEN.

The function STRINGSCOPE opens the input file and feeds the resulting stream into the EXTRACT.STRINGS function, which runs a state machine that recognizes strings and adds them to the output. After receiving the returned list of strings, STRINGSCOPE prompts the user to create a window and sets it up to display the output and respond to repaint and scroll events.

Development

I developed Stringscope on my Chromebox with Interlisp Online, the cloud version of Medley Interlisp accessible from a browser via the noVNC VNC client.

Coding with the SEdit structure editor was smooth and highly productive.

As an image-based environment, in Medley Interlisp the code in memory may be saved to permanent storage by dumping the full memory image. But sources are usually saved to external files by the File Manager that's conceptually similar to the Unix tool Make. The File Manager is a collection of tools to notice, keep track of, and write to files the changes to a Lisp system under development.

Optimization

So far I ran Stringscope on small inputs but I eventually want to process larger files which may require optimizing the code with Spy, the main profiling tool of Medley Interlisp.

To get a feel of Spy I instrumented STRINGSCOPE, soon realizing the profiler collects a lot of unnecessary data related to window and system functions that have little influence on the overall running time. So I'll focus the analysis on EXTRACT.STRINGS which performs the I/O, manipulates the data, and does most of the processing.

Documentation

A lot of the digital documentation that ships with Medley Interlisp and user-contributed software was written with TEdit, the system's versatile rich text editor.

I'd like to use it to write the documentation of Stringscope but TEdit is not fully usable yet. The arrow keys don't work, a showstopper for non trivial text editing. The restoration and modernization of Medley Interlisp is under way and the issue will eventually be fixed, but I'll have to defer natively documenting Stringscope until then.

In the meantime these notes on my blog will have to do.

Sharing

Another major goal of Stringscope is to share my work and get feedback from experienced Interlisp developers.

Interlisp code is stored in “symbolic files”, Medley Interlisp jargon for source files. However, symbolic files are databases rather than traditional source files and they aren't usually edited directly. Instead, code is edited in memory with SEdit and the File Manager takes care of writing and updating symbolic files when the code is modified.

Although symbolic files are text files, exporting and publishing them involves some preparation and adaptation.

First, with Online Interlisp the files need to be downloaded from the cloud. Next, they may need some cleaning.

Symbolic files contain control codes for syntax highlighting and prettyprinting to render different text sizes and attributes such as bold. See for example the Stringscope code in the above screenshot of the desktop.

This is an effective way of presenting code in the environment. The downside is many Lisp symbols are wrapped in sequences of control codes that encode the formatting, which look like spurious characters in ASCII viewers and editors. Moreover, some Interlisp symbols, such as the left arrow ←, have the same ASCII code of the underscore character _.

I manually downloaded Stringscope's symbolic file with the noVNC file manager of Online Interlisp, pasted it into a text editor to strip the control codes, pasted the cleaned up code into a GitHub gist for publication, and replaced _ with ←.

The result is acceptable. But this quick solution is not adequate in the long term as it doesn't scale, and the code can't be directly loaded into Medley Interlisp.

I'll think about how to automate the download and sharing of symbolic files, likely via GitHub. A repo is handy also for hosting PDF files obtained by printing symbolic files to PostScript files in Medley Interlisp and converting to PDF. The PostScript output preserves the text formatting and is easier to read.

Further development

Stringscope is an ongoing project and there's still work to do starting from what is missing, such as profiling and documenting. Another step is to compile the program, as I ran it interpreted so far, making sure the latest compiled binary is loaded.

I'd also like to extend Stringscope with new features such as user interface controls and options for sorting the output, filtering it by string length, and rescanning the input with different minimum string lengths. And it shouldn't be too difficult to call Stringscope from the FileBrowser when a file is selected.

Finally, I want to allow other Lisp programs and functions to call Stringscope to receive the raw output for further processing. In Medley Interlisp, pretty much every piece of code is a building block other code may access and use, so I want Stringscope to contribute too.

Above all, I want Stringscope's evolution to help improve my proficiency with Medley Interlisp.

#stringscope #Interlisp #Lisp

Discuss... Email | Reply @amoroso@fosstodon.org