Programming Language for Old Timers


by David A. Moon
February 2006 .. September 2008

Comments and criticisms to dave underscore moon atsign alum dot mit dot edu.


Previous page   Table of Contents   Next page


Names, Particles, And Tokens

A name is called a "particle" in a particular construct if its meaning in that construct is determined by the syntax of the containing construct rather than by the name's visible definition in the current scope. Many macros have sections of their syntax that are introduced by particles. For example, in an "if" statement, "else" is a particle.

A particle can be either absolute or bound. An absolute particle is recognized solely by the spelling of its name. Thus absolute particles are not affected by hygiene, scopes, or modules. A bound particle is recognized by having a particular definition as the known definition of its name in the context where the particle appears. Thus bound particles are affected by scopes and modules, and can have different spellings in different scopes. However the meaning of a bound particle is not its definition.

Let us be clear about what a name is:

In the data model, there is an abstract class called name. It has one subclass, simple-name, that is useful at compile time and run time, as well as a subclass name-in-context that is only useful at compile time. Simple-names are "interned," i.e. only one instance exists per spelling. This is like a Lisp symbol (without packages and property lists).

All names have a spelling property, which is a string, and a context property, which is false, a module, or a macro-context. The context of a simple-name is always false. These properties are not assignable; in other words, names are immutable.

In the program model, the class name represents names that are bound to things. The subclass name-in-context is used in connection with name-in-module and hygienic macros. It is not interned. The subclass simple-name represents an ordinary identifier and is interned.

When looking up the definition of a name in a binding scope, a name only matches a definition if the spellings are the same and the contexts are eq. When no local definition matches, search a module for a global definition with the same spelling, ignoring context. The module to search is determined by the context of the name; if the context is false, search the current module. If the context is a module, search that module. If the context is a macro-context, first seach an additional local scope indicated by the context, then search the module indicated by the context.

As an absolute particle, a name-in-context and a simple-name are equivalent if they have the same spelling (case-independent).

Use = to compare names according to whether they match in the sense of referring to the same definition when used in the same scope. This is true if and only if the spellings are the same and the contexts are eq. Use same-spelling? to compare absolute particles, ignoring the context and looking only at the spelling. Compare bound particles by checking for a known definition with the expected value. This can be done by calling the match-bound-particle? function.

In the lexical syntax, both delimited name tokens and punctuation tokens are represented by instances of the class simple-name. The difference between punctuation and other name tokens is only in whether delimiting characters are required in the lexical syntax. A keyword is represented by a keyword object whose name slot contains a simple-name with the same spelling as the keyword except the final colon is removed.

Because of macros, any object can be a token. This includes objects that cannot be lexed as tokens when reading from source text. The object just needs to be acceptable to the syntax of the construct where it appears. Many names in the expansion of a macro will be represented by instances of the class name-in-context rather than simple-name.

Many parse methods accept from the token-stream either a sequence of tokens that parses to a given construct, or a single object that represents the already parsed form of that construct. However by convention we always understand sequences as sequences of tokens, never as single tokens. This includes an empty sequence.

The (pseudo) constructor method for the abstract class name has the signature

  name(spelling is string or name,
       optional: context is false or name or macro-context or module,
       result: name is name)

If the spelling argument is a string, it is the spelling of the new name. If the spelling argument is a name, that name's spelling is used.

The context argument can be false, a name, a macro-context, or a module. When it is a name, use that name's context. So by calling name with a name as the second argument one can construct a name that has the same hygienic context as another name.

If the context is false, the result is a simple-name. If the context is a macro-context or a module, the result is a name-in-context with that context.

This is actually implemented as several distinct methods for the function name.

For example, to translate type-name to the name that is bound to the type's class, simply execute

  name("$" + type-name, type-name)

There is a + method that takes a string and a name and concatenates the string with the spelling of the name, returning a string.


Previous page   Table of Contents   Next page