lips/NOTES.md

# lips internal architecture

…since i'm new to parsers and doomed to forget how my own programs work.

## public interface

(note: this should be moved to the readme or something)

in the simplest case, you just call the table returned by
`require 'lips.init'` as seen in `example.lua`.

lips also returns, in said table: the usual package metadata,
and a `writers` table of pre-defined writers.

in the call interface,
a `writer` function and a table of `options` may be passed
as further arguments after `fn_or_asm`.

```
TODO: example
```

`writer` could be one of the `lips.writers` provided,
after instantiating with a call (it makes use of closure locals).

```
TODO: example
```

options is a table of string keys and any type of value.
currently there is:
```
.unsafe (default false)
    if set, don't wrap the main assembler call in a pcall().
    i want to deprecate this, because it's unnecessary code
    that the user could handle just as well from outside the interface.
.offset (deprecated)
    sets options.origin and options.base simultaneously.
    since the address-base feature was added,
    the preferred way of doing this is by setting them individually,
    so this option is deprecated.
.origin (default 0)
    where to initially start writing in the file.
    this simply tacks on an .org directive
    to the start of the internal assembly.
.base (default 0)
    how far to offset the assembler in where it thinks it's writing.
    this is incredibly important for writing code in ROM
    that is read into a static place in RAM.
    this simply tacks on a .base directive
    to the start of the internal assembly.
.path (default containing directory of assembly file, if applicable)
    primarily used internally for handling relative imports.
.labels (default {})
    note: lips modifies the argument in-place.
    allows for exporting/importing of label data.
    that means you can declare labels in one file,
    and allow a second to access them, in two separate passes.
    otherwise, you would have to hardcode label locations.
    the label format is quite simply a dictionary of string/number pairs:
    { mylabel=0xDEADBEEF, ... }
.debug_tokens (default false)
    dumps the statements table after tokenizing and collecting into statements.
    this is after UNARY and RELLABELSYM tokens have been disambiguated.
.debug_pre (default false)
    dumps statements after basic preprocessing:
    variable substitution, expression parsing,
    relative label substitution, etc.
.debug_post (default false)
    dumps statements after expanding preprocessor commands:
    pseudo-instructions, expression evaluation, etc.
.debug_asm (default false)
    is arguably the least useful of states to dump in.
    this will dump statements after being reduced to
    !ORG and !DATA and !BIN statements. anything else is a bug.
    the values of the !BYTE statements are not printed.
```

## init.lua

the path used to import `lips.init` is mangled
so lips can find its components in each file.
this has to be copy-pasted to every internal file,
which is a small inconvenience.

afaik there isn't really a better way of doing this in vanilla Lua 5.1,
besides mandating to lips be an installed Lua package,
which would be an inconvenience to users (and myself!).

iirc the `gsub` can silently "fail" and allow a couple other
methods of importing, `import "lips"` maybe? i don't remember.
it might work without being in a dedicated directory too

other than that there's not a lot to say.
i've intentionally written this file as stripped down as possible.

i've gone for a one-class-per-file style,
so `file` and `class` will often be synonyms in the following text.

### room for improvement

it would be nice to document options in init.lua,
since ATM i'm abusing Lua's default-to-nil behavior of tables.
that means options could be hidden within any file
and don't demand any forward-declaration or inline documentation.

eventually i'd like to make `writer` a key of `options`
just to simplify the interface even further.
maybe i could pull off `writer_or_options` for backwards compatibility?

someday i'd like to add a `reader` option for handling of existing data,
e.g. for implementing an automated `.hook` directive.

## Parser

"Parser" is a bit of a misnomer, since
the class doesn't do any parsing itself.
it defers parsing to the Lexer, Collector, and Preproc classes.
it also handles writing of the parsed data through the Dumper class.

the main method here is `Parser:method`
which simply interfaces all the important bits of the assembling process.

`self.statements` refers to the "commands" so-to-speak of the assembler
at any point. the general format of this table is:
```
statements={
    {'!BEEP', Tokens...},
    ...
    {'!BOOP', Tokens...},
}
```

the `Parser:dump_debug` method allows for dumping the state of the
`self.statements` table after any of the primary stages of assembling.
refer to the `.debug_token`, `.debug_pre`, `.debug_post`, and `.debug_asm`
options above.

### room for improvement

statements could be made type-restricted, instead of
deferring "this crap ain't even assembled" to each individual stage/class.

i'd like to come up with a better name, but i'm not in any rush.

the debug dumper could be slightly prettier in certain cases.

## Lexer

transforms strings into the tokens they represent.
this does not handle nor consider how they will be collected into statements.

`.inc` directives (and their friends) are handled here:
the appropriate files are placed and tokenized inline, not unlike in C.

the `HEX` directive is its own mini-language and thus has its own mini-lexer.

expressions are not parsed nor lexed here.
they are simply extracted as whole strings for later processing.

the `yield` closure wraps around the `_yield` function argument
to pass error-handling metadata: the current filename and line number.

the rest of the code should be self-explanitory, albiet ugly.

### room for improvement

this character-based lexer isn't driven by any particular grammar,
making it unclear what syntax is and isn't valid.

but it works.
it's the code i need to change the least to add new features.

there's a couple TODOs and FIXMEs in here.

## Collector

TODO

## Preproc

transforms complex statements into simpler statements
that Dumper can later understand.

the name is a bit of a misnomer, because there's
very little processing after preprocessing. (see: `Dumper:load`)

the `:check` method
asserts that a token exists and is of a given type (`tt`).
it will defer to the `:lookup` method if the token type mismatches,
which isn't guaranteed to help.

preprocessing is split into two stages: process and expand; and four passes:

### pass 1

resolves variables by substitution, parses expressions,
and collects relative labels.

this pass starts by creating a new, empty table of statements to fill.
statements are passed through, possibly modified, or read and left-out.

the reason for the copying is that taking indexes into an array (statements)
that you're removing elements from is A Bad Idea.

variable-declaring statements (`!VAR`) are read to a dictionary table,
for future replacement of their keys with values by the `:lookup` method.

note that the variable-parsing code itself calls `:lookup` through `:check`,
so new variables can simply copy the values of previous variables.

labels (`!LABEL`) are checked for RELLABEL tokens to collect
for later replacement in pass 2.
the positive and negative relative labels are collected into their own tables,
appended and prepended respectively.
the collection tables are arrays of tables containing the keys
`index` and `name`.

every statement that isn't eaten has its tokens looked-up by the
`:lookup` method. at this state, it just handles variable substitution.

### pass 2

resolves relative labels by substitution.

this code enables `self.do_labels` which tells `:lookup` to start
handling relative labels as well, now that they've all been collected.

`:lookup` is run on every token of every statement.

the appending/prepending done in pass 1 ensures
that the appropriate relative labels are found in the proper order.

### pass 3

attempts to parse and evaluate constant expressions.

### pass 4

expands pseudo-instructions, including the inferrence of implied registers.

pseudo-instructions are defined in `overrides.lua`.
overrides act as extensions to the Preproc class;
they are passed Preproc's `self`.
this keeps boilerplate out of `overrides.lua`,
but makes our own file more of a mess,
with more dependencies for arbitrary token/statement handling.

### room for improvment

as noted above, the name is a bit of a misnomer,
so this class should probably be split in two.

pass 3 (expressions) should be an attempt to evaluate constants,
and parsing should be moved to be part of pass 1.

pass 4 (expansion) is really messy.

looking back, the `new_statements` ordeal
only seems necessary for the (poor) error handling it provides.

the handling of statement tables could be made better.

## Expression

handles parsing and evaluation of simple (usually mathematical) expressions.

this class is actually completely independent of the rest of lips,
besides the requirement of the `Base` class, which isn't specific to lips.

### room for improvement

right now, this is just a quick and dirty port of some
C++ code i wrote a while back. so basically, everything could be improved.

bitwise operators need to be implemented.
possibly with LuaJIT and a Lua 5.1 fallback.
maybe that should be its own file?

i might want to consider generating a abstract syntax tree,
instead of reverse polish notation,
so that i can handle short-circuiting `&&` and `||` operators,
among other things, like evaluating stuff
in logical order instead of right-to-left for everything.

## Dumper

TODO

## helper classes

### Token

implements error-checking for tokens,
and provides convenience methods.

also handles computation of numeric tokens,
since Token objects contain all the data necessary to do so.

### Statement

implements some error-checking for statements.

### Reader

inherited by stuff

### Muncher

inherited by stuff

### room for improvement

Reader and Muncher classes shouldn't even be necessary;
they could at least be reduced into one.

## etc.

etc!

### overrides.lua

refer to the section on Preproc.

### data.lua

contains most of the information required
to assemble MIPS III assembly code.

this file does not expose any functions or methods,
only constant data.
however, some of the data may be generated through local functions.

### util.lua

contains various utility functions to be lightly sprinkled over files.

most of this shouldn't be specific to lips.

### writers.lua

implements a few must-have writer-generators.

`make_tester` is just a variant of `make_verbose`
that only prints addresses as necessary, reducing noise.

### room for improvement in general

for proper documentation,
i need to copy-paste and rewrite most of the crap here into
the appropriate files themselves.

see also the TODO file.