Welcome to the OpenBIOS tokenizer Toke.
This README provides you with a short description of the tokenizer and
of how to use it.
---------------------------------------------------------------------
Table of contents:
1. What is the OpenBIOS tokenizer?
2. What is required to build the OpenBIOS tokenizer?
3. How to use the OpenBIOS tokenizer
4. Toke's Special Features not described in IEEE 1275-1994
5. Text Strings
6. Contact
---------------------------------------------------------------------
1. What is the OpenBIOS tokenizer?
toke is a GPLed FCode tokenizer. It can tokenize (assemble) fcode
source to bytecode files as described by the IEEE 1275-1994 standard.
This program is compliant to IEEE 1275-1994.
Bytecode files normally contain Open Firmware drivers or other
packages for use with an Open Firmware compliant system.
2. What is required to build the OpenBIOS tokenizer?
toke should build with any ANSI compliant C compiler.
For building toke on certain platforms you might have to adjust the
Makefile. If so, please contact Stefan Reinauer <stepan@openbios.org>
Build toke by just enter "make". To clean up an existing build,
use "make clean" or "make distclean"
3. How to use the OpenBIOS tokenizer
tokenize an fcode source file with
toke [-v|--verbose] [-i|--ignore-errors] <file.4th> [<file2.4th>]
Get help with:
toke [-h|--help]
The file extension will be changed to .fc. If no file extension
exists, .fc will be appended. If -i is specified, toke continues
on errors, producing invalid fcode binaries.
4. Toke's Special Features not described in IEEE 1275-1994
In addition to the standard Open Firmware words, some additional
commands have been added to make life easier and for compatibility
to other, mostly commercial, fcode tokenizers. These are:
4.1 NEXT-FCODE
In tokenizer[ .. ]tokenizer context there is an additional control
word besides emit-byte: next-fcode. It sets the increasing fcode
number used to emit new fcode words to the specified value. Thus
allowing sparse fcode numbering and overwriting previously defined
fcode words.
This could look like:
-------------------------------------------
\ next emitted fcode is 0x053, 2dup.
tokenizer[ 053 next-fcode ]tokenizer
\ now define the word.
: 2dup over over ;
-------------------------------------------
Note: Toke is capable of creating fcode files that implement words
defined by the IEEE 1275-1994 standard itself. This is not necessarily
supported by every Open Firmware implementation.
4.2 FCODE-VERSION2
This word generates an FCode header using START1. It should be used
with FCODE-END.
4.3 FCODE-END
This word creates an END0 FCode and fixes up the FCode header
information. Use this with FCODE-VERSION2
4.4 PCI-HEADER
This word creates a PCI option ROM header. PCI-HEADER expects 3
values on the top of the stack when it is invoked: Vendor#, Device#
and Class-Code.
These values are used for the according fields in the PCI Data
structure that is emitted by PCI-HEADER. To include these values
put them on the stack with TOKENIZER[ and ]TOKENIZER.
4.5 PCI-END / PCI-HEADER-END
This word completes the PCI header created by PCI-HEADER. It fills out
the image to a multiple of 512 bytes, and sets the missing values in
the PCI header (image-length, revision).
Example:
HEX
TOKENIZER[ 1234 5678 ABCDEF ]TOKENIZER
PCI-HEADER \ generate PCI option rom header
FCODE-VERSION2 \ generate FCode header (within PCI image)
...
...
FCODE-END \ terminate FCode in image
PCI-END \ complete the PCI header and image.
4.6 PCI-REVISION / SET-REV-LEVEL
Use this word to change the revision field of the PCI header. Like
PCI-HEADER, PCI-REVISION takes it's value from the stack. You can
write: TOKENIZER[ 23 ]TOKENIZER PCI-REVISION
4.7 NOT-LAST-IMAGE
Normally Toke assumes that a PCI image generated by PCI-HEADER is
the only ROM image in the (EEP)ROM it will be written to. With
NOT-LAST-IMAGE Toke sets a flag in the PCI header that other images
will follow in the same ROM.
4.8 FCODE-DATE
FCODE-DATE creates an FCode string that contains the date of the
tokenization. This string could be used to create a device tree
property that can be inspected by drivers, etc. to check when the
image was created. The format of the string emitted by FCODE-DATE
is MM/DD.YYYY
4.9 FCODE-TIME
FCODE-TIME creates an FCode string that contains the time of the
tokenization. This string could be used to create a device tree
property that can be inspected by drivers, etc. to check when the
image was created. The format of the string emitted by FCODE-TIME
is HH:MM:SS
4.10 ENCODE-FILE
encode a binary file. Use as: encode-file filename
5. Text Strings
From "Writing FCode":
Escape Sequences in Text Strings
-----------------------------------------------------------
Syntax Function
-----------------------------------------------------------
"" quote (")
"n newline
"r carriage return
"t tab
"f formfeed
"l linefeed
"b backspace
"! bell
"^x control x, where x is any printable
character
"(hh hh) Sequence of bytes, one byte for each pair
of hex digits hh . Non-hex characters will
be ignored
" followed by any other printable character not mentioned
above is equivalent to that character.
"( means to start parsing pairs of hexadecimal digits as
one or more 8-bit characters in the range 0x00 through
0xFF, delimited by a trailing ) and ignoring
non-hexadecimal digits between pairs of hexadecimal
digits. Both uppercase and lowercase hexadecimal digits are
recognized. Since non-hex characters (such as space or
comma) are ignored between "( and ), these characters make
useful delimiters. (The "makearray" tool can be used in
conjunction with this syntax to easily incorporate large
binary data fields into any FCode Program.)
Any characters thus recognized are appended to any previous
text in the string being assembled. After the ) is
recognized, text assembly continues until a trailing
"<whitespace>.
For example:
" This is "(01 32,8e)abc"nA test! "!"! abcd""hijk"^bl"
^^^^^^^^ ^ ^ ^ ^ ^
3 bytes newline 2 bells " control b
6. Contact
Ideas, bug reports, patches, contributions are welcome.
Stefan Reinauer <stepan@openbios.org>
2759b42cc1