I scan a string or text, picking out Smalltalk syntactic tokens. I look one character ahead. I put each token found into the instance variable, token, and its type (a Symbol) into the variable, tokenType. At the end of the input stream, I pretend to see an endless sequence of special characters called doIts.
aheadChar
- the next character in the input stream
buffer
- a reusable WriteStream on a String which is used for building strings. Shouldn't be used from multiple methods without resetting.
currentComment
- an OrderedCollection of strings which contain all comments between the current token and the previous token or the beginning of the source.
hereChar
- the current character
mark
- the position of the current token in the source stream
source
- the input stream of characters
token
- the current token
tokenType
- the type of the current token. The possible token types are: #binary, #character, #colon, #doIt, #keyword, #leftArrow, #leftBrace, #leftBracket, #leftParenthesis, #literal, #period, #rightBrace, #rightBracket, #rightParenthesis, #semicolon, #string, #upArrow, #verticalBar, #word, #xBinary, #xColon, #xDelimiter, #xDigit, #xDollar, #xDoubleQuote, #xLetter, #xLitQuote, #xSingleQuote, #xUnderscore
typeTable
- an array that maps each an evaluable tokenType to each character with asciiValue between 0 and 255
