about summary refs log tree commit diff
path: root/users/fcuny/exp/monkey/pkg/token (follow)
Commit message (Collapse)AuthorAgeFilesLines
* token: rewrite documentation for `LookupIdent`.franck cuny2020-01-111-3/+4
|
* token: support more operator tokensfranck cuny2020-01-111-3/+10
| | | | | | Support additional tokens for operators (`-`, `*`, etc). This change only adds the tokens to the list of constants, and group all the tokens related to operators together.
* token: initial tokenizer.franck cuny2020-01-111-0/+48
This is the initial tokenizer for the monkey language. For now we recognize a limited number of tokens. We only have two keywords at this stage: `fn` and `let`. `fn` is used to create function, while `let` is used for assigning variables. The other tokens are mostly to parse the source code, and recognize things like brackets, parentheses, etc.