Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | token: rewrite documentation for `LookupIdent`. | franck cuny | 2020-01-11 | 1 | -3/+4 |
| | |||||
* | token: support more operator tokens | franck cuny | 2020-01-11 | 1 | -3/+10 |
| | | | | | | Support additional tokens for operators (`-`, `*`, etc). This change only adds the tokens to the list of constants, and group all the tokens related to operators together. | ||||
* | token: initial tokenizer. | franck cuny | 2020-01-11 | 1 | -0/+48 |
This is the initial tokenizer for the monkey language. For now we recognize a limited number of tokens. We only have two keywords at this stage: `fn` and `let`. `fn` is used to create function, while `let` is used for assigning variables. The other tokens are mostly to parse the source code, and recognize things like brackets, parentheses, etc. |