This article provides a description of a tokenizer used in my toy language project. If you are new to this series, I would recommend to start reading with*this post.Please note that this article describes an initial implementation of the tokenizer. If you want to browse the code while reading the ...
More...