BaseTokenizer
Common methods and variables of all tokenizers.
Gets the current character.
Gets the current insertion point.
Gets if the current index has been normalized (CRLF -> LF).
Gets the current source index.
Gets the allocated string buffer.
Creates a new instance of the base tokenizer.
Advances in the source by 1 character if possible.
Advances in the source by n characters or less if possible.
Goes back in the source by 1 character if possible.
Goes back in the source by n characters or less if possible.
Checks if the source continues with the given string.
The comparison is case-insensitive.
Checks if the source continues with the given string.
The comparison is case-sensitive.
Disposes the tokenizer by releasing the buffer.
Flushes the buffer.
Gets the current text position in the source.
Gets the next character in the source by advancing.
Gets the previous character in the source by going back.
Gets the string formed by the next characters.
Skips the next space character(s).