BaseTokenizer
Common methods and variables of all tokenizers.
The event will be fired once an error has been detected.
Gets the current position.
Fires an error occurred event.
Fires an error occurred event.