class documentation
class CharTokenizer(StringTokenizer): (source)
Tokenize a string into individual characters. If this functionality is ever required directly, use for char in string.
| Method | span |
Identify the tokens using integer offsets (start_i, end_i), where s[start_i:end_i] is the corresponding token. |
| Method | tokenize |
Return a tokenized copy of s. |
Inherited from StringTokenizer:
| Property | _string |
Undocumented |
Inherited from TokenizerI (via StringTokenizer):
| Method | span |
Apply self.span_tokenize() to each element of strings. I.e.: |
| Method | tokenize |
Apply self.tokenize() to each element of strings. I.e.: |