udapi.block.tokenize package¶
Submodules¶
udapi.block.tokenize.onwhitespace module¶
Block tokenize.OnWhitespace
udapi.block.tokenize.simple module¶
Block tokenize.Simple
-
class
udapi.block.tokenize.simple.
Simple
(zones='all')[source]¶ Bases:
udapi.block.tokenize.onwhitespace.OnWhitespace
Simple tokenizer, splits on whitespaces and punctuation, fills SpaceAfter=No.