Public concept · ec02219e-ffbb-46f8-ab5f-7749b09f2b15
tokenization
German
de
TokenisierungPreferred
English
en
tokenisationAdmitted
tokenizationPreferred
“breaking a stream of text up into chunks for analysis or further processing”
Japanese
ja
トークン化Preferred
Chinese (China)
zh-CN
词元化Preferred