This article investigates the morphological and orthographic compatibility of standard tokenization algorithms with the agglutinative structure of the Uzbek language. Through empirical evaluation of 15 transformer-based language models—including global large language models, multilingual encoders,...

This article discusses the development of a specialized digital dataset for training intelligent syntactic analysis models for the Uzbek language. One of the main challenges in Uzbek Natural Language Processing (NLP) is the lack of high-quality, linguistically annotated corpora that account for the...

Tokenization is the process of dividing a text into smaller parts, called tokens. Tokens can be words, punctuation marks, numbers, or other meaningful elements. Tokenization is primarily used in natural language processing (NLP) and is an essential first step for analyzing, understanding, or...