General Purpose Language Identification For Ethiopia Semitic Language Using Hybrid Approach

Abstract

Due to many sophisticated and advanced technologies like the Internet, the world has become a single village. It is possible to get a vast amount of digitized information that are generated, propagated, exchanged, stored and accessed through the internet and other media like mobile network each day across the world. The accumulation of digital data is making information acquisition increasingly difficult, with natural language becoming critically an obstacle. The step towards tackling this obstacle is Natural Language Processing and language identification is the first step among many steps that are used for information acquisition and other advanced NLP applications. It is a technique of labeling each word in a text or sentence with its corresponding language category. In past decades a number of research works have been done in the area of language identification. However, there are issues which are not solved until: multilingual language identification, discriminating the language category of very closely related language documents and labelling the language category for very short texts like words or phrases. In addition to this, as far as the researcher’s knowledge is concerned, there is no language identifier developed for Ethiopian Semitic language though there are many language identifier developed using different approaches for many European languages and resourced languages. In this investigation, we propose a hybrid approach; character ngram and word ngram combined with rule based approach. Which can able to solve these mentioned unsolved issues of language identification on top of Ethiopian Semitic languages (i.e. Amharic, Geeze, Guragigna and Tigrigna). The proposed general purpose language identifier approach has a capability of identify the language of a text at any level (i.e. Word, phrase, sentence and document) in both monolingual as well as multilingual setting. The reason behind this capability of proposed approach is due to the features of word level language identification, in which every words needs to classify with regard to its language category at a time. Text is first pass through preprocessing steps. Then pass through rule based approach word which can handle through rule. Afterwards word ngram of previse word language is conducts, if word not exist, Character ngram (infinite ngram) with location is calculated; afterwards the ngram probability is calculated and ngram probability of word is calculated, which is used to assign a language label for that word. Finally sentence and document reformation is done for all texts.