Awesome Go

lingua-go

CategoryNatural Language Processing
SubcategoryLanguage Detection
Stars1,365
GitHubhttps://github.com/pemistahl/lingua-go

An accurate natural language detection library, suitable for long and short text alike. Supports detecting multiple languages in mixed-language text

About lingua-go

This large README is shown as a plain-text preview. Read the full README at the source

<div align="center">

  ![lingua](https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/logo.png)
  
  [![Build Status](https://github.com/pemistahl/lingua-go/workflows/build/badge.svg?branch=main)](https://github.com/pemistahl/lingua-go/actions?query=workflow%3A%22build%22+branch%3Amain)
  [![codecov](https://codecov.io/gh/pemistahl/lingua-go/branch/main/graph/badge.svg)](https://codecov.io/gh/pemistahl/lingua-go)
  [![supported languages](https://img.shields.io/badge/supported%20languages-75-green.svg)](#3-which-languages-are-supported)
  [![Go Reference](https://pkg.go.dev/badge/github.com/pemistahl/lingua-go.svg)](https://pkg.go.dev/github.com/pemistahl/lingua-go)
  [![Go Report Card](https://goreportcard.com/badge/github.com/pemistahl/lingua-go)](https://goreportcard.com/report/github.com/pemistahl/lingua-go)
  [![license](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)
</div>

<br>

## 1. What does this library do?

Its task is simple: It tells you which language some text is written in.
This is very useful as a preprocessing step for linguistic data in natural language
processing applications such as text classification and spell checking.
Other use cases, for instance, might include routing e-mails to the right geographically
located customer service department, based on the e-mails' languages.

## 2. Why does this library exist?

Language detection is often done as part of large machine learning frameworks or natural
language processing applications. In cases where you don't need the full-fledged
functionality of those systems or don't want to learn the ropes of those,
a small flexible library comes in handy.

So far, the only other comprehensive open source library in the Go ecosystem for
this task is [*Whatlanggo*](https://github.com/abadojack/whatlanggo).
Unfortunately, it has two major drawbacks:

1. Detection only works with quite lengthy text fragments. For very short text snippets
   such as Twitter messages, it does not provide adequate results.
2. The more languages take part in the decision process, the less accurate are the
   detection results.

*Lingua* aims at eliminating these problems. She nearly does not need any configuration and
yields pretty accurate results on both long and short text, even on single words and phrases.
She draws on both rule-based and statistical methods but does not use any dictionaries of words.
She does not need a connection to any external API or service either.
Once the library has been downloaded, it can be used completely offline.

## 3. Which languages are supported?

Compared to other language detection libraries, *Lingua's* focus is on *quality over quantity*, that is,
getting detection right for a small set of languages first before adding new ones.
Currently, the following 75 languages are supported:

- A
    - Afrikaans
    - Albanian
    - Arabic
    - Armenian
    - Azerbaijani
- B
    - Basque
    - Belarusian
    - Bengali
    - Norwegian Bokmal
    - Bosnian
    - Bulgarian
- C
    - Catalan
    - Chinese
    - Croatian
    - Czech
- D
    - Danish
    - Dutch
- E
    - English
    - Esperanto
    - Estonian
- F
    - Finnish
    - French
- G
    - Ganda
    - Georgian
    - German
    - Greek
    - Gujarati
- H
    - Hebrew
    - Hindi
    - Hungarian
- I
    - Icelandic
    - Indonesian
    - Irish
    - Italian
- J
    - Japanese
- K
    - Kazakh
    - Korean
- L
    - Latin
    - Latvian
    - Lithuanian
- M
    - Macedonian
    - Malay
    - Maori
    - Marathi
    - Mongolian
- N
    - Norwegian Nynorsk
- P
    - Persian
    - Polish
    - Portuguese
    - Punjabi
- R
    - Romanian
    - Russian
- S
    - Serbian
    - Shona
    - Slovak
    - Slovene
    - Somali
    - Sotho
    - Spanish
    - Swahili
    - Swedish
- T
    - Tagalog
    - Tamil
    - Telugu
    - Thai
    - Tsonga
    - Tswana
    - Turkish
- U
    - Ukrainian
    - Urdu
- V
    - Vietnamese
- W
    - Welsh
- X
    - Xhosa
- Y
    - Yoruba
- Z
    - Zulu

## 4. How good is it?

*Lingua* is able to report accuracy statistics for some bundled test data available for each
supported language. The test data for each language is split into three parts:

1. a list of single words with a minimum length of 5 characters
2. a list of word pairs with a minimum length of 10 characters
3. a list of complete grammatical sentences of various lengths

Both the language models and the test data have been created from separate documents of the
[Wortschatz corpora](https://wortschatz.uni-leipzig.de) offered by Leipzig University, Germany.
Data crawled from various news websites have been used for training, each corpus comprising one
million sentences. For testing, corpora made of arbitrarily chosen websites have been used,
each comprising ten thousand sentences. From each test corpus, a random unsorted subset of
1000 single words, 1000 word pairs and 1000 sentences has been extracted, respectively.

Given the generated test data, I have compared the detection results of *Lingua* and *Whatlanggo*
running over the data of *Lingua's* supported 75 languages. Additionally, I have added Google's 
[CLD3](https://github.com/google/cld3/) to the comparison with the help of the 
[gocld3](https://github.com/jmhodges/gocld3) bindings. Languages that are not supported
by *CLD3* or *Whatlanggo* are simply ignored during the detection process.

Each of the following sections contains two plots. The bar plot shows the detailed accuracy
results for each supported language. The box plot illustrates the distributions of the
accuracy values for each classifier. The boxes themselves represent the areas which the
middle 50 % of data lie within. Within the colored boxes, the horizontal lines mark the
median of the distributions.

### 4.1 Single word detection

<br/>

<img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/plots/boxplot-single-words.png" alt="Single Word Detection Performance" />

<br/>

<details>
    <summary>Bar plot</summary>
    <img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/plots/barplot-single-words.png" alt="Single Word Detection Performance" />
</details>

<br/><br/>

### 4.2 Word pair detection

<br/>

<img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/plots/boxplot-word-pairs.png" alt="Word Pair Detection Performance" />

<br/>

<details>
    <summary>Bar plot</summary>
    <img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/plots/barplot-word-pairs.png" alt="Word Pair Detection Performance" />
</details>

<br/><br/>

### 4.3 Sentence detection

<br/>

<img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/plots/boxplot-sentences.png" alt="Sentence Detection Performance" />

<br/>

<details>
    <summary>Bar plot</summary>
    <img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/plots/barplot-sentences.png" alt="Sentence Detection Performance" />
</details>

<br/><br/>

### 4.4 Average detection

<br/>

<img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/plots/boxplot-average.png" alt="Average Detection Performance" />

<br/>

<details>
    <summary>Bar plot</summary>
    <img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/plots/barplot-average.png" alt="Average Detection Performance" />
</details>

<br/><br/>

### 4.5 Mean, median and standard deviation

The table below shows detailed statistics for each language and classifier
including mean, median and standard deviation.

<details>
  <summary>Open table</summary>
  <table>
    <tr>
        <th>Language</th>
        <th colspan="4">Average</th>
        <th colspan="4">Single Words</th>
        <th colspan="4">Word Pairs</th>
        <th colspan="4">Sentences</th>
    </tr>
    <tr>
        <th></th>
        <th>Lingua<br>(high accuracy mode)</th>
        <th>Lingua<br>(low accuracy mode)</th>
        <th>&nbsp;&nbsp;CLD3&nbsp;&nbsp;</th>
        <th>Whatlang</th>
        <th>Lingua<br>(high accuracy mode)</th>
        <th>Lingua<br>(low accuracy mode)</th>
        <th>&nbsp;&nbsp;CLD3&nbsp;&nbsp;</th>
        <th>Whatlang</th>
        <th>Lingua<br>(high accuracy mode)</th>
        <th>Lingua<br>(low accuracy mode)</th>
        <th>&nbsp;&nbsp;CLD3&nbsp;&nbsp;</th>
        <th>Whatlang</th>
        <th>Lingua<br>(high accuracy mode)</th>
        <th>Lingua<br>(low accuracy mode)</th>
        <th>&nbsp;&nbsp;CLD3&nbsp;&nbsp;</th>
        <th>Whatlang</th>
    </tr>
    <tr>
		<td>Afrikaans</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 79</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 64</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/yellow.png"> 55</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/yellow.png"> 51</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/yellow.png"> 58</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/orange.png"> 38</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/orange.png"> 22</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/orange.png"> 21</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 81</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 62</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/yellow.png"> 46</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/orange.png"> 39</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 97</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 93</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 98</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 92</td>
	</tr>
	<tr>
		<td>Albanian</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 88</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 80</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/yellow.png"> 55</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 69</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/yellow.png"> 54</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/red.png"> 18</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 95</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 86</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/yellow.png"> 48</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 99</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 98</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
	</tr>
	<tr>
		<td>Arabic</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 98</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 94</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 90</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 89</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 96</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 88</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 79</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 77</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 99</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 96</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 92</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 91</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 99</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 99</td>
	</tr>
	<tr>
		<td>Armenian</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 99</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 100</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 97</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
	</tr>
	<tr>
		<td>Azerbaijani</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 90</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 82</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 81</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 64</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 77</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 71</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 62</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/yellow.png"> 45</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 92</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 78</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 82</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/yellow.png"> 58</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 99</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 96</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 99</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 91</td>
	</tr>
	<tr>
		<td>Basque</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 84</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 75</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 62</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 71</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/yellow.png"> 56</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/orange.png"> 33</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 87</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 76</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 62</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 93</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 92</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 92</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/grey.png"> -</td>
	</tr>
	<tr>
		<td>Belarusian</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 97</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 92</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 84</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 81</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/green.png"> 92</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 80</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 67</td>
		<td><img src="https://raw.githubusercontent.com/pemistahl/lingua-go/main/cmd/images/lightgreen.png"> 64</td>

Frequently Asked Questions

What is lingua-go?

lingua-go is a Natural Language Processing library for the Go programming language. An accurate natural language detection library, suitable for long and short text alike. Supports detecting multiple languages in mixed-language text

How many GitHub stars does lingua-go have?

lingua-go has 1,365 GitHub stars in the directory's latest synchronization.

How do I install lingua-go?

Install lingua-go with the Go module system using `go get pemistahl/lingua-go`. Check the repository for the current installation instructions.

What category does lingua-go belong to?

lingua-go is listed under Natural Language Processing, specifically Language Detection.

← Back to Language Detection