Go bindings for the snowball libstemmer library including porter 2

Go (golang) bindings for libstemmer
===================================

This simple library provides Go (golang) bindings for the snowball libstemmer library including the popular porter and porter2 algorithms.

Requirements
------------

You'll need the development package of libstemmer, usually this is simply a matter of:

    sudo apt-get install libstemmer-dev

... or you might need to [install it from source](http://snowball.tartarus.org/).

Installation
------------

First, ensure you have your GOPATH env variable set to the root of your Go project:

    export GOPATH=`pwd`
    export PATH=$PATH:$GOPATH/bin

Then this cute statement should do the trick:

    go get github.com/rjohnsondev/golibstemmer

Usage
-----

Basic usage:

    package main

    import "github.com/rjohnsondev/golibstemmer"
    import "fmt"
    import "os"

    func main() {
        s, err := stemmer.NewStemmer("english")
        defer s.Close()
        if err != nil {
            fmt.Println("Error creating stemmer: "+err.Error())
            os.Exit(1)
        }
        word := s.StemWord("happy")
        fmt.Println(word)
    }

To get a list of supported stemming algorithms:

    list := stemmer.GetSupportedLanguages()

Testing
-------

You can execute the basic included tests with:

    go test

If you have issues, double check you've installed the libstemmer development library.

If you still have issues, let me know!


golibstemmer

Sentiment analyzer using sentiwordnet lexicon in Go

# 💬 GoSentiwordnet

[![PkgGoDev](https://pkg.go.dev/badge/github.com/dinopuguh/gosentiwordnet/v2)](https://pkg.go.dev/github.com/dinopuguh/gosentiwordnet/v2)
[![Unit Test Status](https://github.com/dinopuguh/gosentiwordnet/actions/workflows/unit-test.yml/badge.svg?branch=master)](https://github.com/dinopuguh/gosentiwordnet/actions) 
[![Go Report Card](https://goreportcard.com/badge/github.com/dinopuguh/gosentiwordnet)](https://goreportcard.com/report/github.com/dinopuguh/gosentiwordnet)
[![codecov](https://codecov.io/gh/dinopuguh/gosentiwordnet/branch/master/graph/badge.svg)](https://codecov.io/gh/dinopuguh/gosentiwordnet)

Sentiment analyzer using [sentiwordnet](https://github.com/aesuli/SentiWordNet) lexicon in Go. This library produce sentiment score for each word, including positive, negative, and objective score.

## ⚙ Installation

First of all, [download](https://golang.org/dl/) and install Go `1.14` or higher is required.

Install this library using the [`go get`](https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them) command:

```bash
$ go get github.com/dinopuguh/gosentiwordnet/v2
```

## ⚡ Quickstart

```go
package main

import (
    "fmt"

    goswn "github.com/dinopuguh/gosentiwordnet/v2"
)

func main() {
    sa := goswn.New()

    scores, exist := sa.GetSentimentScore("love", "v", "2")
    if exist {
        fmt.Println("💬 Sentiment score:", scores) // => 💬 Sentiment score: {1 0 0}
    }
}
```

The `GetSentimentScore` required 3 parameters(word, pos-tag, and word usage):

1. **Word**: the word want to process
2. **POS tag**: part-of-speech tag of the word
3. **Word usage**: 1 for most common usage and a higher number would indicate lesser common usages

## 👍 Contributing

If you want to say **thank you** and/or support the active development of `Gosentiwordnet`:

1. Add a [GitHub Star](https://github.com/dinopuguh/gosentiwordnet/stargazers) to the project.
2. Write a review or tutorial on [Medium](https://medium.com/), [Dev.to](https://dev.to/) or personal blog.
3. Be a part of our [sponsors](https://github.com/sponsors/dinopuguh) to support this project.

## 💻 Contributors

- Dino Puguh (initial works)

Open for any pull requests to develop this project.


gosentiwordnet

Go implementation of [VADER Sentiment Analysis](https://github.com/cjhutto/vaderSentiment)

# GoVader

[![GoDoc](https://godoc.org/github.com/jonreiter/govader?status.svg)](https://godoc.org/github.com/jonreiter/govader)
[![Go Report](https://goreportcard.com/badge/github.com/jonreiter/govader)](https://goreportcard.com/badge/github.com/jonreiter/govader)
[![BuildStatus](https://www.travis-ci.com/jonreiter/govader.svg?branch=master)](https://www.travis-ci.com/github/jonreiter/govader/branches)
[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge-flat.svg)](https://github.com/avelino/awesome-go)
[![codecov](https://codecov.io/gh/jonreiter/govader/branch/master/graph/badge.svg)](https://codecov.io/gh/jonreiter/govader)

GoVader: Vader sentiment analysis in Go

This is a port of [https://github.com/cjhutto/vaderSentiment](https://github.com/cjhutto/vaderSentiment) from
Python to Go.

There are tests which check it gives the same answers as the original package.

Usage:

```go
import (
    "fmt"
    "github.com/jonreiter/govader"
)

analyzer := govader.NewSentimentIntensityAnalyzer()
sentiment := analyzer.PolarityScores("Usage is similar to all the other ports.")

fmt.Println("Compound score:", sentiment.Compound)
fmt.Println("Positive score:", sentiment.Positive)
fmt.Println("Neutral score:", sentiment.Neutral)
fmt.Println("Negative score:", sentiment.Negative)

```
A server wrapper is available in https://github.com/PIMPfiction/govader_backend.


govader

Microservice implementation of [GoVader](https://github.com/jonreiter/govader)

# Govader Backend NLP Package (sentimental analysis NLP)

[![Go Reference](https://pkg.go.dev/badge/github.com/PIMPfiction/govader_backend.svg)](https://pkg.go.dev/github.com/PIMPfiction/govader_backend)
[![Go Report Card](https://goreportcard.com/badge/github.com/PIMPfiction/govader_backend)](https://goreportcard.com/report/github.com/PIMPfiction/govader_backend)
[![codecov](https://codecov.io/gh/PIMPfiction/govader_backend/branch/master/graph/badge.svg?token=3KEBD30Q95)](https://codecov.io/gh/PIMPfiction/govader_backend)
![master](https://github.com/PIMPfiction/govader_backend/actions/workflows/tests.yml/badge.svg)
[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go#morphological-analyzers)  

Govader-Backend is a microservice thats returns sentimental analysis of given sentence.

Server wrapper for GoVader Package[https://github.com/jonreiter/govader](https://github.com/jonreiter/govader)


## Usage:

```sh
go get github.com/PIMPfiction/govader_backend
```

```go
package main

import (
	vaderMicro "github.com/PIMPfiction/govader_backend"
	echo "github.com/labstack/echo/v4"
	"fmt"
)

func main() {
	e := echo.New()
	err := vaderMicro.Serve(e, "8080")
	if err != nil {
		panic(err)
	}
	fmt.Scanln()

}

```


### Sample Get Request:

#### GET: http://localhost:8080?text=I%20am%20looking%20good

### Sample Post Request:

#### POST: http://localhost:8080/
#### RequestBody: ```{"text": "I am looking good"}```


### Sample Response 
```json
{
  "Negative": 0,
  "Neutral": 0.5084745762711864,
  "Positive": 0.4915254237288135,
  "Compound": 0.44043357076016854
}
```


govader-backend

Golang implementation of the Paice/Husk Stemming Algorithm

##Golang Implementation of the Paice/Husk stemming algorithm
This project was created for the [QUT](http://www.qut.edu.au/ "Queensland University of Technology") course [INB344](http://www.qut.edu.au/study/unit-search/unit?unitCode=INB344&idunit=43393). Details on the algorithm can be found [here](http://www.comp.lancs.ac.uk/computing/research/stemming/Links/paice.htm "Lancaster School of Computing"). This implementation is primarily based on the [ANSI C Implementationn](http://www.comp.lancs.ac.uk/computing/research/stemming/Links/implementations.htm) by Andy Stark. Effort has been put into the correctness of the algorithm, but this is hampered by many of the existing implementations giving differing results. Any comments/assistance/pull-requests are welcome.

##TODO
* Benchmarks

##Demo
A demo App Engine project utilizing this package exists [here](http://paicehusk.appspot.com/).


paicehusk

Go port of the Rapid Automatic Keyword Extraction Algorithm (RAKE)

A Go implementation of the Rapid Automatic Keyword Extraction (RAKE) algorithm as described in: Rose, S., Engel, D., Cramer, N., & Cowley, W. (2010). Automatic Keyword Extraction from Individual Documents. In M. W. Berry & J. Kogan (Eds.), Text Mining: Theory and Applications: John Wiley & Sons.

Original Python implementation available at: https://github.com/aneesha/RAKE

The source code is released under the MIT License.

## Docs and Report Card
- godoc.org: https://godoc.org/github.com/afjoseph/RAKE.Go
- goreportcard.com: https://goreportcard.com/report/github.com/afjoseph/RAKE.Go

## Example Usage

```go
package main

import (
	"github.com/afjoseph/RAKE.go"
	"fmt"
)

func main() {
	text := `The growing doubt of human autonomy and reason has created a state of moral confusion where man is left without the guidance of either revelation or reason. The result is the acceptance of a relativistic position which proposes that value judgements and ethical norms are exclusively matters of arbitrary preference and that no objectively valid statement can be made in this realm... But since man cannot live without values and norms, this relativism makes him an easy prey for irrational value systems.`

	candidates := rake.RunRake(text)

	for _, candidate := range candidates {
		fmt.Printf("%s --> %f\n", candidate.Key, candidate.Value)
	}

	fmt.Printf("\nsize: %d\n", len(candidates))
}




 9.000000-->
 4.000000-->
 4.000000-->
 4.000000-->
 4.000000-->
 4.000000-->
 4.000000-->
 4.000000-->
 4.000000-->
 3.500000-->
 1.500000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->
 1.000000-->


```

RAKE.go

Snowball stemmer port (cgo wrapper) for Go. Provides word stem extraction functionality [Snowball native](http://snowball.tartarus.org/)

Description
====

Snowball stemmer port (cgo wrapper) for Go. Provides word stem extraction functionality. For more detailed info see http://snowball.tartarus.org/

Installing
====

```
go get github.com/goodsign/snowball
go test github.com/goodsign/snowball (Must PASS)
```

Done! Use it in your go files. (import 'github.com/goodsign/snowball')

Usage
====

```go
  stemmer, err := NewWordStemmer(algorithm, encoding)
  
  if nil != err {
    /*...handle error...*/
  }
  defer stemmer.Close() 

  wordStem, err := stemmer.Stem(word)
  if nil != err {
    /*...handle error...*/
  }

  /* Use wordStem */

```
Usage notes
-----------

According to Snowball documentation:

```
Creating a stemmer is a relatively expensive operation - the expected
usage pattern is that a new stemmer is created when needed, used
to stem many words, and deleted after some time.
```

Algorithms & encodings
----

File **modules.txt** contains all the main algorithms for each language, in UTF-8, and also with
the most commonly used encoding.

```
Language        Encodings               Algorithms

danish          UTF_8,ISO_8859_1        danish,da,dan
dutch           UTF_8,ISO_8859_1        dutch,nl,dut,nld
english         UTF_8,ISO_8859_1        english,en,eng
finnish         UTF_8,ISO_8859_1        finnish,fi,fin
french          UTF_8,ISO_8859_1        french,fr,fre,fra
german          UTF_8,ISO_8859_1        german,de,ger,deu
hungarian       UTF_8,ISO_8859_1        hungarian,hu,hun
italian         UTF_8,ISO_8859_1        italian,it,ita
norwegian       UTF_8,ISO_8859_1        norwegian,no,nor
portuguese      UTF_8,ISO_8859_1        portuguese,pt,por
romanian        UTF_8,ISO_8859_2        romanian,ro,rum,ron
russian         UTF_8,KOI8_R            russian,ru,rus
spanish         UTF_8,ISO_8859_1        spanish,es,esl,spa
swedish         UTF_8,ISO_8859_1        swedish,sv,swe
turkish         UTF_8                   turkish,tr,tur
```

Thread-safety
====

The original Snowball documentation says:

```
Stemmers are re-entrant, but not threadsafe.  In other words, if
you wish to access the same stemmer object from multiple threads,
you must ensure that all access is protected by a mutex or similar
device.
```

Thus this Go wrapper uses **sync.Mutex** for each stem operation, so it is thread safe.

Snowball Licence
==========

The Snowball library is released under the [BSD Licence](http://opensource.org/licenses/bsd-license.php)

Licence
==========

The goodsign/snowball binding is released under the [BSD Licence](http://opensource.org/licenses/bsd-license.php)

[LICENCE file](https://github.com/goodsign/libtextcat/blob/master/LICENCE)

snowball

Self-contained Machine Learning and Natural Language Processing library in Go

<img src="https://github.com/nlpodyssey/spago/blob/main/assets/spago_logo.png" width="400"/>
 


 <a href="https://github.com/nlpodyssey/spago/actions/workflows/go.yml?query=branch%3Amain">
 <img alt="Build" src="https://github.com/nlpodyssey/spago/actions/workflows/go.yml/badge.svg?branch=main">
 </a>
 <a href="https://codecov.io/gh/nlpodyssey/spago">
 <img alt="Coverage" src="https://codecov.io/gh/nlpodyssey/spago/branch/main/badge.svg">
 </a>
 <a href="https://goreportcard.com/report/github.com/nlpodyssey/spago">
 <img alt="Go Report Card" src="https://goreportcard.com/badge/github.com/nlpodyssey/spago">
 </a>
 <a href="https://codeclimate.com/github/nlpodyssey/spago/maintainability">
 <img alt="Maintainability" src="https://api.codeclimate.com/v1/badges/be7350d3eb1a6a8aa503/maintainability">
 </a>
 <a href="https://pkg.go.dev/github.com/nlpodyssey/spago/">
 <img alt="Documentation" src="https://pkg.go.dev/badge/github.com/nlpodyssey/spago/.svg">
 </a>
 <a href="https://opensource.org/licenses/BSD-2-Clause">
 <img alt="License" src="https://img.shields.io/badge/License-BSD%202--Clause-orange.svg">
 </a>
 <a href="http://makeapullrequest.com">
 <img alt="PRs Welcome" src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square">
 </a>
 <a href="https://github.com/avelino/awesome-go">
 <img alt="Awesome Go" src="https://awesome.re/mentioned-badge.svg">
 </a>


 
 If you like the project, please ★ star this repository to show your support! 🤩
 
 


> 15 Jan 2024 - As I reflect on the journey of Spago, I am filled with gratitude for the enriching experience it has provided me. Mastering Go and revisiting the fundamentals of Deep Learning through Spago has been immensely rewarding. The unique features of Spago, especially its asynchronous computation graph and focusing on clean coding, have made it an extraordinary project to work on. Our goal was to create a minimalist ML framework in Go, eliminating the dependency on Python in production by enabling the creation of standalone executables. This approach of Spago successfully powered several of my projects in challenging production environments.
> 
> However, the endeavor to elevate Spago to a level where it can compete effectively in the evolving 'AI space', which now extensively involves computation on GPUs, requires substantial commitment. At the same time, the vision that Spago aspired to achieve is now being impressively realized by the [Candle](https://github.com/huggingface/candle) project in Rust. With my limited capacity to dedicate the necessary attention to Spago, and in the absence of a supporting maintenance team, I have made the pragmatic decision to pause the project for now.
>
> I am deeply grateful for the journey Spago has taken me on and for the community that has supported it. As we continue to explore the ever-evolving field of machine learning, I look forward to the exciting developments that lie ahead.
> 
> Warm regards,
>
> Matteo Grella

---

Spago is a **Machine Learning** library written in pure Go designed to support relevant neural architectures in **Natural
Language Processing**.

Spago is self-contained, in that it uses its own lightweight computational graph both for training and
inference, easy to understand from start to finish. 

It provides:
- Automatic differentiation via dynamic define-by-run execution
- Feed-forward layers (Linear, Highway, Convolution...)
- Recurrent layers (LSTM, GRU, BiLSTM...)
- Attention layers (Self-Attention, Multi-Head Attention...)
- Gradient descent optimizers (Adam, RAdam, RMS-Prop, AdaGrad, SGD)
- Gob compatible neural models for serialization

If you're interested in NLP-related functionalities, be sure to explore the [Cybertron](https://github.com/nlpodyssey/cybertron) package!

## Usage

Requirements:

* [Go 1.21](https://golang.org/dl/)

Clone this repo or get the library:

```console
go get -u github.com/nlpodyssey/spago
```

### Getting Started

A good place to start is by looking at the implementation of built-in neural models, such as the LSTM.

### Example 1
Here is an example of how to calculate the sum of two variables:

```go
package main

import (
	"fmt"
	"log"

	"github.com/nlpodyssey/spago/ag"
	"github.com/nlpodyssey/spago/mat"
)

func main() {
	// define the type of the elements in the tensors
	type T = float32

	// create a new node of type variable with a scalar
	a := mat.Scalar(T(2.0), mat.WithGrad(true)) // create another node of type variable with a scalar
	b := mat.Scalar(T(5.0), mat.WithGrad(true)) // create an addition operator (the calculation is actually performed here)
	c := ag.Add(a, b)

	// print the result
	fmt.Printf("c = %v (float%d)\n", c.Value(), c.Value().Item().BitSize())

	c.AccGrad(mat.Scalar(T(0.5)))

	if err := ag.Backward(c); err != nil {
		log.Fatalf("error during Backward(): %v", err)
	}

	fmt.Printf("ga = %v\n", a.Grad())
	fmt.Printf("gb = %v\n", b.Grad())
}
```

Output:

```console
c = [7] (float32)
ga = [0.5]
gb = [0.5]
```

### Example 2

Here is a simple implementation of the perceptron formula:

```go
package main

import (
	"fmt"
	
	. "github.com/nlpodyssey/spago/ag"
	"github.com/nlpodyssey/spago/mat"
)

func main() {
	x := mat.Scalar(-0.8)
	w := mat.Scalar(0.4)
	b := mat.Scalar(-0.2)

	y := Sigmoid(Add(Mul(w, x), b))

	fmt.Printf("y = %0.3f\n", y.Value().Item())
}
```

## Contributing

If you think something is missing or could be improved, please open issues and pull requests.

To start contributing, check the [Contributing Guidelines](https://github.com/nlpodyssey/spago/blob/main/CONTRIBUTING.md).

## Contact

We highly encourage you to create an issue as it will contribute to the growth of the community. However, if you prefer to communicate with us privately, please feel free to email [Matteo Grella](mailto:matteogrella@gmail.com) with any questions or comments you may have.

spaGO

Implementation of the porter stemming algorithm

go implementation of the porter stemming algorithm: http://tartarus.org/~martin/PorterStemmer/index.html

go-stem

Cgo binding for libtextcat C library. Guaranteed compatibility with version 2.2

About
==========

Cgo binding for libtextcat C library. Guaranteed compatibility with version 2.2.

Installation
==========

Installation consists of several simple steps. They may be a bit different on your target system (e.g. require more permissions) so adapt them to the parameters of your system.

### Get libtextcat C library code

* Download original libtextcat archive from [libtextcat download section](http://software.wise-guys.nl/libtextcat/). 
* Unarchive it.

NOTE: If this link is not working or there are some problems with downloading, there is a stable version 2.2 snapshot saved in [Downloads](https://github.com/downloads/goodsign/libtextcat/libtextcat-2.2.tar.gz).

### Build and install libtextcat C library

From the directory, where you unarchived libtextcat, run:

```
./configure
make
sudo make install
sudo ldconfig 
```

### Install Go wrapper

```
go get github.com/goodsign/libtextcat
go test github.com/goodsign/libtextcat (must PASS)
```

Installation notes
==========

Make sure that you have your local library paths set correctly and that installation was successful. Otherwise, **go build** or **go test** may fail.

libtextcat is installed in your local library directory (e.g. **/usr/local/lib**) and puts its libraries there. This path should be registered in your system (using ldconfig or exporting LD_LIBRARY_PATH, etc.) or the linker would fail.

Usage
==========

```go
cat, err := NewTextCat(ConfigPath) // See 'Usage notes' section

if nil != err {
    // ... Handle error ...
}
defer cat.Close()

matches, err := cat.Classify(text)

if nil != err {
    // ... Handle error ...
}

// Use matches. 
// NOTE: matches[0] is the best match.

```

Usage notes
==========

libtextcat library needs to load language models to start guessing languages. These models are set using a configuration file and a number of language model (.lm) files.

Configuration file maps .lm files to identifiers used in the library. See [example](https://github.com/goodsign/libtextcat/blob/master/defaultcfg/conf.txt). Path to this file is specified in the **NewTextCat** call.

.lm files contain language patterns and frequencies for a specified language. See [example](https://github.com/goodsign/libtextcat/blob/master/defaultcfg/english.lm). Paths to these files are specified in the config file above. They can be absolute or relative (to the caller).

Quickstart
----------

To immediately get started, copy **/defaultcfg** folder contents to the directory of your target project and use:

```go
cat, err := NewTextCat("defaultcfg/conf.txt")
```

This will give you a standard set of languages described in the **Default configuration** section below.

Default configuration
----------

This package contains a default configuration (/defaultcfg) which is created to work in following conditions:

* Utf-8 only languages
* Language list is taken from [snowball](github.com/goodsign/snowball) package
* Language identifiers are the same as in [snowball](github.com/goodsign/snowball) package

This configuration is meant to be used in pair with the [snowball](github.com/goodsign/snowball) package.

More info
----------

For more information on libtextcat refer to the original [website](http://software.wise-guys.nl/libtextcat/), which contains links on theory and other details.

libtextcat Licence
==========

The libtextcat library is released under the [BSD Licence](http://opensource.org/licenses/bsd-license.php)

[LICENCE file](https://github.com/goodsign/libtextcat/blob/master/LICENCE_libtextcat)

Licence
==========

The goodsign/libtextcat binding is released under the [BSD Licence](http://opensource.org/licenses/bsd-license.php)

[LICENCE file](https://github.com/goodsign/libtextcat/blob/master/LICENCE)

libtextcat

Go Natural Language Processing library supporting LSA (Latent Semantic Analysis)

# Natural Language Processing 
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) 
[![GoDoc](https://godoc.org/github.com/james-bowman/nlp?status.svg)](https://godoc.org/github.com/james-bowman/nlp) 
[![Build Status](https://travis-ci.org/james-bowman/nlp.svg?branch=master)](https://travis-ci.org/james-bowman/nlp)
[![Go Report Card](https://goreportcard.com/badge/github.com/james-bowman/nlp)](https://goreportcard.com/report/github.com/james-bowman/nlp)
[![codecov](https://codecov.io/gh/james-bowman/nlp/branch/master/graph/badge.svg)](https://codecov.io/gh/james-bowman/nlp)
[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge-flat.svg)](https://github.com/avelino/awesome-go)
[![Sourcegraph](https://sourcegraph.com/github.com/james-bowman/nlp/-/badge.svg)](https://sourcegraph.com/github.com/james-bowman/nlp?badge)


<img src="https://github.com/james-bowman/nlp/raw/master/Gophers.008.crop.png" alt="nlp" align="left" />

Implementations of selected machine learning algorithms for natural language processing in golang. The primary focus for the package is the statistical semantics of plain-text documents supporting semantic analysis and retrieval of semantically similar documents.

Built upon the [Gonum](https://www.gonum.org/) package for linear algebra and scientific computing with some inspiration taken from Python's [scikit-learn](http://scikit-learn.org/stable/) and [Gensim](https://radimrehurek.com/gensim/).

Check out [the companion blog post](http://www.jamesbowman.me/post/semantic-analysis-of-webpages-with-machine-learning-in-go/) or [the Go documentation page](https://godoc.org/github.com/james-bowman/nlp) for full usage and examples.

 

## Features

* [LSA (Latent Semantic Analysis aka Latent Semantic Indexing (LSI))][LSA] implementation using truncated [SVD (Singular Value Decomposition)](https://en.wikipedia.org/wiki/Singular-value_decomposition) for dimensionality reduction.
* Fast comparison and retrieval of semantically similar documents using [SimHash](https://en.wikipedia.org/wiki/SimHash)(random hyperplanes/[sign random projection](https://en.wikipedia.org/wiki/Locality-sensitive_hashing#Random_projection)) algorithm with multi-index and Forest schemes for [LSH (Locality Sensitive Hashing)](https://en.wikipedia.org/wiki/Locality-sensitive_hashing) to support fast, approximate cosine similarity/angular distance comparisons and approximate nearest neighbour search using significantly less memory and processing time.
* [Random Indexing (RI)](https://en.wikipedia.org/wiki/Random_indexing) and Reflective Random Indexing (RRI) (which extends RI to support indirect inference) for scalable [Latent Semantic Analysis (LSA)][LSA] over large, web-scale corpora.
* [Latent Dirichlet Allocation (LDA)](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) using a parallelised implementation of the fast [SCVB0 (Stochastic Collapsed Variational Bayesian inference)][SCVB0] algorithm for unsupervised topic extraction. 
* [PCA (Principal Component Analysis)](https://en.wikipedia.org/wiki/Principal_component_analysis)
* [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) weighting to account for frequently occuring words
* [Sparse matrix](http://github.com/james-bowman/sparse) implementations used for more efficient memory usage and processing over large document corpora.
* Stop word removal to remove frequently occuring English words e.g. "the", "and"
* [Feature hashing](https://en.wikipedia.org/wiki/Feature_hashing) ('the hashing trick') implementation (using [MurmurHash3](http://github.com/spaolacci/murmur3)) for reduced memory requirements and reduced reliance on training data
* Similarity/distance measures to calculate the similarity/distance between feature vectors.

## Planned

* Expanded persistence support
* Stemming to treat words with common root as the same e.g. "go" and "going"
* Clustering algorithms e.g. Heirachical, K-means, etc.
* Classification algorithms e.g. SVM, KNN, random forest, etc.

## References

1. [Rosario, Barbara. Latent Semantic Indexing: An overview. INFOSYS 240 Spring 2000](http://people.ischool.berkeley.edu/~rosario/projects/LSI.pdf)
1. [Latent Semantic Analysis, a scholarpedia article on LSA written by Tom Landauer, one of the creators of LSA.](http://www.scholarpedia.org/article/Latent_semantic_analysis)
1. [Thomo, Alex. Latent Semantic Analysis (Tutorial).](http://webhome.cs.uvic.ca/~thomo/svd.pdf)
1. [Latent Semantic Indexing. Standford NLP Course](http://nlp.stanford.edu/IR-book/html/htmledition/latent-semantic-indexing-1.html)
1. [Charikar, Moses S. "Similarity Estimation Techniques from Rounding Algorithms" in Proceedings of the thiry-fourth annual ACM symposium on Theory of computing - STOC ’02, 2002, p. 380.](https://www.cs.princeton.edu/courses/archive/spr04/cos598B/bib/CharikarEstim.pdf)
1. [M. Bawa, T. Condie, and P. Ganesan, “LSH forest: self-tuning indexes for similarity search,” Proc. 14th Int. Conf. World Wide Web - WWW ’05, p. 651, 2005.](http://dl.acm.org/citation.cfm?id=1060745.1060840)
1. [A. Gionis, P. Indyk, and R. Motwani, “Similarity Search in High Dimensions via Hashing,” VLDB ’99 Proc. 25th Int. Conf. Very Large Data Bases, vol. 99, no. 1, pp. 518–529, 1999.](http://www.cs.princeton.edu/courses/archive/spring13/cos598C/Gionis.pdf%5Cnhttp://portal.acm.org/citation.cfm?id=671516)
1. [Kanerva, Pentti, Kristoferson, Jan and Holst, Anders (2000). Random Indexing of Text Samples for Latent Semantic Analysis](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.6523&rep=rep1&type=pdf)
1. [Rangan, Venkat. Discovery of Related Terms in a corpus using Reflective Random Indexing](https://www.umiacs.umd.edu/~oard/desi4/papers/rangan.pdf)
1. [Vasuki, Vidya and Cohen, Trevor. Reflective random indexing for semi-automatic indexing of the biomedical literature](https://ac.els-cdn.com/S1532046410000481/1-s2.0-S1532046410000481-main.pdf?_tid=f31f92e8-028a-11e8-8c31-00000aab0f6c&acdnat=1516965824_e24a804445fff1744281ca6f5898a3a4)
1. [QasemiZadeh, Behrang and Handschuh, Siegfried. Random Indexing Explained with High Probability](http://pars.ie/publications/papers/pre-prints/random-indexing-dr-explained.pdf)
1. [Foulds, James; Boyles, Levi; Dubois, Christopher; Smyth, Padhraic; Welling, Max (2013). Stochastic Collapsed Variational Bayesian Inference for Latent Dirichlet Allocation][SCVB0]



[LSA]: https://en.wikipedia.org/wiki/Latent_semantic_analysis
[SCVB0]: https://arxiv.org/pdf/1305.2452

A spelling corrector for the Spanish language or create your own

# Spelling Corrector

[![Tests](https://github.com/jorelosorio/spellingcorrector/actions/workflows/tests.yml/badge.svg)](https://github.com/jorelosorio/spellingcorrector/actions/workflows/tests.yml)
[![Coverage Status](https://coveralls.io/repos/github/jorelosorio/spellingcorrector/badge.svg?branch=main)](https://coveralls.io/github/jorelosorio/spellingcorrector?branch=main)
[![Go Reference](https://pkg.go.dev/badge/github.com/jorelosorio/spellingcorrector.svg)](https://pkg.go.dev/github.com/jorelosorio/spellingcorrector)
[![Go Report Card](https://goreportcard.com/badge/github.com/jorelosorio/spellingcorrector)](https://goreportcard.com/report/github.com/jorelosorio/spellingcorrector)

A spelling corrector for the Spanish language or create your own.

The solution for this project was based on the proposal made on the following website: http://norvig.com/spell-correct.html and some ideas from https://cxwangyi.wordpress.com/2012/02/15/peter-norvigs-spelling-corrector-in-go/ as well.

> The built-in data was trained using the `Spanish` language.

## Try it

Use it now with a Docker instance. It will open the `8080` port to access the service.

    docker pull jorelosorio/spellingcorrector:latest

    docker run --name spellingcorrector -d -p 8080:80 -t jorelosorio/spellingcorrector:latest

Try it using the following example:

    http://localhost:8080/spelling?word=espanol

## Tools

- GoLang `1.17.x`
- Docker
- Visual Studio Code `Optional!`
    > It requires a `Remote - Containers` extension. for more information please refers to: https://code.visualstudio.com/docs/remote/containers#_getting-started

## Development

This project contains a `Dockerfile` file with all required dependencies to run it using `Visual Studio Code` + `Remote - Containers` extension.
However, if you want to make it run locally in your development machine, please follow the instructions below.

### Install Go

Install it from https://go.dev/dl/

### Build the `Example/Service`

> Make sure the port `80` is currently free. **Optionally could be changed in the code!**

    go build -o ./bin/ ./examples/service.go

Then run the service

    ./bin/service ./dictionaries/es.dic

### Example of correction

Simple usage example of the `Corrector` function.

```go
package main

import (
	"fmt"

	sc "github.com/jorelosorio/spellingcorrector"
)

func main() {
	spelling, _ := sc.NewSpelling("{YOUR_PATH_TO_DICTIONARY}")
	correctedWord := spelling.Correction("espanol")
	fmt.Println(correctedWord)
}
```

> `NewSpelling` functions returns (`Spelling`, `error`), make sure to handle errors when creating a new object.

### Training

Most of the training was made using free versions of books in `Spanish`. However, if you like to train for a new language you can use the following functions

```go
package main

import (
    sc "github.com/jorelosorio/spellingcorrector"
)

func main() {
    dic, _ := sc.NewDictionary("{YOUR_PATH_TO_DICTIONARY}", sc.ESAlphabet) // Or ENAlphabet
    dic.TrainFromTextFile("{YOUR_INPUT_TEXT}")
}
```

> Call `TrainFromTextFile` function as many times you wish with different inputs.

> `NewDictionary` functions returns (`Dictionary`, `error`), make sure to handle errors when creating a new dictionary.

### Build Docker

To build the docker image use `.dockers/Dockerfile.deploy` and the command

    docker build -f Dockerfile.deploy -t jorelosorio/spellingcorrector:latest .

To run the docker image

    docker run --name spellingcorrector -d -p 8080:80 -t jorelosorio/spellingcorrector:latest

Test the `spelling corrector` from the docker image

    http://localhost:8080/spelling?word=espanol


spelling-corrector

Reader and utility functions for word2vec embeddings

## Introduction

[![GoDoc](https://godoc.org/gopkg.in/danieldk/go2vec.v1?status.svg)](https://godoc.org/gopkg.in/danieldk/go2vec.v1)
[![Report card](http://goreportcard.com/badge/danieldk/go2vec)](http://goreportcard.com/report/danieldk/go2vec)
[![Build Status](https://travis-ci.org/danieldk/go2vec.svg?branch=master)](https://travis-ci.org/danieldk/go2vec)

This is a package for reading word2vec vectors in Go and finding similar
words and analogies.

## Installation

This package can be installed with the <tt>go</tt> command:

 go get gopkg.in/danieldk/go2vec.v1

To install the command-line utilities, use:

 go get gopkg.in/danieldk/go2vec.v1/cmd/...

The package documentation is available at:
https://godoc.org/gopkg.in/danieldk/go2vec.v1

go2vec

Convert numbers to Russian words with correct grammatical gender and noun declension

# go-propisyu · Числа прописью на Go

**Русская версия · [English version](README_EN.md)**

[![CI](https://github.com/rekurt/go-propisyu/actions/workflows/ci.yml/badge.svg)](https://github.com/rekurt/go-propisyu/actions/workflows/ci.yml)
[![Go Reference](https://pkg.go.dev/badge/github.com/rekurt/go-propisyu.svg)](https://pkg.go.dev/github.com/rekurt/go-propisyu)
[![Go Report Card](https://goreportcard.com/badge/github.com/rekurt/go-propisyu)](https://goreportcard.com/report/github.com/rekurt/go-propisyu)
[![codecov](https://codecov.io/gh/rekurt/go-propisyu/branch/master/graph/badge.svg)](https://codecov.io/gh/rekurt/go-propisyu)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Go Version](https://img.shields.io/github/go-mod/go-version/rekurt/go-propisyu)](go.mod)

`go-propisyu` — Go-библиотека для конвертации чисел в слова на русском языке с правильными склонениями и грамматическими родами. Числа прописью нужны в счетах, фискальных чеках, бухгалтерских документах, банковских выписках, голосовых ассистентах, чат-ботах и любых сервисах, где необходимо грамотно записывать суммы и количества словами.

```go
propisyu.IntToWords(42)                              // "сорок два"
propisyu.Decline(5, "рубль", "рубля", "рублей")      // "рублей"
propisyu.DecimalToWords("1234.56")                    // "одна тысяча двести тридцать четыре целых пятьдесят шесть сотых"
```

## Кому подходит

| Сфера | Пример |
|-------|--------|
| Финтех и банкинг | Сумма прописью в платёжных поручениях и выписках |
| Бухгалтерия и 1С | Генерация счетов-фактур, актов, накладных |
| Фискальные чеки | Касса / ОФД — сумма словами по 54-ФЗ |
| Голосовые ассистенты | TTS-озвучка сумм и количеств |
| Чат-боты | Ответы с суммами на естественном языке |
| Генерация документов | Шаблоны договоров, доверенностей, актов |

## Ключевые особенности

- Поддержка огромных чисел вплоть до дуодециллионов (10³⁹)
- Грамматические роды: мужской, женский и средний для корректных окончаний
- Функция `Decline` для автоматического склонения существительных
- Работа с десятичными числами через строки или `decimal.Decimal`
- Нулевые внешние зависимости для базовых функций
- Высокое покрытие тестами, CI/CD, линтер

## Содержание

- [Установка](#установка)
- [Быстрый старт](#быстрый-старт)
- [Примеры использования](#примеры-использования)
- [API](#api)
- [Почему go-propisyu](#почему-go-propisyu)
- [Ограничения](#ограничения)
- [Contributing](#contributing)
- [Лицензия](#лицензия)

## Установка

```bash
go get github.com/rekurt/go-propisyu
```

Для работы с `decimal.Decimal`:
```bash
go get github.com/shopspring/decimal
```

## Публичные функции

| Функция | Описание |
|---------|----------|
| `IntToWords(n int) string` | Конвертирует целое число в слова (мужской род) |
| `IntToWordsGender(n int, gender Gender) string` | Конвертирует целое число в слова с указанием рода |
| `DecimalToWords(decimalStr string) (string, error)` | Конвертирует десятичное число из строки в слова |
| `DecimalValueToWords(d decimal.Decimal) (string, error)` | Конвертирует `decimal.Decimal` значение в слова |
| `Decline(n int, one, two, five string) string` | Выбирает правильную форму склонения существительного |

### Константы рода

```go
const (
    GenderMasculine Gender = 1  // Мужской род: "один", "два"
    GenderFeminine  Gender = 2  // Женский род: "одна", "две"
    GenderNeuter    Gender = 3  // Средний род: "одно", "два"
)
```

### Ошибки

- `ErrNumberTooLarge` - число слишком велико для конвертации (не помещается в `int`)

## Быстрый старт

### Целые числа

```go
package main

import (
	"fmt"

	"github.com/rekurt/go-propisyu"
)

func main() {
	// Базовая конвертация (мужской род по умолчанию)
	fmt.Println(propisyu.IntToWords(321))
	// триста двадцать один

	// Конвертация с указанием рода
	fmt.Println(propisyu.IntToWordsGender(2, propisyu.GenderFeminine))
	// две

	fmt.Println(propisyu.IntToWordsGender(2, propisyu.GenderMasculine))
	// два

	// Автоматическое склонение существительных
	fmt.Println(propisyu.Decline(1, "рубль", "рубля", "рублей"))   // рубль
	fmt.Println(propisyu.Decline(2, "рубль", "рубля", "рублей"))   // рубля
	fmt.Println(propisyu.Decline(5, "рубль", "рубля", "рублей"))   // рублей
	fmt.Println(propisyu.Decline(21, "рубль", "рубля", "рублей"))  // рубль
}
```

### Десятичные числа

#### Способ 1: Используя строку

```go
package main

import (
	"fmt"
	"log"

	"github.com/rekurt/go-propisyu"
)

func main() {
	result, err := propisyu.DecimalToWords("123.45")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(result)
	// сто двадцать три целых сорок пять сотых
}
```

#### Способ 2: Используя decimal.Decimal

```go
package main

import (
	"fmt"
	"log"

	"github.com/rekurt/go-propisyu"
	"github.com/shopspring/decimal"
)

func main() {
	d := decimal.NewFromFloat(123.45)
	result, err := propisyu.DecimalValueToWords(d)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(result)
	// сто двадцать три целых сорок пять сотых
}
```

## Примеры использования

### Сумма прописью для чека

```go
package main

import (
	"fmt"

	"github.com/rekurt/go-propisyu"
)

func main() {
	amount := 1234
	rubles := propisyu.IntToWords(amount)
	rublesDecl := propisyu.Decline(amount, "рубль", "рубля", "рублей")

	fmt.Printf("%s %s 00 копеек", rubles, rublesDecl)
	// одна тысяча двести тридцать четыре рубля 00 копеек
}
```

### Большие числа

```go
package main

import (
	"fmt"
	"log"

	"github.com/rekurt/go-propisyu"
)

func main() {
	result, err := propisyu.DecimalToWords("6453345242432.42")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(result)
	// шесть триллионов четыреста пятьдесят три миллиарда
	// триста сорок пять миллионов двести сорок две тысячи четыреста тридцать два
	// целых сорок две сотых
}
```

### Склонение с разными существительными

```go
package main

import (
	"fmt"

	"github.com/rekurt/go-propisyu"
)

func main() {
	count := 5

	// Валюты
	fmt.Println(count, propisyu.Decline(count, "доллар", "доллара", "долларов"))
	// 5 долларов

	// Единицы измерения
	fmt.Println(count, propisyu.Decline(count, "метр", "метра", "метров"))
	// 5 метров

	// Предметы
	fmt.Println(count, propisyu.Decline(count, "товар", "товара", "товаров"))
	// 5 товаров

	// Для 21
	count = 21
	fmt.Println(count, propisyu.Decline(count, "день", "дня", "дней"))
	// 21 день
}
```

## API

### Функции для целых чисел

#### `IntToWords(n int) string`
Конвертирует целое число в слова (мужской род по умолчанию).

```go
propisyu.IntToWords(42)    // "сорок два"
propisyu.IntToWords(1000)  // "одна тысяча"
```

#### `IntToWordsGender(n int, gender Gender) string`
Конвертирует целое число в слова с указанием рода.

Доступные роды:
- `GenderMasculine` (1) - мужской род
- `GenderFeminine` (2) - женский род
- `GenderNeuter` (3) - средний род

```go
propisyu.IntToWordsGender(2, propisyu.GenderMasculine)  // "два"
propisyu.IntToWordsGender(2, propisyu.GenderFeminine)   // "две"
propisyu.IntToWordsGender(1, propisyu.GenderNeuter)     // "одно"
```

### Функции для десятичных чисел

#### `DecimalToWords(decimalStr string) (string, error)`
Конвертирует десятичное число из строки в слова. Дробная часть обрезается до 2 знаков.

```go
result, err := propisyu.DecimalToWords("3.14")
// "три целых четырнадцать сотых"
```

#### `DecimalValueToWords(d decimal.Decimal) (string, error)`
Конвертирует `decimal.Decimal` значение в слова. Дробная часть обрезается (не округляется!) до 2 знаков.

```go
d := decimal.NewFromFloat(3.14159)
result, err := propisyu.DecimalValueToWords(d)
// "три целых четырнадцать сотых" (не округляет 3.14159 до 3.14, а обрезает)
```

**Важно:** Функция обрезает (truncate), а не округляет:
- `1.999` → "один целых девяносто девять сотых"
- `1.995` → "один целых девяносто девять сотых"

### Функция склонения

#### `Decline(n int, one, two, five string) string`
Выбирает правильную форму существительного в зависимости от числа.

Параметры:
- `n` - число
- `one` - форма для 1, 21, 31... (рубль, день, товар)
- `two` - форма для 2-4, 22-24... (рубля, дня, товара)
- `five` - форма для 0, 5-20, 25-30... (рублей, дней, товаров)

```go
propisyu.Decline(1, "рубль", "рубля", "рублей")   // "рубль"
propisyu.Decline(2, "рубль", "рубля", "рублей")   // "рубля"
propisyu.Decline(5, "рубль", "рубля", "рублей")   // "рублей"
propisyu.Decline(11, "рубль", "рубля", "рублей")  // "рублей"
propisyu.Decline(21, "рубль", "рубля", "рублей")  // "рубль"
```

## Почему go-propisyu

- **Чистый Go** — не обёртка над C-библиотекой, легко компилировать и деплоить
- **Корректная грамматика** — три рода, правильные склонения для всех числовых диапазонов
- **Без зависимостей** — `IntToWords` и `Decline` работают без сторонних пакетов
- **production-ready** — CI с линтером, тесты, семантическое версионирование, goreleaser
- **Открытая лицензия** — MIT, можно использовать в коммерческих проектах

## Ограничения

- Целые числа: поддерживаются значения в диапазоне `int` (обычно -2³¹ до 2³¹-1 или -2⁶³ до 2⁶³-1)
- Десятичные числа: поддерживаются только 2 знака после запятой (остальное обрезается)
- `DecimalValueToWords` вернет ошибку `ErrNumberTooLarge`, если число не помещается в `int`

## Тесты

```bash
go test ./...              # Запустить все тесты
go test -v ./...           # С подробным выводом
go test -cover ./...       # С покрытием кода
```

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md).

## Лицензия

MIT


go-propisyu

JP morphological analyzer written in pure Go

[![GoDev](https://pkg.go.dev/badge/github.com/ikawaha/kagome/v2)](https://pkg.go.dev/github.com/ikawaha/kagome/v2)
[![Go](https://github.com/ikawaha/kagome/workflows/Go/badge.svg)](https://github.com/ikawaha/kagome/actions?query=workflow%3AGo)
[![Release](https://github.com/ikawaha/kagome/actions/workflows/release.yml/badge.svg?branch=)](https://github.com/ikawaha/kagome/actions/workflows/release.yml)
[![Coverage Status](https://coveralls.io/repos/github/ikawaha/kagome/badge.svg?branch=v2)](https://coveralls.io/github/ikawaha/kagome?branch=v2)
[![Docker Pulls](https://img.shields.io/docker/pulls/ikawaha/kagome.svg?style)](https://hub.docker.com/r/ikawaha/kagome/)

# Kagome v2

Kagome is an open source Japanese morphological analyzer written in pure Go. It can tokenize Japanese text into words and analyze parts of speech, with dictionaries embedded in the binary for easy deployment.

> [!NOTE]
> **Key features** (Improvements from [v1](https://github.com/ikawaha/kagome/tree/master)):
>
> * Self-contained binaries with embedded dictionaries (MeCab-IPADIC, UniDic)
> * Multiple segmentation modes for different use cases
> * RESTful API server mode for production use
> * WebAssembly support for browser environments
> * C library API for FFI integration (Python, PHP, and other languages)

## Index

* [Basic Usage](#basic-usage)
 * [Command line](#command-line)
 * [As a Go library](#as-a-go-library)
 * [As a C library](#as-a-c-library)
 * [More examples](#more-examples)
* [Install](#install)
* [Commands](#commands)
 * [Tokenize command](#tokenize-command)
 * [Server command](#server-command)
 * [RESTful API](#restful-api)
 * [Web App](#web-app)
 * [Lattice command](#lattice-command)
 * [Sentence command](#sentence-command)
* [Dictionaries](#dictionaries)
* [Segmentation modes](#segmentation-modes)
* [Docker](#docker)
* [WebAssembly](#webassembly)
* [Use from other languages (FFI)](#use-from-other-languages-ffi)
* [Reference](#reference)
* [License](#license)

## Basic Usage

### Command line

```shellsession
% kagome -h
Japanese Morphological Analyzer -- github.com/ikawaha/kagome/v2
usage: kagome <command>
The commands are:
 [tokenize] - command line tokenize (*default)
 server - run tokenize server
 lattice - lattice viewer
 sentence - tiny sentence splitter
 version - show version

tokenize [-file input_file] [-dict dic_file] [-userdict user_dic_file] [-sysdict (ipa|uni)] [-simple false] [-mode (normal|search|extended)] [-split] [-json]
 -dict string
 	dict
 -file string
 	input file
 -json
 	outputs in JSON format
 -mode string
 	tokenize mode (normal|search|extended) (default "normal")
 -simple
 	display abbreviated dictionary contents
 -split
 	use tiny sentence splitter
 -sysdict string
 	system dict type (ipa|uni) (default "ipa")
 -udict string
 	user dict
```

```shellsession
% # piped standard input
% echo "すもももももももものうち" | kagome
すもも	名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
の	助詞,連体化,*,*,*,*,の,ノ,ノ
うち	名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
```

* For more details, see the [Commands section](#commands).

### As a Go library

You can integrate Kagome into your Go applications as follows:

```sh
# Install Kagome module
go get github.com/ikawaha/kagome/v2
```

```Go
package main

import (
 "fmt"
 "strings"

 "github.com/ikawaha/kagome-dict/ipa"
 "github.com/ikawaha/kagome/v2/tokenizer"
)

func main() {
 t, err := tokenizer.New(ipa.Dict(), tokenizer.OmitBosEos())
 if err != nil {
 panic(err)
 }
 // wakati (simple word splitting/segmentation)
 fmt.Println("---wakati---")
 seg := t.Wakati("すもももももももものうち")
 fmt.Println(seg)

 // tokenize w/ morphological analysis
 fmt.Println("---tokenize---")
 tokens := t.Tokenize("すもももももももものうち")
 for _, token := range tokens {
 features := strings.Join(token.Features(), ",")
 fmt.Printf("%s\t%v\n", token.Surface, features)
 }
}
```

output:

```shellsession
---wakati---
[すもも も もも も もも の うち]
---tokenize---
すもも	名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
の	助詞,連体化,*,*,*,*,の,ノ,ノ
うち	名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
```

### As a C library

Kagome is written in pure Go but can be compiled as a C shared library and used from other languages via [FFI](https://en.wikipedia.org/wiki/Foreign_function_interface) (Foreign Function Interface).

See the "[Use from other languages (FFI)](#use-from-other-languages-ffi)" section below for details and examples.

### More examples

We provide various examples demonstrating how to use Kagome in different scenarios:

* [Examples directory](https://github.com/ikawaha/kagome/tree/v2/_examples)
* [Examples in GoDoc](https://pkg.go.dev/github.com/ikawaha/kagome/v2)

## Install

To **get the `kagome` command line tool**, choose your preferred installation method below:

* **Go (recommended)**

 ```shellsession
 go install github.com/ikawaha/kagome/v2@latest
 ```

* **Homebrew**

 ```shellsession
 # macOS and Linux (for both AMD64 and Arm64)
 brew install ikawaha/kagome/kagome
 ```

* **Manual Install**

 * For manual installation, download and extract the appropriate archived file for your OS and architecture from the [releases page](https://github.com/ikawaha/kagome/releases/latest).
 * Note that the extracted binary must be placed in an accessible directory with execution permission.

* **Docker/Docker Compose**

 * See the [Docker section](#docker) below

## Commands

Major sub-commands of `kagome` command line tool.

### Tokenize command

```shellsession
% # interactive/REPL mode
% kagome
すもももももももものうち
すもも	名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
の	助詞,連体化,*,*,*,*,の,ノ,ノ
うち	名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
```

```shellsession
% # piped standard input
% echo "すもももももももものうち" | kagome
すもも 名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も 助詞,係助詞,*,*,*,*,も,モ,モ
もも 名詞,一般,*,*,*,*,もも,モモ,モモ
も 助詞,係助詞,*,*,*,*,も,モ,モ
もも 名詞,一般,*,*,*,*,もも,モモ,モモ
の 助詞,連体化,*,*,*,*,の,ノ,ノ
うち 名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
```

```shellsession
% # JSON output
% # (For jq command see https://jqlang.org/)
% echo "猫" | kagome -json | jq .
[
 {
 "id": 286994,
 "start": 0,
 "end": 1,
 "surface": "猫",
 "class": "KNOWN",
 "pos": [
 "名詞",
 "一般",
 "*",
 "*"
 ],
 "base_form": "猫",
 "reading": "ネコ",
 "pronunciation": "ネコ",
 "features": [
 "名詞",
 "一般",
 "*",
 "*",
 "*",
 "*",
 "猫",
 "ネコ",
 "ネコ"
 ]
 }
]
```

```shellsession
% # word splitting/segmentation only (equivalent to "wakati" functionality)
% echo "すもももももももものうち" | kagome -json | jq -r '[.[].surface] | join("/")'
すもも/も/もも/も/もも/の/うち
```

```shellsession
% # Extract only pronunciations using jq (for Text-to-Speech purposes, etc.)
% echo "私ははにわよわわわんわん" | kagome -json | jq -r '.[].pronunciation'
ワタシ
ワ
ハニワ
ヨ
ワ
ワ
ワンワン
```

### Server command

For continuous usage, `kagome` provides a server mode to decouple the startup time of the tokenizer.

#### RESTful API

Start a server and try to access the "/tokenize" endpoint.

```shellsession
% kagome server &
% curl -XPUT localhost:6060/tokenize -d'{"sentence":"すもももももももものうち", "mode":"normal"}' | jq .
```

#### Web App

Start a server and access `http://localhost:6060` in your browser.

```shellsession
% kagome server &
```

<video src="https://github.com/user-attachments/assets/172d8b0c-be9a-4ee5-bdff-315a0eeee85b" controls width="100%"></video>

> [!IMPORTANT]
> The demo web application uses [graphviz](https://graphviz.org/) to draw a lattice. You need graphviz to be installed on your system.

> [!TIP]
> Kagome can be compiled to WebAssembly (wasm) and run locally in a web browser as well. For details, see the [WebAssembly section](#webassembly).
>
> * Wasm Demo: [https://ikawaha.github.io/kagome/](https://ikawaha.github.io/kagome/)

### Lattice command

A debug tool of tokenize process outputs a lattice in graphviz dot format.

```shellsession
% kagome lattice 私は鰻 | dot -Tpng -o lattice.png
```

![lattice](https://user-images.githubusercontent.com/4232165/89723585-74717000-da33-11ea-886a-baab85f7a06e.png)

### Sentence command

Split long text into sentences:

```shellsession
% echo "吾輩は猫である。名前はまだ無い。" | kagome sentence
吾輩は猫である。
名前はまだ無い。
```

This command is useful if a single line of data is too lengthy, and you want to avoid errors such as `bufio.Scanner: token too long`.

```shellsession
% echo "吾輩は猫である。名前はまだ無い。" | kagome -json | jq -r '[.[].surface] | join("/")'
吾輩/は/猫/で/ある/。/名前/は/まだ/無い/。

% echo "吾輩は猫である。名前はまだ無い。" | kagome sentence | kagome -json | jq -r '[.[].surface] | join("/")'
吾輩/は/猫/で/ある/。
名前/は/まだ/無い/。
```

This command is equivalent to the `-split` option of the `tokenize` command.

```shellsession
% echo "吾輩は猫である。名前はまだ無い。" | kagome -split -json | jq -r '[.[].surface] | join("/")'
吾輩/は/猫/で/ある/。
名前/は/まだ/無い/。
```

## Dictionaries

* Currently supported dictionaries by default.

 |dict| source | package |
 |:---|:---|:---|
 |MeCab IPADIC| mecab-ipadic-2.7.0-20070801 | [github.com/ikawaha/kagome-dict/ipa](https://github.com/ikawaha/kagome-dict/tree/master/ipa)|
 |UniDIC| unidic-mecab-2.1.2_src | [github.com/ikawaha/kagome-dict/uni](https://github.com/ikawaha/kagome-dict/tree/master/uni) |

* Experimental Features

 |dict|source|package|
 |:---|:---|:---|
 |mecab-ipadic-NEologd|mecab-ipadic-neologd| [github.com/ikawaha/kagome-ipa-neologd](https://github.com/ikawaha/kagome-dict-ipa-neologd)|
 |Korean MeCab|mecab-ko-dic-2.1.1-20180720 | [github.com/ikawaha/kagome-dict-ko](https://github.com/ikawaha/kagome-dict-ko)|

> [!NOTE]
> For more details and differences between the dictionaries, see the [wiki](https://github.com/ikawaha/kagome/wiki/About-the-dictionary).

## Segmentation modes

Similar to [Kuromoji](https://www.atilika.org/), Kagome also supports various **segmentation modes** (splitting strategies) to tokenize the input text.

* **Normal:** Regular segmentation
* **Search:** Use a heuristic to perform additional segmentation that is **useful for search** purposes
* **Extended:** Similar to search mode, but also unknown words with [uni-grams](https://en.wikipedia.org/wiki/N-gram)

|Untokenized|Normal|Search|Extended|
|:-------|:---------|:---------|:---------|
|関西国際空港|関西国際空港|関西　国際　空港|関西　国際　空港|
|日本経済新聞|日本経済新聞|日本　経済　新聞|日本　経済　新聞|
|シニアソフトウェアエンジニア|シニアソフトウェアエンジニア|シニア　ソフトウェア　エンジニア|シニア　ソフトウェア　エンジニア|
|デジカメを買った|デジカメ　を　買っ　た|デジカメ　を　買っ　た|デ　ジ　カ　メ　を　買っ　た|

> [!NOTE]
>If your purpose is for search, try changing the mode before switching to another dictionary.

## Docker

[![Docker](https://dockerico.blankenship.io/image/ikawaha/kagome)](https://hub.docker.com/r/ikawaha/kagome)

We provide `scratch`-based Docker images that simply run the `kagome` command line tool on various architectures: AMD64, Arm64, Arm32 (Arm v5, v6 and v7)

* Pull the image

 ```sh
 docker pull ikawaha/kagome:latest
 ```

 ```sh
 # Alternatively, you can pull from GitHub Container Registry
 docker pull ghcr.io/ikawaha/kagome:latest
 ```

* Run the command via Docker

 ```sh
 # Interactive/REPL mode
 docker run --rm -it ikawaha/kagome:latest
 ```

 ```sh
 # If pulling from GitHub Container Registry
 docker run --rm -it ghcr.io/ikawaha/kagome:latest
 ```

* Run the server via Docker

 ```sh
 # Server mode (http://localhost:6060)
 docker run --rm -p 6060:6060 ikawaha/kagome:latest server
 ```

 ```sh
 # If pulling from GitHub Container Registry
 docker run --rm -p 6060:6060 ghcr.io/ikawaha/kagome:latest server
 ```

* `docker-compose.yml` example

 ```yaml
 services:
 kagome:
 image: ikawaha/kagome:latest
 ports: ["6060:6060"]
 command: server
 restart: unless-stopped
 ```

> **Note:** Base image doesn't include Graphviz. For lattice visualization, see [examples](./_examples/server_docker_graphviz/).

## WebAssembly

Kagome compiles to WebAssembly for browser use.

* **Live demo:** [https://ikawaha.github.io/kagome/](https://ikawaha.github.io/kagome/)
* **Source code:** [./_examples/wasm](./_examples/wasm)

### Use from other languages (FFI)

Kagome is written in pure Go but can be compiled as a C shared library and used from other languages via FFI (Foreign Function Interface).

* Currently supported/tested languages:
 * **Python 3.12+** (using `ctypes`)
 * **PHP 8+** (using `FFI`)

```python
# Python example using ctypes
from libkagome import Kagome

kagome = Kagome()
tokens = kagome.tokenize("すもももももももものうち")

for token in tokens:
 print(f"{token.surface}\t{token.pos}")
```

```php

<?php
declare(strict_types=1);

require __DIR__ . '/libkagome.php';

$kagome = new Kagome();
$tokens = $kagome->tokenize("すもももももももものうち");

foreach ($tokens as $token) {
 echo "{$token->surface}\t" . implode(',', $token->pos) . "\n";
}
```

For complete examples and build instructions, see:

* [./_examples/clib/](./_examples/clib/) - C library FFI examples for Python and PHP

> [!NOTE]
> The C library provides thread-safe tokenization with proper memory management and includes comprehensive tests.

## Reference

* Detailed Reference Manual in Japanese:

 [![実践：形態素解析 kagome v2](https://user-images.githubusercontent.com/4232165/102152682-e281e400-3eb8-11eb-91f7-13e08a8977d9.png)](https://zenn.dev/ikawaha/books/kagome-v2-japanese-tokenizer)

* Community Wiki in English:
 * [https://github.com/ikawaha/kagome/wiki](https://github.com/ikawaha/kagome/wiki)

## License

* MIT

kagome

This is a fairly straightforward port of Martin Porter's C implementation of the Porter stemming algorithm

Porter Stemmer for Go
=====================

[![CI](https://github.com/a2800276/porter/actions/workflows/ci.yml/badge.svg)](https://github.com/a2800276/porter/actions/workflows/ci.yml)
[![Go Reference](https://pkg.go.dev/badge/github.com/a2800276/porter.svg)](https://pkg.go.dev/github.com/a2800276/porter)

This is a straightforward port of Martin Porter's C implementation of
the Porter stemming algorithm. The C version this port is based on is
available for download here:
[http://tartarus.org/~martin/PorterStemmer/c_thread_safe.txt](http://tartarus.org/~martin/PorterStemmer/c_thread_safe.txt)

The original algorithm is described in the paper:

    M.F. Porter, 1980, An algorithm for suffix stripping, Program, 14(3) pp
    130-137.

## Features

- Thread-safe implementation
- Multiple APIs: simple string API and zero-allocation byte-slice API
- Command-line tool for batch processing
- Comprehensive test suite
- Benchmarked and optimized
- No external dependencies

## Installation

### Library

```bash
go get github.com/a2800276/porter
```

### CLI Tool

```bash
go install github.com/a2800276/porter/cmd/porter@latest
```

## Usage

### As a Library

```go
package main

import (
    "fmt"
    "log"
    "github.com/a2800276/porter"
)

func main() {
    // Simple string API (with allocations)
    stemmed, err := porter.Stem("running")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(stemmed) // Output: run

    // Efficient byte-slice API (zero allocations)
    word := []byte("running")
    stemmed_bytes, err := porter.StemBytes(word)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(stemmed_bytes)) // Output: run
}
```

### As a CLI Tool

Install the command-line tool:

```bash
go install github.com/a2800276/porter/cmd/porter@latest
```

Use it to stem words:

```bash
# Stem words from arguments
$ porter running jumped easily
run
jump
easili

# Stem words from stdin
$ echo -e "running\njumped\neasily" | porter
run
jump
easili

# Process a file
$ cat words.txt | porter > stemmed.txt

# Count unique stems
$ cat corpus.txt | porter | sort | uniq -c | sort -rn
```

## API

The package provides two functions for different use cases:

### `Stem(word string) (string, error)`

The simplest API that takes a string and returns a stemmed string. Handles case conversion automatically.
Returns an error if stemming fails (though this is rare in normal use).

### `StemBytes(b []byte) ([]byte, error)`

Zero-allocation API that stems the byte slice in-place and returns the stemmed portion as a slice.
The input is converted to lowercase. Best for high-performance scenarios.
Returns an error if stemming fails.

## Performance

The implementation is highly optimized:

### String API (convenient, with allocations)
```
BenchmarkStem-24          14064384    77.29 ns/op    16 B/op    2 allocs/op
```

### Byte-Slice API (fastest, zero allocations)
```
BenchmarkStemBytes-24     23443530    51.85 ns/op     0 B/op    0 allocs/op
```

The byte-slice API (`StemBytes`) is ~35% faster and performs zero allocations,
making it ideal for high-performance applications.

**Note:** Error handling adds minimal overhead (~2ns) but provides explicit feedback on failures.



## Limitations

- The algorithm operates on English words only. Input is automatically converted to lowercase.
- For the `Stem()` function, strings are converted to byte slices internally.
  For zero-copy operation, use `StemBytes()`.
- Unicode handling: The algorithm is designed for ASCII English text. Non-ASCII characters should be handled by the caller before stemming.

## Development

### Building

```bash
make build       # Build the CLI tool
make install     # Install CLI to $GOPATH/bin
```

### Running Tests

```bash
make test        # Run tests
make coverage    # Generate coverage report
make bench       # Run benchmarks
```

### Linting and Formatting

```bash
make fmt         # Format code
make vet         # Run go vet
make lint        # Run golangci-lint (requires installation)
```

## Contributing

Contributions are welcome! Please ensure:

- Tests pass: `make test`
- Code is formatted: `make fmt`
- No linting errors: `make lint`

## License

MIT licensed. See LICENSE file for details. 


porter

porter2
=======

[![GoDoc](http://godoc.org/github.com/surgebase/porter2?status.svg)](http://godoc.org/github.com/surgebase/porter2)

Porter2 implements the [english Porter2 stemmer](http://snowball.tartarus.org/algorithms/english/stemmer.html). It is written completely using finite state machines to do suffix comparison, rather than the string-based or tree-based approaches. As a result, it is 660% faster compare to string comparison-based approach.

```
import "github.com/surgebase/porter2"

fmt.Println(porter2.Stem("seaweed")) // should get seawe
```

This implementation has been successfully validated with the dataset from http://snowball.tartarus.org/algorithms/english/

### Performance

This implementation by far has the highest performance of the various Go-based implementations, AFAICT. I tested a few of the implementations and the results are below. 

| Implementation | Time | Algorithm |
|----------------|------|-----------|
| [surgebase](https://github.com/surgebase/porter2) | 319.009358ms | Porter2 |
| [dchest](https://github.com/dchest/stemmer) | 2.106912401s | Porter2 |
| [kljensen](https://github.com/kljensen/snowball) | 5.725917198s | Porter2 |

To run the test again, you can run cmd/compare/compare.go (`go run compare.go`).

### State Machines

Most of the implementations, like the ones in the table above, rely completely on suffix string comparison. Basically there's a list of suffixes, and the code will loop through the list to see if there's a match. Given most of the time you are looking for the longest match, so you order the list so the longest is the first one. So if you are luckly, the match will be early on the list. But regardless that's a huge performance hit.

This implementation is based completely on finite state machines to perform suffix comparison. You compare each chacter of the string starting at the last character going backwards. The state machines at each step will determine what the longest suffix is. You can think of the state machine as an unrolled tree. 

However, writing large state machines can be very error-prone. So I wrote a [quick tool](https://github.com/surgebase/porter2/tree/master/cmd/suffixfsm) to generate most of the state machines. The tool basically takes a file of suffixes, creates a tree, then unrolls the tree by dumping each of the nodes. 

You can run the tool by `go run suffixfsm.go <filename>`.

### License

Copyright (c) 2014 Dataence, LLC. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

The Best Go Libraries For Natural Language Processing - Morphological Analyzers (17)