This is a GO implementation of [MMSEG](http://technology.chtsai.org/mmseg/) which a Chinese word splitting algorithm

MMSEGO
=====
This is a GO implementation of [MMSEG](http://technology.chtsai.org/mmseg/) which a Chinese word splitting algorithm.

TO DO list
----------
* Documentation/comments
* Benchmark

Usage
---------
#Input Dictionary Format
```sh
Key\tFreq
```
Each key occupies one line. The file should be utf-8 encoded, please refer to [go-darts](https://github.com/awsong/go-darts)

#Code example
```go
package main

import (
    "fmt"
    "time"
    "os"
    "mmsego"
    "bufio"
    "log"
    )

func main() {
    var s = new(mmsego.Segmenter)
    s.Init("darts.lib")
    if err != nil {
	log.Fatal(err)
    }

    t := time.Now()
    offset := 0

    unifile, _ := os.Open("/tmp/a.txt")
    uniLineReader := bufio.NewReaderSize(unifile, 4000)
    line, bufErr := uniLineReader.ReadString('\n')
    for nil == bufErr {
	//takeWord := func(off int, length int){ fmt.Printf("%s ", string(line[off-offset:off-offset+length])) }
	takeWord := func(off, length int){ }
	s.Mmseg(line[:], offset, takeWord, nil, false)
	offset += len(line)
	line, bufErr = uniLineReader.ReadString('\n')
    }
    takeWord := func(off int, length int){ fmt.Printf("%s ", string(line[off-offset:off-offset+length])) }
    s.Mmseg(line, offset, takeWord, nil, true)

    fmt.Printf("Duration: %v\n", time.Since(t))
}
```
LICENSE
-----------
Apache License 2.0


MMSEGO

The shamoji is word filtering package written in Go

# shamoji

[![CI](https://github.com/osamingo/shamoji/actions/workflows/actions.yml/badge.svg)](https://github.com/osamingo/shamoji/actions/workflows/actions.yml)
[![codecov](https://codecov.io/gh/osamingo/shamoji/branch/master/graph/badge.svg)](https://codecov.io/gh/osamingo/shamoji)
[![Go Report Card](https://goreportcard.com/badge/github.com/osamingo/shamoji)](https://goreportcard.com/report/github.com/osamingo/shamoji)
[![codebeat badge](https://codebeat.co/badges/9d9fdf3d-0c6d-455f-8444-8399a07d49ae)](https://codebeat.co/projects/github-com-osamingo-shamoji-master)
[![GoDoc](https://godoc.org/github.com/osamingo/shamoji?status.svg)](https://godoc.org/github.com/osamingo/shamoji)
[![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/osamingo/shamoji/master/LICENSE)

## About

The shamoji (杓文字) is word filtering package.

## Install

```
$ go get github.com/osamingo/shamoji@latest
```

## Usage

```go
package main

import (
	"fmt"
	"sync"

	"github.com/osamingo/shamoji"
	"github.com/osamingo/shamoji/filter"
	"github.com/osamingo/shamoji/tokenizer"
	"golang.org/x/text/unicode/norm"
)

var (
	o sync.Once
	s *shamoji.Serve
)

func main() {
	yes, word := Contains("我が生涯に一片の悔い無し")
	fmt.Printf("Result: %v, Word: %s", yes, word)
}

func Contains(sentence string) (bool, string) {
	o.Do(func() {
		tok, err := tokenizer.NewKagomeTokenizer(norm.NFKC)
		if err != nil {
			panic(err)
		}
		s = &shamoji.Serve{
			Tokenizer: tok,
			Filer:     filter.NewCuckooFilter("涯に", "悔い"),
		}
	})
	return s.Do(sentence)
}
```

## License

Released under the [MIT License](https://github.com/osamingo/shamoji/blob/master/LICENSE).


shamoji

Stemmer packages for Go programming language. Includes English and German stemmers

Stemmer package for Go
======================

Stemmer package provides an interface for stemmers and includes English,
German and Dutch stemmers as sub-packages:

 - `porter2` sub-package implements English (Porter2) stemmer as described in
   <http://snowball.tartarus.org/algorithms/english/stemmer.html>

 - `german` sub-package implements German stemmer as described in
   <http://snowball.tartarus.org/algorithms/german/stemmer.html>

 - `dutch` sub-package implements Dutch stemmer as described in
   <http://snowball.tartarus.org/algorithms/dutch/stemmer.html>


Installation
-------------

English stemmer:

    go get github.com/dchest/stemmer/porter2

German stemmer:

    go get github.com/dchest/stemmer/german

Dutch stemmer:

    go get github.com/dchest/stemmer/dutch

This will also install the top-level `stemmer` package.

Example
-------

    import (
        "github.com/dchest/stemmer/porter2"
        "github.com/dchest/stemmer/german"
        "github.com/dchest/stemmer/dutch"
    )

    // English.
    eng := porter2.Stemmer
    eng.Stem("delicious")   // => delici
    eng.Stem("deliciously") // => delici

    // German.
    ger := german.Stemmer
    ger.Stem("abhängen")   // => abhang
    ger.Stem("abhängiger") // => abhang

    // Dutch.
    dt := dutch.Stemmer
    dt.Stem("lichamelijke") // => licham
    dt.Stem("opglimpende")  // => opglimp

Tests
-----

Included `test_output.txt` and `test_voc.txt` are from the referenced original
implementations, used only when running tests with `go test`.


License
-------

2-clause BSD-like (see LICENSE and AUTHORS files).


stemmer

Go package for n-gram based text categorization, with support for utf-8 and raw text

A Go package for n-gram based text categorization, with support for utf-8 and raw text.

To do:

 * write documentation
 * make it faster

Keywords: text categorization, language detector

## Install

    go get github.com/pebbe/textcat
    go get github.com/pebbe/textcat/textcat
    go get github.com/pebbe/textcat/textpat

## Docs

 * [package help](http://godoc.org/github.com/pebbe/textcat)


textcat

Go efficient text segmentation; support english, chinese, japanese and other

# gse

Go efficient multilingual NLP and text segmentation; support English, Chinese, Japanese and others.
And supports with [elasticsearch](https://github.com/vcaesar/go-gse-elastic) and [bleve](https://github.com/vcaesar/gse-bleve).




[![Build Status](https://github.com/go-ego/gse/workflows/Go/badge.svg)](https://github.com/go-ego/gse/commits/master)
[![CircleCI Status](https://circleci.com/gh/go-ego/gse.svg?style=shield)](https://circleci.com/gh/go-ego/gse)
[![codecov](https://codecov.io/gh/go-ego/gse/branch/master/graph/badge.svg)](https://codecov.io/gh/go-ego/gse)
[![Go Report Card](https://goreportcard.com/badge/github.com/go-ego/gse)](https://goreportcard.com/report/github.com/go-ego/gse)
[![GoDoc](https://godoc.org/github.com/go-ego/gse?status.svg)](https://godoc.org/github.com/go-ego/gse)
[![GitHub release](https://img.shields.io/github/release/go-ego/gse.svg)](https://github.com/go-ego/gse/releases/latest)
[![Join the chat at https://gitter.im/go-ego/ego](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/go-ego/ego?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)




[简体中文](https://github.com/go-ego/gse/blob/master/README_zh.md) | [日本語](https://github.com/go-ego/gse/blob/master/README_ja.md)

Gse is implements jieba by golang, and try add NLP support and more feature

## Feature:

- Support common, search engine, full mode, precise mode and HMM mode multiple word segmentation modes;
- Support user and embed dictionary, Part-of-speech/POS tagging, analyze segment info, stop and trim words
- Support multilingual: English, Chinese, Japanese and others
- Support Traditional Chinese
- Support HMM cut text use Viterbi algorithm
- Support NLP by TensorFlow (in work)
- Named Entity Recognition (in work)
- Supports with [elasticsearch](https://github.com/vcaesar/go-gse-elastic) and bleve
- run<a href="https://github.com/go-ego/gse/blob/master/tools/server/server.go"> JSON RPC service</a>.

## Algorithm:

- [Dictionary](https://github.com/go-ego/gse/blob/master/dictionary.go) with double array trie (Double-Array Trie) to achieve
- [Segmenter](https://github.com/go-ego/gse/blob/master/dag.go) algorithm is the shortest path (based on word frequency and dynamic programming), and DAG and HMM algorithm word segmentation.

## Text Segmentation speed:

- <a href="https://github.com/go-ego/gse/blob/master/tools/benchmark/benchmark.go"> single thread</a> 9.2MB/s
- <a href="https://github.com/go-ego/gse/blob/master/tools/benchmark/goroutines/goroutines.go">goroutines concurrent</a> 26.8MB/s.
- HMM text segmentation single thread 3.2MB/s. (2core 4threads Macbook Pro).

## Binding:

[gse-bind](https://github.com/vcaesar/gse-bind), binding JavaScript and other, support more language.

## Install / update

With Go module support (Go 1.11+), just import:

```go
import "github.com/go-ego/gse"
```

Otherwise, to install the gse package, run the command:

```
go get -u github.com/go-ego/gse
```

## Use

```go
package main

import (
	_ "embed"
	"fmt"

	"github.com/go-ego/gse"
)

//go:embed testdata/test_en2.txt
var testDict string

//go:embed testdata/test_en.txt
var testEn string

var (
	text  = "To be or not to be, that's the question!"
	test1 = "Hiworld, Helloworld!"
)

func main() {
	// keep the origin capital letter
	// gse.ToLower = false

	var seg1 gse.Segmenter
	seg1.DictSep = ","
	err := seg1.LoadDict("./testdata/test_en.txt")
	if err != nil {
		fmt.Println("Load dictionary error: ", err)
	}

	s1 := seg1.Cut(text)
	fmt.Println("seg1 Cut: ", s1)
	// seg1 Cut:  [to be   or   not to be ,   that's the question!]

	var seg2 gse.Segmenter
	seg2.AlphaNum = true
	seg2.LoadDict("./testdata/test_en_dict3.txt")

	s2 := seg2.Cut(test1)
	fmt.Println("seg2 Cut: ", s2)
	// seg2 Cut:  [hi world ,   hello world !]

	var seg3 gse.Segmenter
	seg3.AlphaNum = true
	seg3.DictSep = ","
	err = seg3.LoadDictEmbed(testDict + "\n" + testEn)
	if err != nil {
		fmt.Println("loadDictEmbed error: ", err)
	}
	s3 := seg3.Cut(text + test1)
	fmt.Println("seg3 Cut: ", s3)
	// seg3 Cut:  [to be   or   not to be ,   that's the question! hi world ,   hello world !]

	// example2()
}
```

Example2:

```go
package main

import (
	"fmt"
	"regexp"

	"github.com/go-ego/gse"
	"github.com/go-ego/gse/hmm/pos"
)

var (
	text = "Hello world, Helloworld. Winter is coming! こんにちは世界, 你好世界."

	new, _ = gse.New("zh,testdata/test_en_dict3.txt", "alpha")

	seg gse.Segmenter
	posSeg pos.Segmenter
)

func main() {
	// Loading the default dictionary
	seg.LoadDict()
	// Loading the default dictionary with embed
	// seg.LoadDictEmbed()
	//
	// Loading the Simplified Chinese dictionary
	// seg.LoadDict("zh_s")
	// seg.LoadDictEmbed("zh_s")
	//
	// Loading the Traditional Chinese dictionary
	// seg.LoadDict("zh_t")
	//
	// Loading the Japanese dictionary
	// seg.LoadDict("jp")
	//
	// Load the dictionary
	// seg.LoadDict("your gopath"+"/src/github.com/go-ego/gse/data/dict/dictionary.txt")

	cut()

	segCut()
}

func cut() {
	hmm := new.Cut(text, true)
	fmt.Println("cut use hmm: ", hmm)

	hmm = new.CutSearch(text, true)
	fmt.Println("cut search use hmm: ", hmm)
	fmt.Println("analyze: ", new.Analyze(hmm, text))

	hmm = new.CutAll(text)
	fmt.Println("cut all: ", hmm)

	reg := regexp.MustCompile(`(\d+年|\d+月|\d+日|[\p{Latin}]+|[\p{Hangul}]+|\d+\.\d+|[a-zA-Z0-9]+)`)
	text1 := `헬로월드 헬로 서울, 2021年09月10日, 3.14`
	hmm = seg.CutDAG(text1, reg)
	fmt.Println("Cut with hmm and regexp: ", hmm, hmm[0], hmm[6])
}

func analyzeAndTrim(cut []string) {
	a := seg.Analyze(cut, "")
	fmt.Println("analyze the segment: ", a)

	cut = seg.Trim(cut)
	fmt.Println("cut all: ", cut)

	fmt.Println(seg.String(text, true))
	fmt.Println(seg.Slice(text, true))
}

func cutPos() {
	po := seg.Pos(text, true)
	fmt.Println("pos: ", po)
	po = seg.TrimPos(po)
	fmt.Println("trim pos: ", po)

	pos.WithGse(seg)
	po = posSeg.Cut(text, true)
	fmt.Println("pos: ", po)

	po = posSeg.TrimWithPos(po, "zg")
	fmt.Println("trim pos: ", po)
}

func segCut() {
	// Text Segmentation
	tb := []byte(text)
	fmt.Println(seg.String(text, true))

	segments := seg.Segment(tb)
	// Handle word segmentation results, search mode
	fmt.Println(gse.ToString(segments, true))
}

```

[Look at an custom dictionary example](/examples/dict/main.go)

```Go
package main

import (
	"fmt"
	_ "embed"

	"github.com/go-ego/gse"
)

//go:embed test_en_dict3.txt
var testDict string

func main() {
	// var seg gse.Segmenter
	// seg.LoadDict("zh, testdata/zh/test_dict.txt, testdata/zh/test_dict1.txt")
	// seg.LoadStop()
	seg, err := gse.NewEmbed("zh, word 20 n"+testDict, "en")
	// seg.LoadDictEmbed()
	seg.LoadStopEmbed()

	text1 := "Hello world, こんにちは世界, 你好世界!"
	s1 := seg.Cut(text1, true)
	fmt.Println(s1)
	fmt.Println("trim: ", seg.Trim(s1))
	fmt.Println("stop: ", seg.Stop(s1))
	fmt.Println(seg.String(text1, true))

	segments := seg.Segment([]byte(text1))
	fmt.Println(gse.ToString(segments))
}
```

[Look at an Chinese example](/examples/main.go)

[Look at an Japanese example](/examples/jp/main.go)

## Elasticsearch

How to use it with elasticsearch?

[go-gse-elastic](https://github.com/vcaesar/go-gse-elastic)

## Authors

- [Maintainers](https://github.com/orgs/go-ego/people)
- [Contributors](https://github.com/go-ego/gse/graphs/contributors)

## License

Gse is primarily distributed under the terms of "the Apache License (Version 2.0)".
See [LICENSE-APACHE](http://www.apache.org/licenses/LICENSE-2.0), [LICENSE](https://github.com/go-vgo/robotgo/blob/master/LICENSE).

Thanks for [sego](https://github.com/huichen/sego) and [jieba](https://github.com/fxsjy/jieba)([jiebago](https://github.com/wangbin/jiebago)).

Go library for performing Unicode Text Segmentation as described in [Unicode Standard Annex #29](https://www.unicode.org/reports/tr29/)

# segment

[![Tests](https://github.com/blevesearch/segment/workflows/Tests/badge.svg?branch=master&event=push)](https://github.com/blevesearch/segment/actions?query=workflow%3ATests+event%3Apush+branch%3Amaster)

A Go library for performing Unicode Text Segmentation
as described in [Unicode Standard Annex #29](http://www.unicode.org/reports/tr29/)

## Features

* Currently only segmentation at Word Boundaries is supported.

## License

Apache License Version 2.0

## Usage

The functionality is exposed in two ways:

1.  You can use a bufio.Scanner with the SplitWords implementation of SplitFunc.
The SplitWords function will identify the appropriate word boundaries in the input
text and the Scanner will return tokens at the appropriate place.

		scanner := bufio.NewScanner(...)
		scanner.Split(segment.SplitWords)
		for scanner.Scan() {
			tokenBytes := scanner.Bytes()
		}
		if err := scanner.Err(); err != nil {
			t.Fatal(err)
		}

2.  Sometimes you would also like information returned about the type of token.
To do this we have introduce a new type named Segmenter.  It works just like Scanner
but additionally a token type is returned.

		segmenter := segment.NewWordSegmenter(...)
		for segmenter.Segment() {
			tokenBytes := segmenter.Bytes())
			tokenType := segmenter.Type()
		}
		if err := segmenter.Err(); err != nil {
			t.Fatal(err)
		}

## Choosing Implementation

By default segment does NOT use the fastest runtime implementation.  The reason is that it adds approximately 5s to compilation time and may require more than 1GB of ram on the machine performing compilation.

However, you can choose to build with the fastest runtime implementation by passing the build tag as follows:

		-tags 'prod'

## Generating Code

Several components in this package are generated.

1.  Several Ragel rules files are generated from Unicode properties files.
2.  Ragel machine is generated from the Ragel rules.
3.  Test tables are generated from the Unicode test files.

All of these can be generated by running:

		go generate

## Fuzzing

There is support for fuzzing the segment library with [go-fuzz](https://github.com/dvyukov/go-fuzz).

1.  Install go-fuzz if you haven't already:

		go get github.com/dvyukov/go-fuzz/go-fuzz
		go get github.com/dvyukov/go-fuzz/go-fuzz-build

2.  Build the package with go-fuzz:

		go-fuzz-build github.com/blevesearch/segment

3.  Convert the Unicode provided test cases into the initial corpus for go-fuzz:

		go test -v -run=TestGenerateWordSegmentFuzz -tags gofuzz_generate

4.  Run go-fuzz:

		go-fuzz -bin=segment-fuzz.zip -workdir=workdir

## Status


[![Build Status](https://travis-ci.org/blevesearch/segment.svg?branch=master)](https://travis-ci.org/blevesearch/segment)

[![Coverage Status](https://img.shields.io/coveralls/blevesearch/segment.svg)](https://coveralls.io/r/blevesearch/segment?branch=master)

[![GoDoc](https://godoc.org/github.com/blevesearch/segment?status.svg)](https://godoc.org/github.com/blevesearch/segment)

segment

Sentence tokenizer: converts text into a list of sentences

[![release](https://github.com/neurosnap/sentences/actions/workflows/release.yml/badge.svg)](https://github.com/neurosnap/sentences/actions/workflows/release.yml)
[![GODOC](https://godoc.org/github.com/nathany/looper?status.svg)](https://godoc.org/github.com/neurosnap/sentences)
![MIT](https://img.shields.io/packagist/l/doctrine/orm.svg)
[![Go Report Card](https://goreportcard.com/badge/github.com/neurosnap/sentences)](https://goreportcard.com/report/github.com/neurosnap/sentences)

# Sentences - A command line sentence tokenizer

This command line utility will convert a blob of text into a list of sentences.

* [Demo](https://sentences-231000.appspot.com/)
* [Docs](https://godoc.org/github.com/neurosnap/sentences)

## Features

* Supports multiple languages (english, czech, dutch, estonian, finnish,
  german, greek, italian, norwegian, polish, portuguese, slovene, and turkish)
* Zero dependencies
* Extendable
* Fast

## Install

### arch

[aur](https://aur.archlinux.org/packages/sentences-bin)

### mac

```
brew tap neurosnap/sentences
brew install sentences
```

### other

Or you can find the pre-built binaries on [the github
releases page](https://github.com/neurosnap/sentences/releases).

### using golang

```
go get github.com/neurosnap/sentences
go install github.com/neurosnap/sentences/cmd/sentences
```

## Command

![Command line](sentences.gif?raw=true)

## Get it

```
go get github.com/neurosnap/sentences
```

## Use it

```Go
import (
    "fmt"
    "os"

    "github.com/neurosnap/sentences"
)

func main() {
    text := `A perennial also-ran, Stallings won his seat when longtime lawmaker David Holmes
    died 11 days after the filing deadline. Suddenly, Stallings was a shoo-in, not
    the long shot. In short order, the Legislature attempted to pass a law allowing
    former U.S. Rep. Carolyn Cheeks Kilpatrick to file; Stallings challenged the
    law in court and won. Kilpatrick mounted a write-in campaign, but Stallings won.`

    // download the training data from this repo (./data) and save it somewhere
    b, _ := os.ReadFile("./path/to/english.json")

    // load the training data
    training, _ := sentences.LoadTraining(b)

    // create the default sentence tokenizer
    tokenizer := sentences.NewSentenceTokenizer(training)
    sentences := tokenizer.Tokenize(text)

    for _, s := range sentences {
        fmt.Println(s.Text)
    }
}
```

## English

This package attempts to fix some problems I noticed for english.

```Go
import (
    "fmt"

    "github.com/neurosnap/sentences/english"
)

func main() {
    text := "Hi there. Does this really work?"

    tokenizer, err := english.NewSentenceTokenizer(nil)
    if err != nil {
        panic(err)
    }

    sentences := tokenizer.Tokenize(text)
    for _, s := range sentences {
        fmt.Println(s.Text)
    }
}
```

## Contributing

I need help maintaining this library.  If you are interested in contributing
to this library then please start by looking at the [golden-rules](https://github.com/neurosnap/sentences/tree/golden-rule) branch which
tests the [Golden Rules](https://github.com/diasks2/pragmatic_segmenter/blob/master/README.md#the-golden-rules)
for english sentence tokenization created by the [Pragmatic Segmenter](https://github.com/diasks2/pragmatic_segmenter)
library.

Create an issue for a particular failing test and submit an issue/PR.

I'm happy to help anyone willing to contribute.

## Customize

`sentences` was built around composability, most major components of this package
can be extended.

Eager to make ad-hoc changes but don't know how to start?
Have a look at `github.com/neurosnap/sentences/english` for a solid example.

## Notice

I have not tested this tokenizer in any other language besides English.  By default
the command line utility loads english. I welcome anyone willing to test the
other languages to submit updates as needed.

A primary goal for this package is to be multilingual so I'm willing to help in
any way possible.

This library is a port of the [nltk's](http://www.nltk.org) punkt tokenizer.

## A Punkt Tokenizer

An unsupervised multilingual sentence boundary detection library for golang.
The way the punkt system accomplishes this goal is through training the tokenizer
with text in that given language.  Once the likelihoods of abbreviations, collocations,
and sentence starters are determined, finding sentence boundaries becomes easier.

There are many problems that arise when tokenizing text into sentences, the primary
issue being abbreviations.  The punkt system attempts to determine whether a  word
is an abbreviation, an end to a sentence, or even both through training the system with text
in the given language.  The punkt system incorporates both token- and type-based
analysis on the text through two different phases of annotation.

[Unsupervised multilingual sentence boundary detection](http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=BAE5C34E5C3B9DC60DFC4D93B85D8BB1?doi=10.1.1.85.5017&rep=rep1&type=pdf)

## Performance

Using [Brown Corpus](http://www.hit.uib.no/icame/brown/bcm.html) which is annotated American English
text, we compare this package with other libraries across multiple programming languages.

|Library    | Avg Speed (s, 10 runs) | Accuracy (%)
|:----------|:----------------------:|:-----------:
| Sentences | 1.96                   | 98.95
| NLTK      | 5.22                   | 99.21


sentences

This is a Go implementation of [jieba](https://github.com/fxsjy/jieba) which a Chinese word splitting algorithm

# GoJieba

[![Test](https://github.com/yanyiwu/gojieba/actions/workflows/test.yml/badge.svg)](https://github.com/yanyiwu/gojieba/actions/workflows/test.yml)
[![Author](https://img.shields.io/badge/author-@yanyiwu-blue.svg?style=flat)](http://yanyiwu.com/) 
[![Tag](https://img.shields.io/github/v/tag/yanyiwu/gojieba.svg)](https://github.com/yanyiwu/gojieba/releases)
[![Performance](https://img.shields.io/badge/performance-excellent-brightgreen.svg?style=flat)](http://yanyiwu.com/work/2015/06/14/jieba-series-performance-test.html) 
[![License](https://img.shields.io/badge/license-MIT-yellow.svg?style=flat)](http://yanyiwu.mit-license.org)
[![GoDoc](https://godoc.org/github.com/yanyiwu/gojieba?status.svg)](https://godoc.org/github.com/yanyiwu/gojieba)
[![Coverage Status](https://coveralls.io/repos/yanyiwu/gojieba/badge.svg?branch=master&service=github)](https://coveralls.io/github/yanyiwu/gojieba?branch=master)
[![Go Report Card](https://goreportcard.com/badge/yanyiwu/gojieba)](https://goreportcard.com/report/yanyiwu/gojieba)
[![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/avelino/awesome-go) 

[GoJieba]是"结巴"中文分词的Golang语言版本。

## 简介

+ 支持多种分词方式，包括: 最大概率模式, HMM新词发现模式, 搜索引擎模式, 全模式
+ 核心算法底层由C++实现，性能高效。
+ 字典路径可配置，NewJieba(...string), NewExtractor(...string) 可变形参，当参数为空时使用默认词典(推荐方式)
+ C++ 依赖 ([CppJieba] 及其 [limonp] 库) 已直接打包在 `deps/` 目录中，无需任何 git submodule 初始化，`go get` / `go mod vendor` 开箱即用。

## 用法

```bash
go get github.com/yanyiwu/gojieba
```

`gojieba` 依赖 cgo 和 C++。如果是交叉编译，请确保目标平台的 C/C++ 交叉工具链已经安装，并显式开启 cgo；`CGO_ENABLED=0` 的纯 Go 交叉编译方式不适用于这个库。

`gojieba` requires cgo and C++; cross-compilation needs `CGO_ENABLED=1` plus a target C/C++ toolchain via `CC`/`CXX`.

例如，交叉编译到 Linux amd64 时，需要类似下面这样提供目标平台工具链:

```bash
CGO_ENABLED=1 \
CC=x86_64-linux-gnu-gcc \
CXX=x86_64-linux-gnu-g++ \
GOOS=linux \
GOARCH=amd64 \
go build
```

分词示例

```golang
package main

import (
	"fmt"
	"strings"

	"github.com/yanyiwu/gojieba"
)

func main() {
	var s string
	var words []string
	use_hmm := true
	x := gojieba.NewJieba()
	defer x.Free()

	s = "我来到北京清华大学"
	words = x.CutAll(s)
	fmt.Println(s)
	fmt.Println("全模式:", strings.Join(words, "/"))

	words = x.Cut(s, use_hmm)
	fmt.Println(s)
	fmt.Println("精确模式:", strings.Join(words, "/"))
	s = "比特币"
	words = x.Cut(s, use_hmm)
	fmt.Println(s)
	fmt.Println("精确模式:", strings.Join(words, "/"))

	x.AddWord("比特币")
	// `AddWordEx` 支持指定词语的权重，作为 `AddWord` 权重太低加词失败的补充。
	// `tag` 参数可以为空字符串，也可以指定词性。
	// x.AddWordEx("比特币", 100000, "")
	s = "比特币"
	words = x.Cut(s, use_hmm)
	fmt.Println(s)
	fmt.Println("添加词典后,精确模式:", strings.Join(words, "/"))

	s = "他来到了网易杭研大厦"
	words = x.Cut(s, use_hmm)
	fmt.Println(s)
	fmt.Println("新词识别:", strings.Join(words, "/"))

	s = "小明硕士毕业于中国科学院计算所，后在日本京都大学深造"
	words = x.CutForSearch(s, use_hmm)
	fmt.Println(s)
	fmt.Println("搜索引擎模式:", strings.Join(words, "/"))

	s = "长春市长春药店"
	words = x.Tag(s)
	fmt.Println(s)
	fmt.Println("词性标注:", strings.Join(words, ","))

	s = "区块链"
	words = x.Tag(s)
	fmt.Println(s)
	fmt.Println("词性标注:", strings.Join(words, ","))

	s = "长江大桥"
	words = x.CutForSearch(s, !use_hmm)
	fmt.Println(s)
	fmt.Println("搜索引擎模式:", strings.Join(words, "/"))

	wordinfos := x.Tokenize(s, gojieba.SearchMode, !use_hmm)
	fmt.Println(s)
	fmt.Println("Tokenize:(搜索引擎模式)", wordinfos)

	wordinfos = x.Tokenize(s, gojieba.DefaultMode, !use_hmm)
	fmt.Println(s)
	fmt.Println("Tokenize:(默认模式)", wordinfos)

	keywords := x.ExtractWithWeight(s, 5)
	fmt.Println("Extract:", keywords)
}
```

```
我来到北京清华大学
全模式: 我/来到/北京/清华/清华大学/华大/大学
我来到北京清华大学
精确模式: 我/来到/北京/清华大学
比特币
精确模式: 比特/币
比特币
添加词典后,精确模式: 比特币
他来到了网易杭研大厦
新词识别: 他/来到/了/网易/杭研/大厦
小明硕士毕业于中国科学院计算所，后在日本京都大学深造
搜索引擎模式: 小明/硕士/毕业/于/中国/科学/学院/科学院/中国科学院/计算/计算所/，/后/在/日本/京都/大学/日本京都大学/深造
长春市长春药店
词性标注: 长春市/ns,长春/ns,药店/n
区块链
词性标注: 区块链/nz
长江大桥
搜索引擎模式: 长江/大桥/长江大桥
长江大桥
Tokenize: [{长江 0 6} {大桥 6 12} {长江大桥 0 12}]
```

See Details in [gojieba-demo](http://github.com/yanyiwu/gojieba-demo)
See example in [jieba_test](jieba_test.go), [extractor_test](extractor_test.go)

## Benchmark

[Jieba中文分词系列性能评测]

Unittest

```bash
go test ./...
```

Benchmark

```bash
go test -bench "Jieba" -test.benchtime 10s
go test -bench "Extractor" -test.benchtime 10s
```

## Contributors

### Code Contributors

This project exists thanks to all the people who contribute.
<a href="https://github.com/yanyiwu/gojieba/graphs/contributors"><img src="https://opencollective.com/gojieba/contributors.svg?width=890&button=false" /></a>

[CppJieba]:http://github.com/yanyiwu/cppjieba
[GoJieba]:http://github.com/yanyiwu/gojieba
[Jieba]:https://github.com/fxsjy/jieba
[limonp]:https://github.com/yanyiwu/limonp
[Jieba中文分词系列性能评测]:http://yanyiwu.com/work/2015/06/14/jieba-series-performance-test.html


gojieba

A tokenizer based on the dictionary and Bigram language models for Golang. (Now only support chinese segmentation)

# gotokenizer [![GoDoc](https://godoc.org/github.com/xujiajun/gotokenizer?status.svg)](https://godoc.org/github.com/xujiajun/gotokenizer) <a href="https://travis-ci.org/xujiajun/gotokenizer"><img src="https://travis-ci.org/xujiajun/gotokenizer.svg?branch=master" alt="Build Status"></a> [![Coverage Status](https://coveralls.io/repos/github/xujiajun/gotokenizer/badge.svg?branch=master)](https://coveralls.io/github/xujiajun/gotokenizer?branch=master) [![Go Report Card](https://goreportcard.com/badge/github.com/xujiajun/gotokenizer)](https://goreportcard.com/report/github.com/xujiajun/gotokenizer) [![License](https://img.shields.io/badge/license-Apache2.0-blue.svg?style=flat-square)](https://opensource.org/licenses/Apache-2.0) [![Awesome](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go#natural-language-processing) 
A tokenizer based on the dictionary and Bigram language models for Go.  (Now only support chinese segmentation)

## Motivation

I wanted a simple tokenizer that has no unnecessary overhead using the standard library only, following good practices and well tested code.

## Features

* Support Maximum Matching Method
* Support Minimum Matching Method
* Support Reverse Maximum Matching
* Support Reverse Minimum Matching
* Support Bidirectional Maximum Matching
* Support Bidirectional Minimum Matching
* Support using Stop Tokens
* Support Custom word Filter

## Installation

```
go get -u github.com/xujiajun/gotokenizer
```

## Usage

```
package main

import (
	"fmt"

	"github.com/xujiajun/gotokenizer"
)

func main() {
	text := "gotokenizer是一款基于字典和Bigram模型纯go语言编写的分词器，支持6种分词算法。支持stopToken过滤和自定义word过滤功能。"

	dictPath := "/Users/xujiajun/go/src/github.com/xujiajun/gotokenizer/data/zh/dict.txt"
	// NewMaxMatch default wordFilter is NumAndLetterWordFilter
	mm := gotokenizer.NewMaxMatch(dictPath)
	// load dict
	mm.LoadDict()

	fmt.Println(mm.Get(text)) //[gotokenizer 是 一款 基于 字典 和 Bigram 模型 纯 go 语言 编写 的 分词器 ， 支持 6 种 分词 算法 。 支持 stopToken 过滤 和 自定义 word 过滤 功能 。] <nil>

	// enabled filter stop tokens 
	mm.EnabledFilterStopToken = true
	mm.StopTokens = gotokenizer.NewStopTokens()
	stopTokenDicPath := "/Users/xujiajun/go/src/github.com/xujiajun/gotokenizer/data/zh/stop_tokens.txt"
	mm.StopTokens.Load(stopTokenDicPath)

	fmt.Println(mm.Get(text)) //[gotokenizer 一款 字典 Bigram 模型 go 语言 编写 分词器 支持 6 种 分词 算法 支持 stopToken 过滤 自定义 word 过滤 功能] <nil>
	fmt.Println(mm.GetFrequency(text)) //map[6:1 种:1 算法:1 过滤:2 支持:2 Bigram:1 模型:1 编写:1 gotokenizer:1 go:1 分词器:1 分词:1 word:1 功能:1 一款:1 语言:1 stopToken:1 自定义:1 字典:1] <nil>

}

```

> More examples see tests

## Contributing

If you'd like to help out with the project. You can put up a Pull Request.


## Author

* [xujiajun](https://github.com/xujiajun)

## License

The gotokenizer is open-sourced software licensed under the [Apache-2.0](https://opensource.org/licenses/Apache-2.0)

## Acknowledgements

This package is inspired by the following:

https://github.com/ysc/word


The Best Go Libraries For Natural Language Processing - Tokenizers (9)