# Elastic

**This is a development branch that is actively being worked on. DO NOT USE IN PRODUCTION! If you want to use stable versions of Elastic, please use Go modules for the 7.x release (or later) or a dependency manager like [dep](https://github.com/golang/dep) for earlier releases.**

Elastic is an [Elasticsearch](http://www.elasticsearch.org/) client for the
[Go](http://www.golang.org/) programming language.

[![Build Status](https://github.com/olivere/elastic/workflows/Test/badge.svg)](https://github.com/olivere/elastic/actions)
[![Godoc](http://img.shields.io/badge/godoc-reference-blue.svg?style=flat)](https://pkg.go.dev/github.com/olivere/elastic/v7?tab=doc)
[![license](http://img.shields.io/badge/license-MIT-red.svg?style=flat)](https://raw.githubusercontent.com/olivere/elastic/master/LICENSE)

See the [wiki](https://github.com/olivere/elastic/wiki) for additional information about Elastic.

<a href="https://www.buymeacoffee.com/Bjd96U8fm" target="_blank"><img src="https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png" alt="Buy Me A Coffee" style="height: 41px !important;width: 174px !important;box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;-webkit-box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;" ></a>

## Releases

**The release branches (e.g. [`release-branch.v7`](https://github.com/olivere/elastic/tree/release-branch.v7))
are actively being worked on and can break at any time.
If you want to use stable versions of Elastic, please use Go modules.**

Here's the version matrix:

Elasticsearch version | Elastic version  | Package URL | Remarks |
----------------------|------------------|-------------|---------|
7.x                   | 7.0              | [`github.com/olivere/elastic/v7`](https://github.com/olivere/elastic) ([source](https://github.com/olivere/elastic/tree/release-branch.v7) [doc](http://godoc.org/github.com/olivere/elastic)) | Use Go modules.
6.x                   | 6.0              | [`github.com/olivere/elastic`](https://github.com/olivere/elastic) ([source](https://github.com/olivere/elastic/tree/release-branch.v6) [doc](http://godoc.org/github.com/olivere/elastic)) | Use a dependency manager (see below).
5.x                   | 5.0              | [`gopkg.in/olivere/elastic.v5`](https://gopkg.in/olivere/elastic.v5) ([source](https://github.com/olivere/elastic/tree/release-branch.v5) [doc](http://godoc.org/gopkg.in/olivere/elastic.v5)) | Actively maintained.
2.x                   | 3.0              | [`gopkg.in/olivere/elastic.v3`](https://gopkg.in/olivere/elastic.v3) ([source](https://github.com/olivere/elastic/tree/release-branch.v3) [doc](http://godoc.org/gopkg.in/olivere/elastic.v3)) | Deprecated. Please update.
1.x                   | 2.0              | [`gopkg.in/olivere/elastic.v2`](https://gopkg.in/olivere/elastic.v2) ([source](https://github.com/olivere/elastic/tree/release-branch.v2) [doc](http://godoc.org/gopkg.in/olivere/elastic.v2)) | Deprecated. Please update.
0.9-1.3               | 1.0              | [`gopkg.in/olivere/elastic.v1`](https://gopkg.in/olivere/elastic.v1) ([source](https://github.com/olivere/elastic/tree/release-branch.v1) [doc](http://godoc.org/gopkg.in/olivere/elastic.v1)) | Deprecated. Please update.

**Example:**

You have installed Elasticsearch 7.0.0 and want to use Elastic.
As listed above, you should use Elastic 7.0 (code is in `release-branch.v7`).

To use the required version of Elastic in your application, you
should use [Go modules](https://github.com/golang/go/wiki/Modules)
to manage dependencies. Make sure to use a version such as `7.0.0` or later.

To use Elastic, import:

```go
import "github.com/olivere/elastic/v7"
```

### Elastic 7.0

Elastic 7.0 targets Elasticsearch 7.x which [was released on April 10th 2019](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/release-notes-7.0.0.html).

As always with major version, there are a lot of [breaking changes](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/release-notes-7.0.0.html#breaking-7.0.0).
We will use this as an opportunity to [clean up and refactor Elastic](https://github.com/olivere/elastic/blob/release-branch.v7/CHANGELOG-7.0.md),
as we already did in earlier (major) releases.

### Elastic 6.0

Elastic 6.0 targets Elasticsearch 6.x which was [released on 14th November 2017](https://www.elastic.co/blog/elasticsearch-6-0-0-released).

Notice that there are a lot of [breaking changes in Elasticsearch 6.0](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/breaking-changes-6.0.html)
and we used this as an opportunity to [clean up and refactor Elastic](https://github.com/olivere/elastic/blob/release-branch.v6/CHANGELOG-6.0.md)
as we did in the transition from earlier versions of Elastic.

### Elastic 5.0

Elastic 5.0 targets Elasticsearch 5.0.0 and later. Elasticsearch 5.0.0 was
[released on 26th October 2016](https://www.elastic.co/blog/elasticsearch-5-0-0-released).

Notice that there are will be a lot of [breaking changes in Elasticsearch 5.0](https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking-changes-5.0.html)
and we used this as an opportunity to [clean up and refactor Elastic](https://github.com/olivere/elastic/blob/release-branch.v5/CHANGELOG-5.0.md)
as we did in the transition from Elastic 2.0 (for Elasticsearch 1.x) to Elastic 3.0 (for Elasticsearch 2.x).

Furthermore, the jump in version numbers will give us a chance to be in sync with the Elastic Stack.

### Elastic 3.0

Elastic 3.0 targets Elasticsearch 2.x and is published via [`gopkg.in/olivere/elastic.v3`](https://gopkg.in/olivere/elastic.v3).

Elastic 3.0 will only get critical bug fixes. You should update to a recent version.

### Elastic 2.0

Elastic 2.0 targets Elasticsearch 1.x and is published via [`gopkg.in/olivere/elastic.v2`](https://gopkg.in/olivere/elastic.v2).

Elastic 2.0 will only get critical bug fixes. You should update to a recent version.

### Elastic 1.0

Elastic 1.0 is deprecated. You should really update Elasticsearch and Elastic
to a recent version.

However, if you cannot update for some reason, don't worry. Version 1.0 is
still available. All you need to do is go-get it and change your import path
as described above.


## Status

We use Elastic in production since 2012. Elastic is stable but the API changes
now and then. We strive for API compatibility.
However, Elasticsearch sometimes introduces [breaking changes](https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes.html)
and we sometimes have to adapt.

Having said that, there have been no big API changes that required you
to rewrite your application big time. More often than not it's renaming APIs
and adding/removing features so that Elastic is in sync with Elasticsearch.

Elastic has been used in production starting with Elasticsearch 0.90 up to recent 7.x
versions.
We recently switched to [GitHub Actions for testing](https://github.com/olivere/elastic/actions).
Before that, we used [Travis CI](https://travis-ci.org/olivere/elastic) successfully for years).

Elasticsearch has quite a few features. Most of them are implemented
by Elastic. I add features and APIs as required. It's straightforward
to implement missing pieces. I'm accepting pull requests :-)

Having said that, I hope you find the project useful.


## Getting Started

The first thing you do is to create a [Client](https://github.com/olivere/elastic/blob/master/client.go).
The client connects to Elasticsearch on `http://127.0.0.1:9200` by default.

You typically create one client for your app. Here's a complete example of
creating a client, creating an index, adding a document, executing a search etc.

An example is available [here](https://olivere.github.io/elastic/).

Here's a [link to a complete working example for v6](https://gist.github.com/olivere/e4a376b4783c0914e44ea4f745ce2ebf).

Here are a few tips on how to get used to Elastic:

1. Head over to the [Wiki](https://github.com/olivere/elastic/wiki) for detailed information and
   topics like e.g. [how to add a middleware](https://github.com/olivere/elastic/wiki/HttpTransport)
   or how to [connect to AWS](https://github.com/olivere/elastic/wiki/Using-with-AWS-Elasticsearch-Service).
2. If you are unsure how to implement something, read the tests (all `_test.go` files).
   They not only serve as a guard against changes, but also as a reference.
3. The [recipes](https://github.com/olivere/elastic/tree/release-branch.v6/recipes)
   contains small examples on how to implement something, e.g. bulk indexing, scrolling etc.


## API Status

### Document APIs

- [x] Index API
- [x] Get API
- [x] Delete API
- [x] Delete By Query API
- [x] Update API
- [x] Update By Query API
- [x] Multi Get API
- [x] Bulk API
- [x] Reindex API
- [x] Term Vectors
- [x] Multi termvectors API

### Search APIs

- [x] Search
- [x] Search Template
- [ ] Multi Search Template
- [x] Search Shards API
- [x] Suggesters
  - [x] Term Suggester
  - [x] Phrase Suggester
  - [x] Completion Suggester
  - [x] Context Suggester
- [x] Multi Search API
- [x] Count API
- [x] Validate API
- [x] Explain API
- [x] Profile API
- [x] Field Capabilities API

### Aggregations

- Metrics Aggregations
  - [x] Avg
  - [ ] Boxplot (X-pack)
  - [x] Cardinality
  - [x] Extended Stats
  - [x] Geo Bounds
  - [x] Geo Centroid
  - [x] Matrix stats
  - [x] Max
  - [x] Median absolute deviation
  - [x] Min
  - [x] Percentile Ranks
  - [x] Percentiles
  - [ ] Rate (X-pack)
  - [ ] Scripted Metric
  - [x] Stats
  - [ ] String stats (X-pack)
  - [x] Sum
  - [ ] T-test (X-pack)
  - [x] Top Hits
  - [x] Top metrics (X-pack)
  - [x] Value Count
  - [x] Weighted avg
- Bucket Aggregations
  - [x] Adjacency Matrix
  - [x] Auto-interval Date Histogram
  - [x] Children
  - [x] Composite
  - [x] Date Histogram
  - [x] Date Range
  - [x] Diversified Sampler
  - [x] Filter
  - [x] Filters
  - [x] Geo Distance
  - [x] Geohash Grid
  - [x] Geotile grid
  - [x] Global
  - [x] Histogram
  - [x] IP Range
  - [x] Missing
  - [x] Nested
  - [ ] Parent
  - [x] Range
  - [ ] Rare terms
  - [x] Reverse Nested
  - [x] Sampler
  - [x] Significant Terms
  - [x] Significant Text
  - [x] Terms
  - [ ] Variable width histogram
- Pipeline Aggregations
  - [x] Avg Bucket
  - [x] Bucket Script
  - [x] Bucket Selector
  - [x] Bucket Sort
  - [ ] Cumulative cardinality (X-pack)
  - [x] Cumulative Sum
  - [x] Derivative
  - [ ] Extended Stats Bucket
  - [ ] Inference bucket (X-pack)
  - [x] Max Bucket
  - [x] Min Bucket
  - [x] Moving Average
  - [x] Moving function
  - [ ] Moving percentiles (X-pack)
  - [ ] Normalize (X-pack)
  - [x] Percentiles Bucket
  - [x] Serial Differencing
  - [x] Stats Bucket
  - [x] Sum Bucket
- [x] Aggregation Metadata

### Indices APIs

- [x] Create Index
- [x] Delete Index
- [x] Get Index
- [x] Indices Exists
- [x] Open / Close Index
- [x] Shrink Index
- [x] Rollover Index
- [x] Put Mapping
- [x] Get Mapping
- [x] Get Field Mapping
- [x] Types Exists
- [x] Index Aliases
- [x] Update Indices Settings
- [x] Get Settings
- [x] Analyze
  - [x] Explain Analyze
- [x] Index Templates
- [x] Indices Stats
- [x] Indices Segments
- [ ] Indices Recovery
- [ ] Indices Shard Stores
- [x] Clear Cache
- [x] Flush
  - [x] Synced Flush
- [x] Refresh
- [x] Force Merge

### Index Lifecycle Management APIs

- [x] Create Policy
- [x] Get Policy
- [x] Delete Policy
- [ ] Move to Step
- [ ] Remove Policy
- [ ] Retry Policy
- [ ] Get Ilm Status
- [ ] Explain Lifecycle
- [ ] Start Ilm
- [ ] Stop Ilm

### cat APIs

- [X] cat aliases
- [X] cat allocation
- [X] cat count
- [X] cat fielddata
- [X] cat health
- [X] cat indices
- [x] cat master
- [ ] cat nodeattrs
- [ ] cat nodes
- [ ] cat pending tasks
- [ ] cat plugins
- [ ] cat recovery
- [ ] cat repositories
- [ ] cat thread pool
- [ ] cat shards
- [ ] cat segments
- [X] cat snapshots
- [ ] cat templates

### Cluster APIs

- [x] Cluster Health
- [x] Cluster State
- [x] Cluster Stats
- [ ] Pending Cluster Tasks
- [x] Cluster Reroute
- [ ] Cluster Update Settings
- [x] Nodes Stats
- [x] Nodes Info
- [ ] Nodes Feature Usage
- [ ] Remote Cluster Info
- [x] Task Management API
- [ ] Nodes hot_threads
- [ ] Cluster Allocation Explain API

### Rollup APIs (XPack)
- [x] Create Job
- [x] Delete Job
- [x] Get Job
- [x] Start Job
- [x] Stop Job

### Query DSL

- [x] Match All Query
- [x] Inner hits
- Full text queries
  - [x] Match Query
  - [x] Match Boolean Prefix Query
  - [x] Match Phrase Query
  - [x] Match Phrase Prefix Query
  - [x] Multi Match Query
  - [x] Common Terms Query
  - [x] Query String Query
  - [x] Simple Query String Query
  - [x] Combined Fields Query
  - [x] Intervals Query
- Term level queries
  - [x] Term Query
  - [x] Terms Query
  - [x] Terms Set Query
  - [x] Range Query
  - [x] Exists Query
  - [x] Prefix Query
  - [x] Wildcard Query
  - [x] Regexp Query
  - [x] Fuzzy Query
  - [x] Type Query
  - [x] Ids Query
- Compound queries
  - [x] Constant Score Query
  - [x] Bool Query
  - [x] Dis Max Query
  - [x] Function Score Query
  - [x] Boosting Query
- Joining queries
  - [x] Nested Query
  - [x] Has Child Query
  - [x] Has Parent Query
  - [x] Parent Id Query
- Geo queries
  - [ ] GeoShape Query
  - [x] Geo Bounding Box Query
  - [x] Geo Distance Query
  - [x] Geo Polygon Query
- Specialized queries
  - [x] Distance Feature Query
  - [x] More Like This Query
  - [x] Script Query
  - [x] Script Score Query
  - [x] Percolate Query
- Span queries
  - [x] Span Term Query
  - [ ] Span Multi Term Query
  - [x] Span First Query
  - [x] Span Near Query
  - [ ] Span Or Query
  - [ ] Span Not Query
  - [ ] Span Containing Query
  - [ ] Span Within Query
  - [ ] Span Field Masking Query
- [ ] Minimum Should Match
- [ ] Multi Term Query Rewrite

### Modules

- Snapshot and Restore
  - [x] Repositories
  - [x] Snapshot get
  - [x] Snapshot create
  - [x] Snapshot delete
  - [ ] Restore
  - [ ] Snapshot status
  - [ ] Monitoring snapshot/restore status
  - [ ] Stopping currently running snapshot and restore
- Scripting
  - [x] GetScript
  - [x] PutScript
  - [x] DeleteScript

### Sorting

- [x] Sort by score
- [x] Sort by field
- [x] Sort by geo distance
- [x] Sort by script
- [x] Sort by doc

### Scrolling

Scrolling is supported via a  `ScrollService`. It supports an iterator-like interface.
The `ClearScroll` API is implemented as well.

A pattern for [efficiently scrolling in parallel](https://github.com/olivere/elastic/wiki/ScrollParallel)
is described in the [Wiki](https://github.com/olivere/elastic/wiki).

## How to contribute

Read [the contribution guidelines](https://github.com/olivere/elastic/blob/master/CONTRIBUTING.md).

## Credits

Thanks a lot for the great folks working hard on
[Elasticsearch](https://www.elastic.co/products/elasticsearch)
and
[Go](https://golang.org/).

Elastic uses portions of the
[uritemplates](https://github.com/jtacoma/uritemplates) library
by Joshua Tacoma,
[backoff](https://github.com/cenkalti/backoff) by Cenk Altı and
[leaktest](https://github.com/fortytw2/leaktest) by Ian Chiles.

## LICENSE

MIT-LICENSE. See [LICENSE](http://olivere.mit-license.org/)
or the LICENSE file provided in the repository for details.


elastic

```
 _____ _         _     ____ _____ ___  ____  ____   ___   _
| ____| |       / \   / ___|_   _|_ _|/ ___|/ ___| / _ \ | |
|  _| | |      / _ \  \___ \ | |  | || |    \___ \| | | || |
| |___| |___  / ___ \  ___) || |  | || |___  ___) | |_| || |___
|_____|_____|/_/   \_\|____/ |_| |___|\____||____/ \__\_\|_____|
```

Overview
-----------
[![Build Status](https://travis-ci.org/cch123/elasticsql.svg?branch=master)](https://travis-ci.org/cch123/elasticsql)
[![Go Documentation](http://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)](https://godoc.org/github.com/cch123/elasticsql)
[![Coverage Status](https://coveralls.io/repos/github/cch123/elasticsql/badge.svg?branch=master)](https://coveralls.io/github/cch123/elasticsql?branch=master)
[![Go Report Card](https://goreportcard.com/badge/github.com/cch123/elasticsql)](https://goreportcard.com/report/github.com/cch123/elasticsql)



This tool converts sql to elasticsearch dsl

Currently support:

- [x] sql and expression
- [x] sql or expression
- [x] equal(=) support
- [x] not equal(!=) support
- [x] gt(>) support
- [x] gte(>=) support
- [x] lt(<) support
- [x] lte(<=) support
- [x] sql in (eg. id in (1,2,3) ) expression
- [x] sql not in (eg. id not in (1,2,3) ) expression
- [x] paren bool support (eg. where (a=1 or b=1) and (c=1 or d=1))
- [x] sql like expression (currently use match phrase, perhaps will change to wildcard in the future)
- [x] sql order by support
- [x] sql limit support
- [x] sql not like expression
- [x] field missing check
- [x] support aggregation like count(\*), count(field), min(field), max(field), avg(field)
- [x] support aggregation like stats(field), extended_stats(field), percentiles(field) which are not standard sql function
- [ ] null check expression(is null/is not null)
- [ ] join expression
- [ ] having support

Usage
-------------

> go get -u github.com/cch123/elasticsql

Demo :
```go
package main

import (
    "fmt"

    "github.com/cch123/elasticsql"
)

var sql = `
select * from aaa
where a=1 and x = '三个男人'
and create_time between '2015-01-01T00:00:00+0800' and '2016-01-01T00:00:00+0800'
and process_id > 1 order by id desc limit 100,10
`

func main() {
    dsl, esType, _ := elasticsql.Convert(sql)
    fmt.Println(dsl)
    fmt.Println(esType)
}

```
will produce :
```json
{
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "a": {
                            "query": "1",
                            "type": "phrase"
                        }
                    }
                },
                {
                    "match": {
                        "x": {
                            "query": "三个男人",
                            "type": "phrase"
                        }
                    }
                },
                {
                    "range": {
                        "create_time": {
                            "from": "2015-01-01T00:00:00+0800",
                            "to": "2016-01-01T00:00:00+0800"
                        }
                    }
                },
                {
                    "range": {
                        "process_id": {
                            "gt": "1"
                        }
                    }
                }
            ]
        }
    },
    "from": 100,
    "size": 10,
    "sort": [
        {
            "id": "desc"
        }
    ]
}

aaa
```

If your sql contains some keywords, eg. order, timestamp, don't forget to escape these fields as follows:

```
select * from `order` where `timestamp` = 1 and `desc`.id > 0
```

Warning
------------
To use this tool, you need to understand the term query and match phrase query of elasticsearch.

Setting a field to analyzed or not analyzed will get different results.

Details
------------
For more details of convertion, please refer to the [wiki](https://github.com/cch123/elasticsql/wiki)

Other info
------------
When writing this tool, I tried to avoid the deprecated dsl filters and aggregations, so it is compatible with most versions of the elasticsearch

If you have any advices or ideas, welcome to submit an issue or Pull Request!

License
-----------
MIT


elasticsql

elastigo v2.0 
-------------

[![Build Status](https://drone.io/github.com/mattbaird/elastigo/status.png)](https://drone.io/github.com/mattbaird/elastigo)

Big thanks to @alicebob for helping to get the drone.io CI working (note: the badge is being cached, known issue).

A Go (Golang) based Elasticsearch client, implements core api for Indexing and searching.   
GoDoc http://godoc.org/github.com/mattbaird/elastigo


NOTE: Based on the great work from Jeremy Shute, Elastigo now supports multiple connections. We attempted to make this backwards compatible, however in the end it wasn't possible, so we tagged the older single connection code as v1.0 and started work on v2.0.

If you want to use v1.0, you can use a tool like GoDep to make that possible. See http://bit.ly/VLG2et for full details.

The godep tool saves the exact version of the dependencies you’re building your project against, which means that upstream modifications in third-party dependencies won’t break your build.

```bash
go get github.com/tools/godep
```

Now, to pull in an existing project with godep:
```bash
	godep go get github.com/myuser/myproject
```

When your code compiles in your workspace, ala:

```bash
cd $HOME/gopath/src/github.com/myuser/myproject
# hack hack hack
go build ./...
```

You can freeze your dependencies thusly:

```bash
godep save github.com/myuser/myproject
git add Godeps
```

The godep tool will examine your code to find and save the transitive closure of your dependencies in the current directory, observing their versions.  If you want to restore or update these versions, see the documentation for the tool.

Note, in particular, that if your current directory contains a group of binaries or packages, you may save all of them at once:

```bash
godep save ./...
```

To get the Chef based Vagrantfile working, be sure to pull like so::

    # This will pull submodules.
    git clone --recursive git@github.com:mattbaird/elastigo.git

It's easier to use the ElasticSearch provided Docker image found here: https://github.com/dockerfile/elasticsearch

Non-persistent usage is:
```bash
docker run -d -p 9200:9200 -p 9300:9300 dockerfile/elasticsearch
```

Quick Start with Docker
=======================
Make sure docker is installed. If you are running docker on a mac, you must expose ports 9200 and 9300. Shut down docker:
```bash
boot2docker stop
```
and run
```bash
for i in {9200..9300}; do
 VBoxManage modifyvm "boot2docker-vm" --natpf1 "tcp-port$i,tcp,,$i,,$i";
 VBoxManage modifyvm "boot2docker-vm" --natpf1 "udp-port$i,udp,,$i,,$i";
done
```
The following will allow you to get the code, and run the tests against your docker based non-persistent elasticsearch:

```bash
docker run -d -p 9200:9200 -p 9300:9300 dockerfile/elasticsearch
git clone git@github.com:mattbaird/elastigo.git
cd elastigo
go get -u ./...
cd lib
go test -v -host localhost -loaddata
cd ..
go test -v ./...
```

Usage Examples - Currently out of date, being rewritten for v2.0
----------------------------------------------------------------

Adding content to Elasticsearch
-------------------------------

```go
import "github.com/mattbaird/elastigo/api"
import "github.com/mattbaird/elastigo/core"

type Tweet struct {
  User     string    `json:"user"`
  Message  string    `json:"message"`
}

// Set the Elasticsearch Host to Connect to
api.Domain = "localhost"
// api.Port = "9300"

// add single go struct entity
response, _ := core.Index("twitter", "tweet", "1", nil, Tweet{"kimchy", "Search is cool"})

// you have bytes
tw := Tweet{"kimchy", "Search is cool part 2"}
bytesLine, err := json.Marshal(tw)
response, _ := core.Index("twitter", "tweet", "2", nil, bytesLine)

// Bulk Indexing
t := time.Now()
core.IndexBulk("twitter", "tweet", "3", &t, Tweet{"kimchy", "Search is now cooler"})

// Search Using Raw json String
searchJson := `{
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}`
out, err := core.SearchRequest(true, "twitter", "tweet", searchJson, "")
if len(out.Hits.Hits) == 1 {
  fmt.Println(string(out.Hits.Hits[0].Source))
}
```

A Faceted, ranged Search using the `Search DSL` :

```go
import "github.com/mattbaird/elastigo/api"
import "github.com/mattbaird/elastigo/core"

// Set the Elasticsearch Host to Connect to
api.Domain = "localhost"
// api.Port = "9300"

out, err := Search("github").Size("1").Facet(
  Facet().Fields("actor").Size("500"),
).Query(
  Query().Range(
     Range().Field("created_at").From("2012-12-10T15:00:00-08:00").To("2012-12-10T15:10:00-08:00"),
  ).Search("add"),
).Result()
```

A Ranged Search using the `Search DSL` :

```go
out, err := Search("github").Type("Issues").Pretty().Query(
  Query().Range(
     Range().Field("created_at").From("2012-12-10T15:00:00-08:00").To("2012-12-10T15:10:00-08:00"),
  ).Search("add"),
).Result()
```

A Simple Search using the `Search DSL` :

```go
out, err := Search("github").Type("Issues").Size("100").Search("add").Result()
```

A Direct Search using the api :

```go
qry := map[string]interface{}{
  "query":map[string]interface{}{
     "term":map[string]string{"user": "kimchy"},
  },
}
core.SearchRequest(true, "github", "Issues", qry, "", 0)
```

A Direct Search using the query string Api :

```go
core.SearchUri("github", "Issues", "user:kimchy", "", 0)
```

A Filtered search `Search DSL` :

```go
out, err := Search("github").Filter(
  Filter().Exists("repository.name"),
).Result()
```

Adding content to Elasticsearch in Bulk
----------------------------------------------

```go
import "github.com/mattbaird/elastigo/api"
import "github.com/mattbaird/elastigo/core"

// Set the Elasticsearch Host to Connect to
api.Domain = "localhost"
// api.Port = "9300"

indexer := core.NewBulkIndexerErrors(10, 60)
done := make(chan bool)
indexer.Run(done)

go func() {
  for errBuf := range indexer.ErrorChannel {
    // just blissfully print errors forever
    fmt.Println(errBuf.Err)
  }
}()
for i := 0; i < 20; i++ {
  indexer.Index("twitter", "user", strconv.Itoa(i), "", nil, `{"name":"bob"}`, false)
}
done <- true
// Indexing might take a while. So make sure the program runs
// a little longer when trying this in main.
```

status updates
========================

* *2014-07-09* Version 2.0 development started. Focused on multi-connection support, using Dial idiom.
* *2014-5-21* Note: Drone.io tests are failing, I don't know why because the build and tests are working fine for me on my ubuntu box running the docker elasticsearch image. It's possible there is a timing issue. Any Ideas?
* *2013-9-27* Fleshing out cluster and indices APIs, updated vagrant image to 0.90.3
* *2013-7-10* Improvements/changes to bulk indexer (includes breaking changes to support TTL),
         Search dsl supports And/Or/Not
    * *SearchDsl* should still be considered beta at this
         point, there will be minor breaking changes as more of the
         elasticsearch feature set is implemented.
* *2013-1-26* expansion of search dsl for greater coverage
* *2012-12-30* new bulk indexing and search dsl
* *2012-10-12* early in development, not ready for production yet.

license
=======
    Copyright 2012 Matthew Baird, Aaron Raddon, Jeremy Shute and more!

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.


elastigo

# go-elasticsearch

The official Go client for [Elasticsearch](https://www.elastic.co/products/elasticsearch).

**[Download the latest version of Elasticsearch](https://www.elastic.co/downloads/elasticsearch)**
or
**[sign-up](https://cloud.elastic.co/registration?elektra=en-ess-sign-up-page)**
**for a free trial of Elastic Cloud**.

[![Go Reference](https://pkg.go.dev/badge/github.com/elastic/go-elasticsearch/v9.svg)](https://pkg.go.dev/github.com/elastic/go-elasticsearch/v9)
[![Go Report Card](https://goreportcard.com/badge/github.com/elastic/go-elasticsearch)](https://goreportcard.com/report/github.com/elastic/go-elasticsearch)
[![Build](https://github.com/elastic/go-elasticsearch/actions/workflows/build.yml/badge.svg?branch=main)](https://github.com/elastic/go-elasticsearch/actions/workflows/build.yml?query=branch%3Amain)
[![Unit](https://github.com/elastic/go-elasticsearch/actions/workflows/test-unit.yml/badge.svg?branch=main)](https://github.com/elastic/go-elasticsearch/actions/workflows/test-unit.yml?query=branch%3Amain)
[![Integration](https://github.com/elastic/go-elasticsearch/actions/workflows/test-integration.yml/badge.svg?branch=main)](https://github.com/elastic/go-elasticsearch/actions/workflows/test-integration.yml?query=branch%3Amain)
[![API](https://github.com/elastic/go-elasticsearch/actions/workflows/test-api.yml/badge.svg?branch=main)](https://github.com/elastic/go-elasticsearch/actions/workflows/test-api.yml?query=branch%3Amain)
[![Lint](https://github.com/elastic/go-elasticsearch/actions/workflows/lint.yml/badge.svg?branch=main&event=push)](https://github.com/elastic/go-elasticsearch/actions/workflows/lint.yml)

## Compatibility

### Go

Starting from version `8.12.0`, this library follows the Go language's [release policy](https://go.dev/doc/devel/release#policy). Each major Go release is supported until there are two newer major releases. For example, Go 1.5 was supported until the Go 1.7 release, and Go 1.6 was supported until the Go 1.8 release.

### Elasticsearch

Language clients are forward compatible; meaning that clients support communicating with greater or equal minor versions of Elasticsearch.
Elasticsearch language clients are only backward compatible with default distributions and without guarantees made.

When using Go modules, include the version in the import path, and specify either an explicit version or a branch:

    require github.com/elastic/go-elasticsearch/v9 v9.x.x
    require github.com/elastic/go-elasticsearch/v8 v8.x.x

It's possible to use multiple versions of the client in a single project:

    // go.mod
    github.com/elastic/go-elasticsearch/v8 v8.18.0
    github.com/elastic/go-elasticsearch/v9 v9.0.0

    // main.go
    import (
      elasticsearch8 "github.com/elastic/go-elasticsearch/v8"
      elasticsearch9 "github.com/elastic/go-elasticsearch/v9"
    )
    // ...
    es8, _ := elasticsearch8.NewDefaultClient()
    es9, _ := elasticsearch9.NewDefaultClient()

The `main` branch of the client is compatible with the current `master` branch of Elasticsearch.



## Installation

Refer to the [Installation](https://www.elastic.co/docs/reference/elasticsearch/clients/go/installation)
page of the documentation.



## Connecting

Refer to the [Connecting](https://www.elastic.co/docs/reference/elasticsearch/clients/go/connecting)
page of the documentation.

## Operations

- [Creating an index](https://www.elastic.co/docs/reference/elasticsearch/clients/go/using-the-api/crud-operations)
- [Indexing documents](https://www.elastic.co/docs/reference/elasticsearch/clients/go/using-the-api/crud-operations)
- [Getting documents](https://www.elastic.co/docs/reference/elasticsearch/clients/go/using-the-api/crud-operations)
- [Searching documents](https://www.elastic.co/docs/reference/elasticsearch/clients/go/using-the-api/searching)
- [Updating documents](https://www.elastic.co/docs/reference/elasticsearch/clients/go/using-the-api/crud-operations)
- [Deleting documents](https://www.elastic.co/docs/reference/elasticsearch/clients/go/using-the-api/crud-operations)
- [Deleting an index](https://www.elastic.co/docs/reference/elasticsearch/clients/go/using-the-api/crud-operations)


## Helpers

The `esutil` package provides convenience helpers for working with the client. At the moment, it provides the `esutil.JSONReader()` and the [`esutil.BulkIndexer`](https://www.elastic.co/docs/reference/elasticsearch/clients/go/using-the-api/bulk-indexing) helpers.



## Documentation

Find the full reference documentation at [elastic.co](https://www.elastic.co/docs/reference/elasticsearch/clients/go):

- [Getting started](https://www.elastic.co/docs/reference/elasticsearch/clients/go/getting-started)
- [Configuration](https://www.elastic.co/docs/reference/elasticsearch/clients/go/configuration)
- [Using the API](https://www.elastic.co/docs/reference/elasticsearch/clients/go/using-the-api) (low-level and typed)
- [Typed API](https://www.elastic.co/docs/reference/elasticsearch/clients/go/typed-api) and [esdsl builders](https://www.elastic.co/docs/reference/elasticsearch/clients/go/typed-api/esdsl)
- [Advanced topics](https://www.elastic.co/docs/reference/elasticsearch/clients/go/advanced) (interceptors, observability)



## Examples

The **[`_examples`](./_examples)** folder contains a number of recipes and comprehensive examples to get you started with the client,
including configuration and customization of the client, using a custom certificate authority (CA) for security (TLS),
mocking the transport for unit tests, embedding the client in a custom type, building queries, performing requests individually and in bulk, and parsing the responses.



## License

This software is licensed under the [Apache 2 license](./LICENSE). See [NOTICE](./NOTICE).

go-elasticsearch

# effdsl

![GitHub Release](https://img.shields.io/github/v/release/sdqri/effdsl)
[![GoDoc](https://pkg.go.dev/badge/github.com/sdqri/effdsl/v2?status.svg)](https://pkg.go.dev/github.com/sdqri/effdsl/v2?tab=doc)
[![Go Report Card](https://goreportcard.com/badge/github.com/sdqri/effdsl)](https://goreportcard.com/report/github.com/sdqri/effdsl)
![GitHub License](https://img.shields.io/github/license/sdqri/effdsl)
<a href="https://github.com/sdqri/effdsl/pulls" style="text-decoration: none;"><img src="https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat" alt="Contributions welcome"></a>
[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)  

effdsl is a composable Go DSL for building Elasticsearch queries and aggregations with type-safe options and predictable JSON output.

## Key Features

- Type-safe query construction without raw JSON maps
- Functional options API for readable, composable queries
- Broad aggregation coverage (metrics, bucket, pipeline)

For details and examples, see the [documentation](https://sdqri.github.io/effdsl).

## Getting started

### Install

With [Go module](https://github.com/golang/go/wiki/Modules) support, add the import and Go will fetch dependencies automatically:

```go
import "github.com/sdqri/effdsl/v2"
```

Or install directly:

```sh
go get -u github.com/sdqri/effdsl/v2
```

### How to use

Start with `effdsl.Define()` and compose queries with options.

### Examples

**Raw JSON example**

Example match query using a raw JSON string:

```go
import (
    es "github.com/elastic/go-elasticsearch/v8"
)

query := `{
  "query": {
    "match": {
      "message": {
        "query": "Hello World"
      }
    }
  }
}`

res, err := es.Search(
  es.Search.WithBody(strings.NewReader(query)),
)
```

**Using effdsl**

The same query using effdsl:

```go
import (
    es "github.com/elastic/go-elasticsearch/v8"
    
    "github.com/sdqri/effdsl/v2"
    mq "github.com/sdqri/effdsl/v2/queries/matchquery"
)

query, err := effdsl.Define(
    effdsl.WithQuery(
        mq.MatchQuery("message", "Hello World"),
    ),
)

res, err := es.Search(
  es.Search.WithBody(strings.NewReader(query)),
)
```

For more examples and details on query parameters, visit the [documentation](https://sdqri.github.io/effdsl).

## Contributing
Contributions are welcome. Thanks for helping improve effdsl. 🤝 Please see [CONTRIBUTING.md](CONTRIBUTING.md) to get started.

## License
This project is licensed under the MIT License. See [LICENSE.md](LICENSE.md).


effdsl

> **Note**: If you are switching to `OwnLocal/goes` from `belogik/goes` you will want to point whatever
> dependency management system you are using at the `v1.0.0` tag to maintain backward compatibility
> (or change your code to support any API changes which may exist). The `master` branch will contain
> whatever the latest released version is and the `develop` branch will contain the currently
> in-progress version. Either of these branches may contain backwards-incompatible changes from the
> `v1.0.0` tag (which contains the code as it was when the code was forked from `belogik/goes`).

Goes : a library to interact with ElasticSearch
===============================================

Supported operations
--------------------

- index creation
- index removal
- simple indexing (document)
- bulk indexing
- search
- get

Example
-------

You will find examples in example_test.go

License
-------

Copyright (c) 2013 Belogik. All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

   * Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
   * Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
   * Neither the name of Belogik nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


goes

probabilistic data-structures service and storage

<img src="http://i.imgur.com/9z47NdA.png" align="center" height="190" width="600">
<br>

[![Build Status](https://travis-ci.org/skizzehq/skizze.svg?branch=master)](https://travis-ci.org/skizzehq/skizze) [![license](http://img.shields.io/badge/license-Apache-blue.svg)](https://raw.githubusercontent.com/skizzehq/skizze/master/LICENSE)

Skizze ([ˈskɪt͡sə]: german for sketch) is a sketch data store to deal with all problems around counting and sketching using probabilistic data-structures.

Unlike a Key-Value store, Skizze does not store values, but rather appends values to defined sketches, allowing one to solve frequency and cardinality queries in near O(1) time, with minimal memory footprint.

<b> Current status ==> Alpha (tagged v0.0.2) </b>

## Motivation

Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in areas like web analytics and Internet advertising. Analysis of such large data sets often requires powerful distributed data stores like Hadoop and heavy data processing with techniques like MapReduce. This approach often leads to heavyweight high-latency analytical processes and poor applicability to realtime use cases. On the other hand, when one is interested only in simple additive metrics like total page views or average price of conversion, it is obvious that raw data can be efficiently summarized, for example, on a daily basis or using simple in-stream counters.  Computation of more advanced metrics like a number of unique visitor or most frequent items is more challenging and requires a lot of resources if implemented straightforwardly.

Skizze is a (fire and forget) service that provides a probabilistic data structures (sketches) storage that allows estimation of these and many other metrics, with a trade off in precision of the estimations for the memory consumption. These data structures can be used both as temporary data accumulators in query processing procedures and, perhaps more important, as a compact – sometimes astonishingly compact – replacement of raw data in stream-based computing.

## Example use cases (queries)
* How many distinct elements are in the data set (i.e. what is the cardinality of the data set)?
* What are the most frequent elements (the terms “heavy hitters” and “top-k elements” are also used)?
* What are the frequencies of the most frequent elements?
* How many elements belong to the specified range (range query, in SQL it looks like `SELECT count(v) WHERE v >= c1 AND v < c2)`?
* Does the data set contain a particular element (membership query)?

## How to build and run
```
make dist
./bin/skizze
```

## Bindings

Two bindings are currently available:

 * [Go](https://github.com/skizzehq/goskizze) 
   * `go get github.com/skizzehq/goskizze/skizze` [Documentation](https://godoc.org/github.com/skizzehq/goskizze/skizze)
   
 * [Node.js](http://github.com/skizzehq/node-skizze)
   * `npm install --save skizze` [Documentation](https://github.com/skizzehq/node-skizze#documentation)
 

## Example usage:

Skizze comes with a CLI to help test and explore the server. It can be run via

```
./bin/skizze-cli
```

### Commands
**Create** a new Domain (Collection of Sketches):
```{r, engine='bash', count_lines}
#CREATE DOM $name $estCardinality $topk
CREATE DOM demostream 10000000 100
```

**Add** values to the domain:
```{r, engine='bash', count_lines}
#ADD DOM $name $value1, $value2 ....
ADD DOM demostream zod joker grod zod zod grod
```

**Get** the *cardinality* of the domain:
```{r, engine='bash', count_lines}
# GET CARD $name
GET CARD demostream

# returns:
# Cardinality: 9
```

**Get** the *rankings* of the domain:
```{r, engine='bash', count_lines}
# GET RANK $name
GET RANK demostream

# returns:
# Rank: 1	  Value: zod	  Hits: 3
# Rank: 2	  Value: grod	  Hits: 2
# Rank: 3	  Value: joker	  Hits: 1
```

**Get** the *frequencies* of values in the domain:
```{r, engine='bash', count_lines}
# GET FREQ $name $value1 $value2 ...
GET FREQ demostream zod joker batman grod

# returns
# Value: zod	  Hits: 3
# Value: joker	  Hits: 1
# Value: batman	  Hits: 0
# Value: grod	  Hits: 2
```

**Get** the *membership* of values in the domain:
```{r, engine='bash', count_lines}
# GET MEMB $name $value1 $value2 ...
GET MEMB demostream zod joker batman grod

# returns
# Value: zod	  Member: true
# Value: joker	  Member: true
# Value: batman	  Member: false
# Value: grod	  Member: true
```

**List** all available sketches (created by domains):
```{r, engine='bash', count_lines}
LIST

# returns
# Name: demostream  Type: CARD
# Name: demostream  Type: FREQ
# Name: demostream  Type: MEMB
# Name: demostream  Type: RANK
```

**Create** a new sketch of type $type (CARD, MEMB, FREQ or RANK):
```{r, engine='bash', count_lines}
# CREATE CARD $name
CREATE CARD demosketch
```

**Add** values to the sketch of type $type (CARD, MEMB, FREQ or RANK):
```{r, engine='bash', count_lines}
#ADD $type $name $value1, $value2 ....
ADD CARD demostream zod joker grod zod zod grod
```

### License
Skizze is available under the Apache License, Version 2.0.


### Authors
- [Seif Lotfy](https://twitter.com/seiflotfy)
- [Neil Jagdish Patel](https://twitter.com/njpatel)


skizze

ClickHouse SQL client for Go with a `database/sql` compatibility

clickhouse-go

# Zoekt: fast code search

    "Zoekt, en gij zult spinazie eten" - Jan Eertink

    ("seek, and ye shall eat spinach" - My primary school teacher)

Zoekt is a text search engine intended for use with source
code. (Pronunciation: roughly as you would pronounce "zooked" in English)

**Note:** This has been the maintained source for Zoekt since 2017, when it was forked from the
original repository [github.com/google/zoekt](https://github.com/google/zoekt).

## Background

Zoekt supports fast substring and regexp matching on source code, with a rich query language
that includes boolean operators (and, or, not). It can search individual repositories, and search
across many repositories in a large codebase. Zoekt ranks search results using a combination of code-related signals
like whether the match is on a symbol. Because of its general design based on trigram indexing and syntactic
parsing, it works well for a variety of programming languages.

The two main ways to use the project are
* Through individual commands, to index repositories and perform searches through Zoekt's [query language](doc/query_syntax.md)
* Or, through the indexserver and webserver, which support syncing repositories from a code host and searching them through a web UI or API

For more details on Zoekt's design, see the [docs directory](doc/).

## Usage

### Installation

    go get github.com/sourcegraph/zoekt/

**Note**: It is also recommended to install [Universal ctags](https://github.com/universal-ctags/ctags), as symbol
information is a key signal in ranking search results. See [ctags.md](doc/ctags.md) for more information.

### Command-based usage

Zoekt supports indexing and searching repositories on the command line. This is most helpful
for simple local usage, or for testing and development.

#### Indexing a local git repo

    go install github.com/sourcegraph/zoekt/cmd/zoekt-git-index
    $GOPATH/bin/zoekt-git-index -index ~/.zoekt /path/to/repo

#### Indexing a local directory (not git-specific)

    go install github.com/sourcegraph/zoekt/cmd/zoekt-index
    $GOPATH/bin/zoekt-index -index ~/.zoekt /path/to/repo

#### Searching an index

    go install github.com/sourcegraph/zoekt/cmd/zoekt
    $GOPATH/bin/zoekt 'hello'
    $GOPATH/bin/zoekt 'hello file:README'

### Zoekt services

Zoekt also contains an index server and web server to support larger-scale indexing and searching
of remote repositories. The index server can be configured to periodically fetch and reindex repositories
from a code host. The webserver can be configured to serve search results through a web UI or API.

#### Indexing a GitHub organization
    
    go install github.com/sourcegraph/zoekt/cmd/zoekt-indexserver

    echo YOUR_GITHUB_TOKEN_HERE > token.txt
    echo '[{"GitHubOrg": "apache", "CredentialPath": "token.txt"}]' > config.json

    $GOPATH/bin/zoekt-indexserver -mirror_config config.json -data_dir ~/.zoekt/ 

This will fetch all repos under 'github.com/apache', then index the repositories. The indexserver takes care of
periodically fetching and indexing new data, and cleaning up logfiles. See [config.go](cmd/zoekt-indexserver/config.go)
for more details on this configuration.

#### Starting the web server

    go install github.com/sourcegraph/zoekt/cmd/zoekt-webserver
    $GOPATH/bin/zoekt-webserver -index ~/.zoekt/

This will start a web server with a simple search UI at http://localhost:6070.
See the [query syntax docs](doc/query_syntax.md) for more details on the query
language.

If you start the web server with `-rpc`, it exposes a [simple JSON search
API](doc/json-api.md) at `http://localhost:6070/api/search`.

The JSON API supports advanced features including:
- Streaming search results (using the `FlushWallTime` option)
- Alternative BM25 scoring (using the `UseBM25Scoring` option)
- Context lines around matches (using the `NumContextLines` option)

Finally, the web server exposes a gRPC API that supports [structured query objects](query/query.go) and advanced search options.

## Acknowledgements

Thanks to Han-Wen Nienhuys for creating Zoekt. Thanks to Alexander Neubeck for
coming up with this idea, and helping Han-Wen Nienhuys flesh it out.


The Best Go Libraries For Database Drivers - Search and Analytic Databases (9)