Fast, zero-allocation, lexicographical-order-preserving packing of native types to bytes

# bingo

[![Go Report Card](https://goreportcard.com/badge/github.com/iancmcc/bingo?style=flat-square)](https://goreportcard.com/report/github.com/iancmcc/bingo)
[![Go Reference](https://pkg.go.dev/badge/github.com/iancmcc/bingo.svg)](https://pkg.go.dev/github.com/iancmcc/bingo)
![Tests](https://github.com/iancmcc/bingo/actions/workflows/tests.yml/badge.svg)
[![Coverage](https://coveralls.io/repos/github/iancmcc/bingo/badge.svg?branch=main)](https://coveralls.io/github/iancmcc/bingo?branch=main)
[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)

Fast, zero-allocation, lexicographic-order-preserving packing/unpacking of native Go types to bytes.

## Features

* Encode `bool`, `string`, `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, `uint64`, `float32`, `float64`, and `time.Time`
* Packed values maintain original sort order
* Pack values in descending order
* Pack to an existing byte slice with no additional allocations
* Create and pack values to a new byte slice (one allocation)
* Unpack all values or just specific indexes

## Usage

Import `bingo`:

```go
import "github.com/iancmcc/bingo"
```

### Packing

```go
// Create and return a byte slice with encoded values
key := bingo.MustPack(uint8(12), "cool string bro")

// Now unpack
var (
    first uint8
    second string
)
bingo.Unpack(key, &first, &second)


// Pack so results will sort the second value descending
key = bingo.WithDesc(false, true, false).MustPack(1, time.Now(), true)

// Just unpack the middle value
var t time.Time
bingo.UnpackIndex(key, 1, &t)


// Pack to an existing byte slice
existingSlice := make([]byte, 100)
bingo.MustPackTo(existingSlice, uint16(7), "abc123")
```

## Benchmarks

```sh
$ go test -bench BenchmarkCodecs
```

```
goos: linux
goarch: amd64
pkg: github.com/iancmcc/bingo
cpu: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
BenchmarkCodecs/int8/encode/natural-8         	100000000	        10.35 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int8/encode/inverse-8         	89794872	        12.60 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int8/decode/natural-8         	65864187	        17.11 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int8/decode/inverse-8         	63386660	        17.27 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int16/encode/natural-8        	100000000	        10.04 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int16/encode/inverse-8        	85618111	        13.08 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int16/decode/natural-8        	64825372	        16.55 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int16/decode/inverse-8        	58359490	        19.44 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int32/encode/natural-8        	120393369	         9.942 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int32/encode/inverse-8        	80418766	        13.67 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int32/decode/natural-8        	53423986	        21.03 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int32/decode/inverse-8        	47975257	        24.68 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int64/encode/natural-8        	100000000	        10.95 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int64/encode/inverse-8        	67093402	        17.79 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int64/decode/natural-8        	47149765	        23.97 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/int64/decode/inverse-8        	39063907	        27.36 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/float32/encode/natural-8      	100000000	        10.42 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/float32/encode/inverse-8      	79910834	        14.11 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/float32/decode/natural-8      	63494347	        16.88 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/float32/decode/inverse-8      	46677241	        25.55 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/float64/encode/natural-8      	100000000	        11.05 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/float64/encode/inverse-8      	62377978	        17.48 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/float64/decode/natural-8      	62370994	        17.26 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/float64/decode/inverse-8      	54139144	        20.68 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/string/encode/natural-8       	47404435	        23.80 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/string/encode/inverse-8       	29144544	        39.74 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/string/decode/natural-8       	39683684	        31.39 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/string/decode/inverse-8       	23373333	        48.06 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/time/encode/natural-8         	47027752	        24.72 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/time/encode/inverse-8         	38363001	        29.22 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/time/decode/natural-8         	42566337	        26.76 ns/op	       0 B/op	       0 allocs/op
BenchmarkCodecs/time/decode/inverse-8         	30851250	        36.88 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/iancmcc/bingo	37.644s
```


bingo

Go package implementing buffers for handling various datatypes easily

<h1 align="center"><img height="340" src="https://github.com/superwhiskers/crunch/raw/canon/.github/cookie.png" alt="cookie with a bite taken out of it"/><br />crunch</h1>

<p align="center">
	<b>a library for easily manipulating bits and bytes in golang</b>
</p>

<p align="center">
	<a href="https://github.com/superwhiskers/crunch/blob/canon/LICENSE.txt">
		<img src="https://img.shields.io/badge/license-MPL--2.0-brightgreen" alt="license" />
	</a>
  <a href="https://pkg.go.dev/github.com/superwhiskers/crunch/v3">
    <img src="https://pkg.go.dev/badge/github.com/superwhiskers/crunch/v3.svg" alt="documentation" />
  </a>
	<a href="https://github.com/superwhiskers/crunch/actions/workflows/go.yml">
		<img src="https://github.com/superwhiskers/crunch/actions/workflows/go.yml/badge.svg" alt="go" />
	</a>
	<a href="https://goreportcard.com/report/github.com/superwhiskers/crunch">
		<img src="https://goreportcard.com/badge/github.com/superwhiskers/crunch" alt="go report card" />
	</a>
	<a href="https://repl.it/github/https://github.com/superwhiskers/crunch?ref=button">
		<img src="https://img.shields.io/badge/try%20it%20on-repl.it-%2359646A.svg" alt="try it on repl.it" />
	</a>
</p>

<p align="center">
	<a href="#features">features</a> | <a href="#installation">installation</a> | <a href="#benchmarks">benchmarks</a> | <a href="#examples">examples</a>
</p>

## features

- **feature-rich**: supports reading and writing integers of varying sizes in both little and big endian
- **performant**: performs more than twice as fast as the standard library's `bytes.Buffer`
- **simple and familiar**: has a consistent and easy-to-use api
- **licensed under the mpl-2.0**: use it anywhere you wish, just don't change it privately

## installation

### install with the `go` tool

```bash
$ go get github.com/superwhiskers/crunch/v3
```

then, just import it in your project like this. easy!

```go
package "yourpackage"

import crunch "github.com/superwhiskers/crunch/v3"
```

### install using git submodules (not recommended)

```bash
# this assumes that you are in a git repository
$ git submodule add https://github.com/superwhiskers/crunch path/to/where/you/want/crunch
```

then, you can import it like this

```go
package "yourpackage"

import crunch "github.com/your-username/project/path/to/crunch/v3"
```

## benchmarks

both `Buffer` and `MiniBuffer` perform on average more than twice as fast as `bytes.Buffer` in both writing and reading
```
goos: linux
goarch: amd64
pkg: github.com/superwhiskers/crunch/v3
cpu: Intel(R) Core(TM) i5-4300M CPU @ 2.60GHz
BenchmarkBufferWriteBytes-4       	612593820	         1.948 ns/op	       0 B/op	       0 allocs/op
BenchmarkBufferReadBytes-4        	1000000000	         0.5476 ns/op	       0 B/op	       0 allocs/op
BenchmarkBufferWriteU32LE-4       	125229171	         9.528 ns/op	       0 B/op	       0 allocs/op
BenchmarkBufferReadU32LE-4        	44677784	        24.73 ns/op	       8 B/op	       1 allocs/op
BenchmarkBufferReadBit-4          	1000000000	         0.5709 ns/op	       0 B/op	       0 allocs/op
BenchmarkBufferReadBits-4         	620840577	         1.869 ns/op	       0 B/op	       0 allocs/op
BenchmarkBufferSetBit-4           	600202990	         1.929 ns/op	       0 B/op	       0 allocs/op
BenchmarkBufferClearBit-4         	625814206	         1.993 ns/op	       0 B/op	       0 allocs/op
BenchmarkBufferGrow-4             	252735192	         6.638 ns/op	       0 B/op	       0 allocs/op
BenchmarkMiniBufferWriteBytes-4   	577940112	         2.112 ns/op	       0 B/op	       0 allocs/op
BenchmarkMiniBufferReadBytes-4    	1000000000	         0.5531 ns/op	       0 B/op	       0 allocs/op
BenchmarkMiniBufferWriteU32LE-4   	116178949	        12.90 ns/op	       0 B/op	       0 allocs/op
BenchmarkMiniBufferReadU32LE-4    	189681555	         6.296 ns/op	       0 B/op	       0 allocs/op
BenchmarkMiniBufferWriteF32LE-4   	121033429	        13.06 ns/op	       0 B/op	       0 allocs/op
BenchmarkMiniBufferReadF32LE-4    	170977377	         7.244 ns/op	       0 B/op	       0 allocs/op
BenchmarkMiniBufferReadBit-4      	1000000000	         0.4730 ns/op	       0 B/op	       0 allocs/op
BenchmarkMiniBufferReadBits-4     	249350655	         4.968 ns/op	       0 B/op	       0 allocs/op
BenchmarkMiniBufferSetBit-4       	566985802	         2.173 ns/op	       0 B/op	       0 allocs/op
BenchmarkMiniBufferClearBit-4     	531134203	         2.195 ns/op	       0 B/op	       0 allocs/op
BenchmarkMiniBufferGrow-4         	271458589	         4.434 ns/op	       0 B/op	       0 allocs/op
BenchmarkStdByteBufferWrite-4     	127322588	         9.249 ns/op	       0 B/op	       0 allocs/op
BenchmarkStdByteBufferRead-4      	319744824	         3.742 ns/op	       0 B/op	       0 allocs/op
BenchmarkStdByteBufferGrow-4      	274847331	         4.367 ns/op	       0 B/op	       0 allocs/op
```

## examples

examples can be found in the [examples](https://github.com/superwhiskers/crunch/blob/canon/examples) directory

## acknowledgements

icon (cookie logo) made by [freepik](https://www.freepik.com/) from [flaticon.com](https://www.flaticon.com)


crunch

Go package implementing compressed bitsets

# roaring 

[![GoDoc](https://godoc.org/github.com/RoaringBitmap/roaring?status.svg)](https://godoc.org/github.com/RoaringBitmap/roaring) [![Go Report Card](https://goreportcard.com/badge/RoaringBitmap/roaring)](https://goreportcard.com/report/github.com/RoaringBitmap/roaring)

![Go-CI](https://github.com/RoaringBitmap/roaring/workflows/Go-CI/badge.svg)
![Go-ARM-CI](https://github.com/RoaringBitmap/roaring/workflows/Go-ARM-CI/badge.svg)
![Go-Windows-CI](https://github.com/RoaringBitmap/roaring/workflows/Go-Windows-CI/badge.svg)
=============

This is a go version of the Roaring bitmap data structure. 

Roaring bitmaps are used by several major systems such as [Apache Lucene][lucene] and derivative systems such as [Solr][solr] and
[Elasticsearch][elasticsearch], [Apache Druid (Incubating)][druid], [LinkedIn Pinot][pinot], [Netflix Atlas][atlas],  [Apache Spark][spark], [OpenSearchServer][opensearchserver], [anacrolix/torrent][anacrolix/torrent], [Whoosh][whoosh], [Redpanda](https://github.com/redpanda-data/redpanda), [Pilosa][pilosa],  [Microsoft Visual Studio Team Services (VSTS)][vsts], and eBay's [Apache Kylin][kylin]. The YouTube SQL Engine, [Google Procella](https://research.google/pubs/pub48388/), uses Roaring bitmaps for indexing.

[lucene]: https://lucene.apache.org/
[solr]: https://lucene.apache.org/solr/
[elasticsearch]: https://www.elastic.co/products/elasticsearch
[druid]: https://druid.apache.org/
[spark]: https://spark.apache.org/
[opensearchserver]: http://www.opensearchserver.com
[anacrolix/torrent]: https://github.com/anacrolix/torrent
[whoosh]: https://bitbucket.org/mchaput/whoosh/wiki/Home
[pilosa]: https://www.pilosa.com/
[kylin]: http://kylin.apache.org/
[pinot]: http://github.com/linkedin/pinot/wiki
[vsts]: https://www.visualstudio.com/team-services/
[atlas]: https://github.com/Netflix/atlas
[quanta]: https://github.com/disney/quanta

Roaring bitmaps are found to work well in many important applications:

> Use Roaring for bitmap compression whenever possible. Do not use other bitmap compression methods ([Wang et al., SIGMOD 2017](http://db.ucsd.edu/wp-content/uploads/2017/03/sidm338-wangA.pdf))


The ``roaring`` Go library is used by
* [anacrolix/torrent]
* [InfluxDB](https://www.influxdata.com)
* [Pilosa](https://www.pilosa.com/)
* [Bleve](http://www.blevesearch.com)
* [Weaviate](https://github.com/weaviate/weaviate)
* [lindb](https://github.com/lindb/lindb)
* [Elasticell](https://github.com/deepfabric/elasticell)
* [SourceGraph](https://github.com/sourcegraph/sourcegraph)
* [M3](https://github.com/m3db/m3)
* [trident](https://github.com/NetApp/trident)
* [Husky](https://www.datadoghq.com/blog/engineering/introducing-husky/)
* [FrostDB](https://github.com/polarsignals/frostdb)
* [Disney Quanta](https://github.com/disney/quanta)




This library is used in production in several systems, it is part of the [Awesome Go collection](https://awesome-go.com).


There are also  [Java](https://github.com/RoaringBitmap/RoaringBitmap) and [C/C++](https://github.com/RoaringBitmap/CRoaring) versions.  The Java, C, C++ and Go version are binary compatible: e.g,  you can save bitmaps
from a Java program and load them back in Go, and vice versa. We have a [format specification](https://github.com/RoaringBitmap/RoaringFormatSpec).


This code is licensed under Apache License, Version 2.0 (ASL2.0).

Copyright 2016-... by the authors.

When should you use a bitmap?
===================================


Sets are a fundamental abstraction in
software. They can be implemented in various
ways, as hash sets, as trees, and so forth.
In databases and search engines, sets are often an integral
part of indexes. For example, we may need to maintain a set
of all documents or rows  (represented by numerical identifier)
that satisfy some property. Besides adding or removing
elements from the set, we need fast functions
to compute the intersection, the union, the difference between sets, and so on.


To implement a set
of integers, a particularly appealing strategy is the
bitmap (also called bitset or bit vector). Using n bits,
we can represent any set made of the integers from the range
[0,n): the ith bit is set to one if integer i is present in the set.
Commodity processors use words of W=32 or W=64 bits. By combining many such words, we can
support large values of n. Intersections, unions and differences can then be implemented
 as bitwise AND, OR and ANDNOT operations.
More complicated set functions can also be implemented as bitwise operations.

When the bitset approach is applicable, it can be orders of
magnitude faster than other possible implementation of a set (e.g., as a hash set)
while using several times less memory.

However, a bitset, even a compressed one is not always applicable. For example, if
you have 1000 random-looking integers, then a simple array might be the best representation.
We refer to this case as the "sparse" scenario.

When should you use compressed bitmaps?
===================================

An uncompressed BitSet can use a lot of memory. For example, if you take a BitSet
and set the bit at position 1,000,000 to true and you have just over 100kB. That is over 100kB
to store the position of one bit. This is wasteful  even if you do not care about memory:
suppose that you need to compute the intersection between this BitSet and another one
that has a bit at position 1,000,001 to true, then you need to go through all these zeroes,
whether you like it or not. That can become very wasteful.

This being said, there are definitively cases where attempting to use compressed bitmaps is wasteful.
For example, if you have a small universe size. E.g., your bitmaps represent sets of integers
from [0,n) where n is small (e.g., n=64 or n=128). If you can use uncompressed BitSet and
it does not blow up your memory usage,  then compressed bitmaps are probably not useful
to you. In fact, if you do not need compression, then a BitSet offers remarkable speed.

The sparse scenario is another use case where compressed bitmaps should not be used.
Keep in mind that random-looking data is usually not compressible. E.g., if you have a small set of
32-bit random integers, it is not mathematically possible to use far less than 32 bits per integer,
and attempts at compression can be counterproductive.

How does Roaring compares with the alternatives?
==================================================


Most alternatives to Roaring are part of a larger family of compressed bitmaps that are run-length-encoded
bitmaps. They identify long runs of 1s or 0s and they represent them with a marker word.
If you have a local mix of 1s and 0, you use an uncompressed word.

There are many formats in this family:

* Oracle's BBC is an obsolete format at this point: though it may provide good compression,
it is likely much slower than more recent alternatives due to excessive branching.
* WAH is a patented variation on BBC that provides better performance.
* Concise is a variation on the patented WAH. It some specific instances, it can compress
much better than WAH (up to 2x better), but it is generally slower.
* EWAH is both free of patent, and it is faster than all the above. On the downside, it
does not compress quite as well. It is faster because it allows some form of "skipping"
over uncompressed words. So though none of these formats are great at random access, EWAH
is better than the alternatives.



There is a big problem with these formats however that can hurt you badly in some cases: there is no random access. If you want to check whether a given value is present in the set, you have to start from the beginning and "uncompress" the whole thing. This means that if you want to intersect a big set with a large set, you still have to uncompress the whole big set in the worst case...

Roaring solves this problem. It works in the following manner. It divides the data into chunks of 2<sup>16</sup> integers
(e.g., [0, 2<sup>16</sup>), [2<sup>16</sup>, 2 x 2<sup>16</sup>), ...). Within a chunk, it can use an uncompressed bitmap, a simple list of integers,
or a list of runs. Whatever format it uses, they all allow you to check for the presence of any one value quickly
(e.g., with a binary search). The net result is that Roaring can compute many operations much faster than run-length-encoded
formats like WAH, EWAH, Concise... Maybe surprisingly, Roaring also generally offers better compression ratios.





### References

- Daniel Lemire, Owen Kaser, Nathan Kurz, Luca Deri, Chris O'Hara, François Saint-Jacques, Gregory Ssi-Yan-Kai, Roaring Bitmaps: Implementation of an Optimized Software Library, Software: Practice and Experience 48 (4), 2018 [arXiv:1709.07821](https://arxiv.org/abs/1709.07821)
-  Samy Chambi, Daniel Lemire, Owen Kaser, Robert Godin,
Better bitmap performance with Roaring bitmaps,
Software: Practice and Experience 46 (5), 2016.[arXiv:1402.6407](http://arxiv.org/abs/1402.6407) This paper used data from http://lemire.me/data/realroaring2014.html
- Daniel Lemire, Gregory Ssi-Yan-Kai, Owen Kaser, Consistently faster and smaller compressed bitmaps with Roaring, Software: Practice and Experience 46 (11), 2016. [arXiv:1603.06549](http://arxiv.org/abs/1603.06549)

### Dependencies

Dependencies are fetched automatically by giving the `-t` flag to `go get`.

they include
  - github.com/bits-and-blooms/bitset
  - github.com/mschoch/smat
  - github.com/glycerine/go-unsnap-stream
  - github.com/philhofer/fwd
  - github.com/jtolds/gls

Note that the smat library requires Go 1.15 or better.

#### Installation

  - go get -t github.com/RoaringBitmap/roaring

### Instructions for contributors

Using bash or other common shells:
```
$ git clone git@github.com:RoaringBitmap/roaring.git
$ export GO111MODULE=on 
$ go mod tidy
$ go test -v
```

### Example

Here is a simplified but complete example:

```go
package main

import (
    "fmt"
    "github.com/RoaringBitmap/roaring/v2"
    "bytes"
)


func main() {
    // example inspired by https://github.com/fzandona/goroar
    fmt.Println("==roaring==")
    rb1 := roaring.BitmapOf(1, 2, 3, 4, 5, 100, 1000)
    fmt.Println(rb1.String())

    rb2 := roaring.BitmapOf(3, 4, 1000)
    fmt.Println(rb2.String())

    rb3 := roaring.New()
    fmt.Println(rb3.String())

    fmt.Println("Cardinality: ", rb1.GetCardinality())

    fmt.Println("Contains 3? ", rb1.Contains(3))

    rb1.And(rb2)

    rb3.Add(1)
    rb3.Add(5)

    rb3.Or(rb1)

    // computes union of the three bitmaps in parallel using 4 workers  
    roaring.ParOr(4, rb1, rb2, rb3)
    // computes intersection of the three bitmaps in parallel using 4 workers  
    roaring.ParAnd(4, rb1, rb2, rb3)


    // prints 1, 3, 4, 5, 1000
    i := rb3.Iterator()
    for i.HasNext() {
        fmt.Println(i.Next())
    }
    fmt.Println()

    // next we include an example of serialization
    buf := new(bytes.Buffer)
    rb1.WriteTo(buf) // we omit error handling
    newrb:= roaring.New()
    newrb.ReadFrom(buf)
    if rb1.Equals(newrb) {
    	fmt.Println("I wrote the content to a byte stream and read it back.")
    }
    // you can iterate over bitmaps using ReverseIterator(), Iterator, ManyIterator()
}
```

If you wish to use serialization and handle errors, you might want to
consider the following sample of code:

```go
	rb := BitmapOf(1, 2, 3, 4, 5, 100, 1000)
	buf := new(bytes.Buffer)
	size,err:=rb.WriteTo(buf)
	if err != nil {
		fmt.Println("Failed writing") // return or panic
	}
	newrb:= New()
	size,err=newrb.ReadFrom(buf)
	if err != nil {
		fmt.Println("Failed reading") // return or panic
	}
	// if buf is an untrusted source, you should validate the result
	// (this adds a bit of complexity but it is necessary for security)
	if newrb.Validate() != nil {
		fmt.Println("Failed validation") // return or panic
	}
	if ! rb.Equals(newrb) {
		fmt.Println("Cannot retrieve serialized version")
	}
```

Given N integers in [0,x), then the serialized size in bytes of
a Roaring bitmap should never exceed this bound:

`` 8 + 9 * ((long)x+65535)/65536 + 2 * N ``

That is, given a fixed overhead for the universe size (x), Roaring
bitmaps never use more than 2 bytes per integer. You can call
``BoundSerializedSizeInBytes`` for a more precise estimate.

### 64-bit Roaring

By default, roaring is used to stored unsigned 32-bit integers. However, we also offer
an extension dedicated to 64-bit integers. It supports roughly the same functions:

```go
package main

import (
    "fmt"
    "github.com/RoaringBitmap/roaring/v2/roaring64"
    "bytes"
)


func main() {
    // example inspired by https://github.com/fzandona/goroar
    fmt.Println("==roaring64==")
    rb1 := roaring64.BitmapOf(1, 2, 3, 4, 5, 100, 1000)
    fmt.Println(rb1.String())

    rb2 := roaring64.BitmapOf(3, 4, 1000)
    fmt.Println(rb2.String())

    rb3 := roaring64.New()
    fmt.Println(rb3.String())

    fmt.Println("Cardinality: ", rb1.GetCardinality())

    fmt.Println("Contains 3? ", rb1.Contains(3))

    rb1.And(rb2)

    rb3.Add(1)
    rb3.Add(5)

    rb3.Or(rb1)



    // prints 1, 3, 4, 5, 1000
    i := rb3.Iterator()
    for i.HasNext() {
        fmt.Println(i.Next())
    }
    fmt.Println()

    // next we include an example of serialization
    buf := new(bytes.Buffer)
    rb1.WriteTo(buf) // we omit error handling
    newrb:= roaring64.New()
    newrb.ReadFrom(buf)
    if rb1.Equals(newrb) {
    	fmt.Println("I wrote the content to a byte stream and read it back.")
    }
    // you can iterate over bitmaps using ReverseIterator(), Iterator, ManyIterator()
}
```

Only the 32-bit roaring format is standard and cross-operable between Java, C++, C and Go. There is no guarantee that the 64-bit versions are compatible.

### Documentation

Current documentation is available at https://pkg.go.dev/github.com/RoaringBitmap/roaring and https://pkg.go.dev/github.com/RoaringBitmap/roaring/roaring64

### Goroutine safety

In general, it should not generally be considered safe to access
the same bitmaps using different goroutines--they are left
unsynchronized for performance. Should you want to access
a Bitmap from more than one goroutine, you should
provide synchronization. Typically this is done by using channels to pass
the *Bitmap around (in Go style; so there is only ever one owner),
or by using `sync.Mutex` to serialize operations on Bitmaps.

### Coverage

We test our software. For a report on our test coverage, see

https://coveralls.io/github/RoaringBitmap/roaring?branch=master

### Benchmark

Type

         go test -bench Benchmark -run -

To run benchmarks on [Real Roaring Datasets](https://github.com/RoaringBitmap/real-roaring-datasets)
run the following:

```sh
go get github.com/RoaringBitmap/real-roaring-datasets
BENCH_REAL_DATA=1 go test -bench BenchmarkRealData -run -
```

### Interactive use

You can use roaring with gore:

- go install github.com/x-motemen/gore/cmd/gore@latest
- Make sure that ``$GOPATH/bin`` is in your ``$PATH``.

```go
$ gore
gore version 0.2.6  :help for help
gore> :import github.com/RoaringBitmap/roaring
gore> x:=roaring.New()
gore> x.Add(1)
gore> x.String()
"{1}"
```


### Fuzzy testing

You can help us test further the library with fuzzy testing:

         go get github.com/dvyukov/go-fuzz/go-fuzz
         go get github.com/dvyukov/go-fuzz/go-fuzz-build
         go test -tags=gofuzz -run=TestGenerateSmatCorpus
         go-fuzz-build github.com/RoaringBitmap/roaring
         go-fuzz -bin=./roaring-fuzz.zip -workdir=workdir/ -timeout=200 -func FuzzSmat

Let it run, and if the # of crashers is > 0, check out the reports in
the workdir where you should be able to find the panic goroutine stack
traces.

You may also replace `-func FuzzSmat`  by `-func FuzzSerializationBuffer` or `-func FuzzSerializationStream`.

### Alternative in Go

There is a Go version wrapping the C/C++ implementation https://github.com/RoaringBitmap/gocroaring

For an alternative implementation in Go, see https://github.com/fzandona/goroar
The two versions were written independently.


### Mailing list/discussion group

https://groups.google.com/g/roaring-bitmaps

## Stars


[![Star History Chart](https://api.star-history.com/svg?repos=RoaringBitmap/roaring&type=Date)](https://www.star-history.com/#RoaringBitmap/roaring&Date)

### Further reading

<p>Mastering Programming: From Testing to Performance in Go</p>
<div><a href="https://www.amazon.com/dp/B0FMPGSWR5"><img style="margin-left: auto; margin-right: auto;" src="https://m.media-amazon.com/images/I/61feneHS7kL._SL1499_.jpg" alt="" width="250px" /></a></div>


roaring

Binary packer and unpacker helps user build custom binary stream

# binpacker [![Build Status](https://travis-ci.org/zhuangsirui/binpacker.svg?branch=master)](https://travis-ci.org/zhuangsirui/binpacker) [![GoDoc](https://godoc.org/github.com/zhuangsirui/binpacker?status.svg)](https://godoc.org/github.com/zhuangsirui/binpacker) [![Go Report Card](https://goreportcard.com/badge/github.com/zhuangsirui/binpacker)](https://goreportcard.com/report/github.com/zhuangsirui/binpacker)
A binary packer and unpacker.

# Install

```bash
go get github.com/zhuangsirui/binpacker
```

# Examples

## Packer

```go
buffer := new(bytes.Buffer)
packer := binpacker.NewPacker(binary.BigEndian, buffer)
packer.PushByte(0x01)
packer.PushBytes([]byte{0x02, 0x03})
packer.PushUint16(math.MaxUint16)
```

```go
// You can push data like this
buffer := new(bytes.Buffer)
packer := binpacker.NewPacker(binary.BigEndian, buffer)
packer.PushByte(0x01).PushBytes([]byte{0x02, 0x03}).PushUint16(math.MaxUint16)
packer.Error() // Make sure error is nil
```

## Unpacker

**Example data**

```go
buffer := new(bytes.Buffer)
packer := binpacker.NewPacker(binary.BigEndian, buffer)
unpacker := binpacker.NewUnpacker(binary.BigEndian, buffer)
packer.PushByte(0x01)
packer.PushUint16(math.MaxUint16)
```

```go
var val1 byte
var val2 uint16
var err error
val1, err = unpacker.ShiftByte()
val2, err = unpacker.ShiftUint16()
```

```go
var val1 byte
var val2 uint16
var err error
unpacker.FetchByte(&val1).FetchUint16(&val2)
unpacker.Error() // Make sure error is nil
```


binpacker

Golang set data structure with bonus bit-twiddling functions

# Your basic bit [![GoDoc](https://godoc.org/github.com/yourbasic/bit?status.svg)][godoc-bit]

### Set data structure for positive numbers

A bit array, or bit set, is an efficient set data structure.
It consists of an array that compactly stores bits and it uses
bit-level parallelism to perform operations quickly.

![Venn diagram](venn.png)

### Installation

Once you have [installed Go][golang-install], run this command
to install the `bit` package:

    go get github.com/yourbasic/bit
    
### Documentation

There is an online reference for the package at
[godoc.org/github.com/yourbasic/bit][godoc-bit].

### Roadmap

* The API of this library is frozen.
* Version numbers adhere to [semantic versioning][sv].

The only accepted reason to modify the API of this package is to
handle issues that can't be resolved in any other reasonable way.

Stefan Nilsson – [korthaj](https://github.com/korthaj)

[godoc-bit]: https://godoc.org/github.com/yourbasic/bit
[golang-install]: http://golang.org/doc/install.html
[sv]: http://semver.org/


A Go implementation of the Elias-Fano encoding

# go-ef 
_A Go implementation of the Elias-Fano encoding_

[![Build Status](https://travis-ci.org/amallia/go-ef.svg?branch=master)](https://travis-ci.org/amallia/go-ef) [![GoDoc](https://godoc.org/github.com/amallia/go-ef?status.svg)](https://godoc.org/github.com/amallia/go-ef) [![Go Report Card](https://goreportcard.com/badge/github.com/amallia/go-ef)](https://goreportcard.com/report/github.com/amallia/go-ef) ![cover.run go](https://cover.run/go/github.com/amallia/go-ef.svg)
### Example
```go
package main
import (
    "fmt"
    "github.com/amallia/go-ef"
    "os"
)

func main() {
    array := []uint64{1,5,10}
    size := len(array)
    max := array[size-1]
    obj := ef.New(max, size)

    obj.Compress(array)

    v, err := obj.Next()
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }
    fmt.Println(v) // 1

    obj.Next()
    fmt.Println(obj.Value()) // 5
}
```


The Best Go Libraries For Data Structures and Algorithms - Bit-packing and Compression (6)

bingo

crunch

roaring

binpacker

bit

go-ef