Framework to compare the performance between different Go versions.

autobench
=========

`autobench` is a framework to compare the performance of Go 1.2 and Go 1.3.

usage
-----

`autobench` downloads and builds the latest Go 1.1 and Go tip branches and runs a set of Go 1 benchmarks for comparison.

Useful targets are
	
    make bench		# runs all benchmarks, _once_
    make go1 		# runs bench/go1 benchmarks _once_
    make runtime 	# runs bench/runtime benchmarks _once_
    make http	 	# runs bench/http benchmarks _once_
    make float	 	# runs bench/float benchmarks _once_
    make extra		# runs extra benchmarks. 
    make clean 		# removes any previous benchmark results
    make update		# updates both branches to the latest revision, clears any benchmark results

You can optionally benchmark with gccgo instead of gc by either uncommenting the corresponding line in the Makefile or by setting TESTFLAGS to an appropriate value:

    make TESTFLAGS=-compiler=gccgo bench

known issues
------------

 * If you are benchmarking on Freebsd, you may need to use `gmake`.

contributing
------------

Contributions and pull requests are always welcome. If you are submitting a pull request with benchmark data, please include the value of OLD and NEW at the top of the Makefile in the suffix of your file (follow the examples) so we can trace which revision this benchmark was taken from. If you want to include commentry in your benchmark, comments should start with a #.

     make > linux-386-go1.2.1-vs-go.1.3beta1.txt

licence
-------

This package uses benchmark code from the Go project. Where otherwise unspecified this code is released into the public domain.


autobench

Powerful HTTP-benchmark tool mixed with Аb, Wrk, Siege tools. Gathering statistics and various parameters for benchmarks and comparison results.

# Go Benchmark App 
[![Build Status](https://travis-ci.org/mrLSD/go-benchmark-app.svg?branch=master)](https://travis-ci.org/mrLSD/go-benchmark-app)   [![License](http://img.shields.io/badge/license-mit-blue.svg?style=flat-square)](https://raw.githubusercontent.com/mrLSD/go-benchmark-app/master/LICENSE)  [![codecov](https://codecov.io/gh/mrLSD/go-benchmark-app/branch/master/graph/badge.svg)](https://codecov.io/gh/mrLSD/go-benchmark-app)  [![Coverage Status](https://coveralls.io/repos/github/mrLSD/go-benchmark-app/badge.svg?branch=master)](https://coveralls.io/github/mrLSD/go-benchmark-app?branch=master)  [![Go Report Card](https://goreportcard.com/badge/github.com/mrLSD/go-benchmark-app)](https://goreportcard.com/report/github.com/mrLSD/go-benchmark-app)  [![GoDoc](https://godoc.org/github.com/mrLSD/go-benchmark-app?status.png)](https://godoc.org/github.com/mrLSD/go-benchmark-app)

_The efficiency and speed of application - our goal and the basic idea._

![Go Benchmark App ](http://letzgro.net/wp-content/uploads/2016/01/banners-4.png)

Application for HTTP-benchmarking via different rules and configurations.

Configurations provided via TOML-config.

Our main aims - compare different web-applications 
that we want to compare with same benchmarks tools, 
same parameters, repeated and distributed in time.
Get average results and compare it with other applications.

This will allow us to analyze and compare the performance 
bottlenecks, and to take appropriate measures.
The efficiency and speed of application - our goal and the 
basic idea.

## Benchmarks tools 
`ab`, `wrk`, `siege`

It should be installed.

## How to configure
All config at `config/main.toml`

````
# Title for describing benchmarks
title = "CMS benchmarks"
# Benchmarks version
version = "0.1"
# Delay via try in seconds
delay = 20
# How much we should try
try = 10

# ab benchmarks parametres
[ab]
concurency = 5000
keepalive = false
requests = 10000

# wrk benchmarks parametres
[wrk]
connections = 5000
duration = 10
threads = 1000

# siege benchmarks parametres
[siege]
concurrent = 100
time = 30

# Applications parametres - list
[[app]]
title = "Application Banchamrs Title"
path = "fool/path/to/app"
url = "http://localhost:5000/test"
````

## How to use
* Installed `Go 1.6+`
* Check is benchmarks tools installed:

	`$ whereis ab`
	
	`$ whereis wrk`
	
	`$ whereis siege` 
* Configure `config/main.toml` in current dirrectory as mentioned before.
* Install application: `$ go install github.com/mrlsd/go-benchmark-app`
* Command-line help: `$ go-benchmark-app -h`
```
Go Benchmark Applications v1.0.0
Options:
  -c FILE
    	load configuration from FILE (default "config/main.toml")
  -v	verbose output
```
* To change config file - run: `$ go-benchmark-app -c path/to/cfg.toml`
* Verbose output: `$ go-benchmark-app -v`

## Tips & Tricks
* When your benchmarks test failed with socket error 
(more resources unawailable), try increase `delay` options
 at config file (for example 60 sec) or/and change system 
 limits to opened files, TCP/IP configuration and other 
 system limitations.
* For flexibility running test applications you can create
recipe for that. For example set at config file:
```
[[app]]
title = "My App"
path = "apps/myapp.sh"
url = "http://localhost:5000/test"
```

and `apps/myapp.sh` file:
```
#!/bin/sh
prepare -to -run
/full/path/to/app -v param -d param -etc
```
in that way you can run `docker`, `nginx` or another
useful commands.

For one application we use one `URL`, because it's simplify
results analyze, interpretation, comparison.

#### License: MIT [![License](http://img.shields.io/badge/license-mit-blue.svg?style=flat-square)](https://raw.githubusercontent.com/mrLSD/go-benchmark-app/master/LICENSE)


go-benchmark-app

Few miscellaneous Go microbenchmarks. Compare some language features to alternative approaches.

# go-benchmarks

Benchmarks used in [So You Wanna Go Fast?](http://bravenewgeek.com/so-you-wanna-go-fast/).


go-benchmarks

Go HTTP request router benchmark and comparison.

go-http-routing-benchmark

[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)

# Introduction

This is a benchmark for the json packages.
You are welcome to open an issue if you find anything wrong or out of date.

# TL;DR

In conclusion, the `github.com/json-iterator/go` is your first choice,
it provides you with lots of useful features, oo design and high performance.

# How to run

```shell
go test -v -bench ./...
```

# Example result

```shell
GOGC=off go test -bench=. -benchmem
```

The result may vary according to the different environment.
Use the result only as a reference.

| | | | ns/op | B/op | allocs/op |
| --- | --- | --- | ----- | ---- | --------- |
| 10Fields_Iterator_Object | jsoniter-readObjCB-8 | 1458022 | 810 | 144 | 14 |
| 10Fields_Iterator_Object | jsoniter-readObj-8 | 1369156 | 813 | 144 | 14 |
| 10Fields_Iterator_Object | jsonparser-8 | 1357131 | 908 | 80 | 4 |
| 10Fields_Iterator_Object | jzon-readObj-8 | 1239000 | 955 | 144 | 14 |
| 10Fields_Iterator_Object | jzon-readObjCB-8 | 1204593 | 991 | 144 | 14 |
| | | | | | |
| 10Fields_Unmarshal_Interface | djson-8 | 440643 | 2576 | 1174 | 27 |
| 10Fields_Unmarshal_Interface | easyjson-8 | 422460 | 2894 | 1174 | 27 |
| 10Fields_Unmarshal_Interface | jzon-8 | 414544 | 2904 | 1190 | 28 |
| 10Fields_Unmarshal_Interface | jzon-fast-8 | 425950 | 2917 | 1190 | 28 |
| 10Fields_Unmarshal_Interface | ujson-8 | 337881 | 3494 | 1494 | 38 |
| 10Fields_Unmarshal_Interface | jsoniter-compatible-8 | 320720 | 3758 | 1350 | 38 |
| 10Fields_Unmarshal_Interface | jsoniter-8 | 347126 | 4216 | 1350 | 38 |
| 10Fields_Unmarshal_Interface | json-8 | 229218 | 5532 | 1414 | 36 |
| 10Fields_Unmarshal_Interface | ugorji-8 | 212991 | 5608 | 2478 | 36 |
| | | | | | |
| 10Fields_Unmarshal_StructWithTag | jsoniter-8 | 1000000 | 1028 | 192 | 5 |
| 10Fields_Unmarshal_StructWithTag | jsoniter-compatible-8 | 1223282 | 1034 | 192 | 5 |
| 10Fields_Unmarshal_StructWithTag | jzon-8 | 1000000 | 1106 | 192 | 5 |
| 10Fields_Unmarshal_StructWithTag | jzon-fast-8 | 1000000 | 1132 | 192 | 5 |
| 10Fields_Unmarshal_StructWithTag | ugorji-8 | 476871 | 2705 | 1088 | 7 |
| 10Fields_Unmarshal_StructWithTag | json-8 | 274917 | 4346 | 432 | 14 |
| | | | | | |
| 10Fields_Unmarshal_StructWoTag | jsoniter-compatible-8 | 887858 | 1406 | 256 | 15 |
| 10Fields_Unmarshal_StructWoTag | jsoniter-8 | 872038 | 1451 | 256 | 15 |
| 10Fields_Unmarshal_StructWoTag | jzon-8 | 694782 | 1607 | 192 | 5 |
| 10Fields_Unmarshal_StructWoTag | json-8 | 242374 | 4879 | 432 | 14 |
| | | | | | |
| 10Fields_Valid | gjson-8 | 2991547 | 400 | 0 | 0 |
| 10Fields_Valid | jzon-8 | 2099695 | 567 | 0 | 0 |
| 10Fields_Valid | jsoniter-compatible-8 | 1583869 | 736 | 64 | 10 |
| 10Fields_Valid | jsoniter-8 | 1628780 | 758 | 64 | 10 |
| 10Fields_Valid | json-8 | 1000000 | 1114 | 0 | 0 |
| | | | | | |
| 20Fields_Unmarshal_Interface | jzon-fast-8 | 187482 | 6285 | 2732 | 53 |
| 20Fields_Unmarshal_Interface | jzon-8 | 193900 | 6516 | 2731 | 53 |
| 20Fields_Unmarshal_Interface | djson-8 | 238552 | 6606 | 2716 | 52 |
| 20Fields_Unmarshal_Interface | easyjson-8 | 194299 | 6974 | 2716 | 52 |
| 20Fields_Unmarshal_Interface | ujson-8 | 163780 | 7551 | 3340 | 73 |
| 20Fields_Unmarshal_Interface | jsoniter-compatible-8 | 150829 | 7872 | 3051 | 73 |
| 20Fields_Unmarshal_Interface | jsoniter-8 | 139962 | 8286 | 3051 | 73 |
| 20Fields_Unmarshal_Interface | json-8 | 121875 | 10689 | 3003 | 67 |
| 20Fields_Unmarshal_Interface | ugorji-8 | 119270 | 10768 | 4018 | 61 |
| | | | | | |
| 20Fields_Unmarshal_StructWithTag | jzon-8 | 512673 | 2345 | 368 | 9 |
| 20Fields_Unmarshal_StructWithTag | jzon-fast-8 | 497636 | 2402 | 368 | 9 |
| 20Fields_Unmarshal_StructWithTag | ugorji-8 | 484503 | 2532 | 1088 | 7 |
| 20Fields_Unmarshal_StructWithTag | jsoniter-8 | 420992 | 2780 | 512 | 29 |
| 20Fields_Unmarshal_StructWithTag | jsoniter-compatible-8 | 420163 | 2877 | 512 | 29 |
| 20Fields_Unmarshal_StructWithTag | json-8 | 145645 | 8466 | 648 | 24 |
| | | | | | |
| 20Fields_Unmarshal_StructWoTag | jsoniter-compatible-8 | 425262 | 2724 | 512 | 29 |
| 20Fields_Unmarshal_StructWoTag | jsoniter-8 | 435942 | 2758 | 512 | 29 |
| 20Fields_Unmarshal_StructWoTag | jzon-8 | 251348 | 4908 | 1057 | 9 |
| 20Fields_Unmarshal_StructWoTag | json-8 | 116900 | 9759 | 648 | 24 |
| | | | | | |
| 20Fields_Valid | gjson-8 | 1483011 | 776 | 0 | 0 |
| 20Fields_Valid | jzon-8 | 1000000 | 1070 | 0 | 0 |
| 20Fields_Valid | jsoniter-compatible-8 | 813631 | 1376 | 144 | 20 |
| 20Fields_Valid | jsoniter-8 | 772488 | 1418 | 144 | 20 |
| 20Fields_Valid | json-8 | 564940 | 2138 | 0 | 0 |
| | | | | | |

# TODO

- add medium & large payload tests
- auto benchmark

# Packages tested

## feature list

| repo | version | unmarshal (interface) | unmarshal (struct) | iterator | valid |
| -------------------------------------- | ------- | --------------- | --------------- | --- | --- |
| encoding/json | 1.15.2 | x | x | | x |
| github.com/json-iterator/go | 1.1.10 | x | x | x | x |
| github.com/buger/jsonparser | 1.0.0 | | | x | |
| github.com/a8m/djson | c02c5ae | x | | | |
| github.com/mailru/easyjson | 0.7.6 | x | (1) | | |
| github.com/mreiferson/go-ujson | c02629f | x(2) | | | |
| github.com/ugorji/go/codec | 1.1.10 | x(3) | x(4) | | |
| github.com/tidwall/gjson(5) | 1.6.1 | | (6) | | x |
| github.com/zerosnake0/jzon | 0.0.5 | x | x | x | x |

### notes:
1. easyjson use its own marshaler/unmarshaler, which is not tested
2. ujson use ujson.numeric which is its internal type for numbers
3. ugorji need to be configured in order to be compatible with standard library
4. ugorji decode into structure:
 - can decode into a structure with tag
 - cannot decode into a structure without a tag unless all field names are as same as tag names
 - use its own marshaler/unmarshaler (codec.Selfer) interface to decode structure
5. gjson deprecated its unmarshal methods from `v1.4.0`
6. gjson only decode into structure fields with tag (`<= v1.3.6`)


# The following packages are not tested

| repo | reason |
| ------------------------------- | ------------------------------------------ |
| github.com/Jeffail/gabs | the encoding/decoding uses `encoding/json` |
| github.com/bitly/go-simplejson | the encoding/decoding uses `encoding/json` |
| github.com/antonholmquist/jason | the encoding/decoding uses `encoding/json` |
| github.com/pquerna/ffjson | the encoding/decoding uses `encoding/json` (if the custom interface is not provided) |

go-json-benchmark

benchmarks for machine learning inference in Go.

# Go Machine Learning Benchmarks

> Given a raw data in a Go service, how quickly can I get machine learning inference for it?

Typically, Go is dealing with structured single sample data.
Thus, we are focusing on tabular machine learning models only, such as popular [XGBoost](https://github.com/dmlc/xgboost).
It is common to run Go service in a backed form and on Linux platform, thus we do not consider other deployment options.
In the work bellow, we compare typical implementations on how this inference task can be performed.

![diagram](docs/go-ml-benchmarks.png)

```
host: AWS EC2 t2.xlarge shared
os: Ubuntu 20.04 LTS 
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
BenchmarkXGB_Go_GoFeatureProcessing_GoLeaves_noalloc                              491 ns/op
BenchmarkXGB_Go_GoFeatureProcessing_GoLeaves                                      575 ns/op
BenchmarkXGB_Go_GoFeatureProcessing_UDS_RawBytes_Python_XGB                    243056 ns/op
BenchmarkXGB_CGo_GoFeatureProcessing_XGB                                       244941 ns/op
BenchmarkXGB_Go_GoFeatureProcessing_UDS_gRPC_CPP_XGB                           367433 ns/op
BenchmarkXGB_Go_GoFeatureProcessing_UDS_gRPC_Python_XGB                        785147 ns/op
BenchmarkXGB_Go_UDS_gRPC_Python_sklearn_XGB                                  21699830 ns/op
BenchmarkXGB_Go_HTTP_JSON_Python_Gunicorn_Flask_sklearn_XGB                  21935237 ns/op
```

### Abbreviations and Frameworks

- Transport: Unix Domain Sockets (UDS), TCP, HTTP
- Encoding: JSON, [gRPC](https://grpc.io/), raw bytes with fixed number of float64 IEEE 754
- Preprocessing: [go-featureprocessing](https://github.com/nikolaydubina/go-featureprocessing), [sklearn](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing)
- Model: [XGBoost](https://github.com/dmlc/xgboost), [Leaves](https://github.com/dmitryikh/leaves) (Leaves is XGBoost in native Go)
- Web Servers: for Python used [Gunicorn](https://gunicorn.org/) + [Flask](https://flask.palletsprojects.com/en/1.1.x/)

### Dataset and Model

We are using classic [Titanic dataset](https://www.kaggle.com/c/titanic).
It contains numerical and categorical features, which makes it a representative of typical case.
Data and notebooks to train model and preprocessor is available in /data and /notebooks.

### Some numbers for reference

How fast do you need to get?

```
                   200ps - 4.6GHz single cycle time
                1ns      - L1 cache latency
               10ns      - L2/L3 cache SRAM latency
               20ns      - DDR4 CAS, first byte from memory latency
               20ns      - C++ raw hardcoded structs access
               80ns      - C++ FlatBuffers decode/traverse/dealloc
              150ns      - PCIe bus latency
              171ns      - cgo call boundary, 2015
              200ns      - HFT FPGA
              475ns      - 2020 MLPerf winner recommendation inference time per sample
 ---------->  500ns      - go-featureprocessing + leaves
              800ns      - Go Protocol Buffers Marshal
              837ns      - Go json-iterator/go json unmarshal
           1µs           - Go protocol buffers unmarshal
           3µs           - Go JSON Marshal
           7µs           - Go JSON Unmarshal
          10µs           - PCIe/NVLink startup time
          17µs           - Python JSON encode/decode times
          30µs           - UNIX domain socket; eventfd; fifo pipes
         100µs           - Redis intrinsic latency; KDB+; HFT direct market access
         200µs           - 1GB/s network air latency; Go garbage collector pauses interval 2018
         230µs           - San Francisco to San Jose at speed of light
         500µs           - NGINX/Kong added latency
     10ms                - AWS DynamoDB; WIFI6 "air" latency
     15ms                - AWS Sagemaker latency; "Flash Boys" 300million USD HFT drama
     30ms                - 5G "air" latency
     36ms                - San Francisco to Hong-Kong at speed of light
    100ms                - typical roundtrip from mobile to backend
    200ms                - AWS RDS MySQL/PostgreSQL; AWS Aurora
 10s                     - AWS Cloudfront 1MB transfer time
```

### Profiling and Analysis

**[491ns/575ns]** Leaves — we see that most of time taken in Leaves Random Forest code. Leaves code does not have mallocs. Inplace preprocessing does not have mallocs, with non-inplace version malloc happen and takes and takes half of time of preprocessing.
![leaves](docs/profiles-readme/leaves.png)

**[243µs]** UDS Raw bytes Python — we see that Python takes much longer time than preprocessing in Go, however Go is at least visible on the chart. We also note that Python spends most of the time in `libgomp.so` call, this library is in GNU OpenMP written in C which does parallel operations.

![uds](docs/profiles-readme/uds.png)

**[244µs]** CGo version — similarly, we see that call to `libgomp.so` is being done. It is much smaller compare to rest of o CGo code, as compared to Python version above. Over overall results are not better then? Likely this is due to performance degradation from Go to CGo. We also note that malloc is done.

![cgo](docs/profiles-readme/cgo.png)

**[367µs]** gRPC over UDS to C++ — we see that Go code is around 50% of C++ version. In C++ 50% of time spend on gRPC code.
Lastly, C++ also uses `libgomp.so`. We don't see on this chart, but likely Go code also spends considerable time on gRPC code.

![cgo](docs/profiles-readme/grpc-cpp.png)

**[785µs]** gRPC over UDS to Python wihout sklearn — we see that Go code is visible in the chart. Python spends only portion on time in `libgomp.so`.

![cgo](docs/profiles-readme/grpc-python-processed.png)

**[21ms]** gRPC over UDS to Python with sklearn — we see that Go code (`main.test`) is no longer visible the chart. Python spends only small fraction of time on `libgomp.so`.

![cgo](docs/profiles-readme/grpc-python-sklearn.png)


**[22ms]** REST service version with sklearn — similarly, we see that Go code (`main.test`) is no longer visible in the chart. Python spends more time in `libgomp.so` as compared to Python + gRPC + skelarn version, however it is not clear why results are worse.

![cgo](docs/profiles-readme/rest.png)

### Future work

- [ ] go-featureprocessing - gRPCFlatBuffers - C++ - XGB
- [ ] batch mode
- [ ] UDS - gRPC - C++ - ONNX (sklearn + XGBoost)
- [ ] UDS - gRPC - Python - ONNX (sklearn + XGBoost)
- [ ] cgo ONNX (sklearn + XGBoost) (examples: [1](http://onnx.ai/sklearn-onnx/auto_examples/plot_pipeline_xgboost.html))
- [ ] native Go ONNX (sklearn + XGBoost) — no official support, https://github.com/owulveryck/onnx-go is not complete
- [ ] text
- [ ] images
- [ ] videos

### Reference

- [Go GC updates, 2018](https://blog.golang.org/ismmkeynote)
- [cgo performance, GopherCon'18](https://about.sourcegraph.com/go/gophercon-2018-adventures-in-cgo-performance/)
- [cgo performance, CockroachDB](https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/)
- [cgo call to CPython, Datadog](https://www.datadoghq.com/blog/engineering/cgo-and-python/)
- [cgo call to CPython, EuroPython'19](https://ep2019.europython.eu/talks/Zktoaai-golang-to-python/)
- [HFT latency](https://en.wikipedia.org/wiki/Ultra-low_latency_direct_market_access)
- [HFT FPGA latency](https://ieeexplore.ieee.org/document/6299067)
- [HFT FPGA 200 nanoseconds, 2018](https://apnews.com/press-release/pr-businesswire/2edb1f8f12d64ab490ef0c180e648e24)
- [Google TPU latency](https://ai.googleblog.com/2019/08/efficientnet-edgetpu-creating.html)
- [PCIe latency](https://www.cl.cam.ac.uk/research/srg/netos/projects/pcie-bench/neugebauer2018understanding.pdf)
- "Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect", 2019
- ["Evaluation of Inter-Process Communication Mechanisms"](http://pages.cs.wisc.edu/~adityav/Evaluation_of_Inter_Process_Communication_Mechanisms.pdf)
- [UNIX local IPC latencies](http://kamalmarhubi.com/blog/2015/06/10/some-early-linux-ipc-latency-data/)
- [Cache and DRAM latency](https://en.wikipedia.org/wiki/CPU_cache)
- [MLPerf benchmarks](https://github.com/mlcommons/inference)
- [MLPerf benchmarks results, 2020](https://mlperf.org/inference-results-0-7)
- [Redis latency](https://redis.io/topics/latency)
- [Huawei WIFI6 latency](https://e.huawei.com/sg/products/enterprise-networking/wlan/wifi-6)
- [Verizon 5G latency](https://www.verizon.com/about/our-company/5g/5g-latency)
- [NGINX added latency](https://www.nginx.com/blog/nginx-controller-api-management-module-vs-kong-performance-comparison/)
- [AWS Sagemaker latency](https://aws.amazon.com/blogs/machine-learning/load-test-and-optimize-an-amazon-sagemaker-endpoint-using-automatic-scaling/)
- [AWS Aurora latency](https://aws.amazon.com/blogs/database/using-aurora-to-drive-3x-latency-improvement-for-end-users/)
- [AWS Cloudfront transfer rates](https://media.amazonwebservices.com/FS_WP_AWS_CDN_CloudFront.pdf)


go-ml-benchmarks

# go-web-framework-benchmark
This benchmark suite aims to compare the performance of Go web frameworks. It is inspired by [Go HTTP Router Benchmark](https://github.com/julienschmidt/go-http-routing-benchmark) but this benchmark suite is different with that. Go HTTP Router Benchmark suit aims to compare the performance of **routers** but this Benchmark suit aims to compare whole HTTP request processing.

**Last Test Updated:** 2020-05

*test environment*

* CPU:      KVM Virtual CPU version(2 GHz, 4 cores)
* Memory:   16G
* Go:       go1.18.5 linux/amd64
* OS:       Ubuntu 22.04.1 LTS with Kernel 5.15.0-41-generic


## Tested web frameworks (in alphabetical order)

**Only test those webframeworks which are stable**

* [atreugo](https://github.com/savsgio/atreugo)
* [baa](https://github.com/go-baa/baa)
* [beego](https://github.com/astaxie/beego)
* [bone](https://github.com/go-zoo/bone)
* [chi](https://github.com/go-chi/chi)
* [clevergo](https://github.com/clevergo/clevergo)
* [default http](https://golang.org/pkg/net/http/)
* [denco](https://github.com/naoina/denco)
* [don](https://github.com/abemedia/go-don)
* [echo](https://github.com/labstack/echo)
* [fasthttp-routing](https://github.com/qiangxue/fasthttp-routing)
* [fasthttp/router](https://github.com/fasthttp/router)
* [fasthttp](https://github.com/valyala/fasthttp)
* [fiber](https://gofiber.io/)
* [gear](http://github.com/teambition/gear)
* [gearbox](https://github.com/gogearbox/gearbox)
* [gem](https://github.com/go-gem/gem)
* [gin](https://github.com/gin-gonic/gin)
* [goframe](https://github.com/gogf/gf)
* [go-ozzo](https://github.com/go-ozzo/ozzo-routing)
* [go-restful](https://github.com/emicklei/go-restful)
* [go-tigertonic](https://github.com/rcrowley/go-tigertonic)
* [goji](https://github.com/zenazn/goji/web)
* [goji](http://goji.io)
* [golf](https://github.com/dinever/golf)
* [gorilla](https://github.com/gorilla/mux)
* [gorouter](https://github.com/vardius/gorouter)
* [goyave](https://github.com/System-Glitch/goyave)
* [httprouter](https://github.com/julienschmidt/httprouter)
* [httptreemux](https://github.com/dimfeld/httptreemux)
* [indigo](https://github.com/indigo-web/indigo)
* [lars](https://github.com/go-playground/lars)
* [lion](https://github.com/celrenheit/lion)
* [macaron](https://github.com/Unknwon/macaron)
* [muxie](https://github.com/kataras/muxie)
* [negroni](https://github.com/urfave/negroni)
* [pat](https://github.com/bmizerany/pat)
* [pulse](https://github.com/gopulse/pulse)
* [pure](https://github.com/go-playground/pure)
* [r2router](https://github.com/vanng822/r2router)
* [tango](https://github.com/lunny/tango)
* [tinyrouter](https://github.com/go101/tinyrouter)
* [treemux](https://github.com/vmihailenco/treemux)
* [violetear](https://github.com/nbari/violetear)
* [vulcan](https://github.com/mailgun/route)
* [webgo](https://github.com/bnkamalesh/webgo)

**some libs have not been maintained and the test code has removed them**

## Motivation
When I investigated performance of Go web frameworks, I found [Go HTTP Router Benchmark](https://github.com/julienschmidt/go-http-routing-benchmark), created by Julien Schmidt. He also developed a high performance http router: [httprouter](https://github.com/julienschmidt/httprouter). I had thought I got the performance result until I created a piece of codes to mock the real business logics:

```go
api.Get("/rest/hello", func(c *XXXXX.Context) {
	sleepTime := strconv.Atoi(os.Args[1]) //10ms
	if sleepTime > 0 {
		time.Sleep(time.Duration(sleepTime) * time.Millisecond)
	}

	c.Text("Hello world")
})
```

When I use the above codes to test those web frameworks, the token time of route selection is not so important in the whole http request processing, although performance of route selection of web frameworks are very different.

So I create this project to compare performance of web frameworks including connection, route selection, handler processing. It mocks business logics and can set a special processing time.

Then you can get some interesting results if you use it to test.

## Implementation
When you test a web framework, this test suit will starts a simple http server implemented by this web framework. It is a real http server and only contains GET url: "/hello".

When this server processes this url, it will sleep n milliseconds in this handler. It mocks the business logics such as:
* read data from sockets
* write data to disk
* access databases
* access cache servers
* invoke other microservices
* ……

It contains a test.sh that can do those tests automatically.

It uses [wrk](https://github.com/wg/wrk/) to test.

## Basic Test
The first test case is to mock 0 ms, 10 ms, 100 ms, 500 ms processing time in handlers.

![Benchmark (Round 3)](benchmark.png)
the concurrency clients are 5000.

![Latency (Round 3)](benchmark_latency.png)
Latency is the time of real processing time by web servers. The smaller is the better.

![Allocs (Round 3)](benchmark_alloc.png)
Allocs is the heap allocations by web servers when test is running. The unit is MB. The smaller is the better.


If we enable http pipelining, test result as below:

![benchmark pipelining (Round 2)](benchmark-pipeline.png)

## Concurrency Test
In 30 ms processing time, the test result for 100, 1000, 5000 clients is:

![concurrency (Round 3)](concurrency.png)

![Latency (Round 3)](concurrency_latency.png)

![Latency (Round 3)](concurrency_alloc.png)


If we enable http pipelining, test result as below:

![concurrency pipelining(Round 2)](concurrency-pipeline.png)


## cpu-bound case Test

![cpu-bound (5000 concurrency)](cpubound_benchmark.png)

## Usage
You should install this package first if you want to run this test.

```
go get github.com/smallnest/go-web-framework-benchmark
```

It takes a while to install a large number of dependencies that need to be downloaded. Once that command completes, you can run:

```
cd $GOPATH/src/github.com/smallnest/go-web-framework-benchmark
go build -o gowebbenchmark .
./test.sh
```

It will  generate test results in processtime.csv and concurrency.csv. You can modify test.sh to execute your customized test cases.


* If you also want to generate latency data and allocation data, you can run the script:
```
./test-latency.sh
```

* If you don't want use keepalive, you can run:
```
./test-latency-nonkeepalive.sh
```

* If you want to test http pipelining, you can run:
```
./test-pipelining.sh
```

* If you want to test some of web frameworks, you can modify the test script and only keep your selected web frameworks:
```
……
web_frameworks=("default" "atreugo" "beego" "bone" "chi" "denco" "don" "echo" "fasthttp" "fasthttp-routing" "fasthttp/router" "fiber" "gear" "gearbox" "gin" "goframe" "goji" "gorestful" "gorilla" "gorouter" "gorouterfasthttp" "go-ozzo" "goyave" "httprouter" "httptreemux" "indigo" "lars" "lion" "muxie" "negroni" "pat" "pulse" "pure" "r2router" "tango" "tiger" "tinyrouter" "violetear" "vulcan" "webgo")
……
```
* If you want to test all cases, you can run:

```
./test-all.sh
```

NOTE: comparing 2 webframeworks consumes approx. 11-13 minutes (doesn't depend on the machine). Just `test.sh` with all
the webframeworks enabled will take a couple of hours to run.

## Plot
All the graphs are generated automatically as the `./test.sh` finishes. However, if the run was interrupted, you can
generate them manually of partial data by executing `plot.sh` in testresults directory.

### Add new web framework
Welcome to add new Go web frameworks. You can follow the below steps and send me a pull request.

1. add your web framework link in README
2. add a hello implementation in server.go
3. add your webframework in libs.sh

Please add your web framework alphabetically.


go-web-framework-benchmark

# Benchmarks of Go serialization methods

[![Gitter chat](https://badges.gitter.im/alecthomas.png)](https://gitter.im/alecthomas/Lobby)

This is a test suite for benchmarking various Go serialization methods.

# Current Serialization Results

https://alecthomas.github.io/go_serialization_benchmarks

## Running the benchmarks

To benchmark and validate, without cloning the repository, replace the `.` from the commands below with `github.com/alecthomas/go_serialization_benchmarks@latest`.

```bash
go run .
```

To validate the correctness of the serializers:
```bash
go run . --validate
```

To update the benchmark report:
```bash
go run . --genreport
```

To update the benchmark report with a longer benchmark run (to get more accurate results):
```bash
go test -tags genreport -run TestGenerateReport -benchtime 10s -timeout 1h #--validate
```

## Recommendation

If correctness and interoperability are the most
important factors [JSON](http://golang.org/pkg/encoding/json/) or [Protobuf](https://google.golang.org/protobuf) are your best options.

But as always, make your own choice based on your requirements.

## Adding New Serializers

Review the following instructions _before_ opening the PR to add a new
serializer:

- Create all the required serializer code in `internal/<short serializer name>`.
- Add an entry to the serializer in [benchmarks.go](benchmarks.go).
- If the serializer supports both reusing/not reusing its marshalling buffer:
 - Add both a `serializer` and `serializer/reuse` entries, each one
 respectively reusing/not reusing the resulting marshalling buffer. Set the
 `BufferReuseMarshal` flag accordingly.
- If the serializer supports both safe and unsafe string unmarshalling:
 - Add both a `serializer` and `serializer/unsafe` entries, each one
 respectively unmarshalling into safe and unsafe strings. Set the
 `UnsafeStringUnmarshal` flag accordingly.
- If the serializer supports both marshalling buffer reuse and unsafe string
 unmarshalling, merge both options into a single `serializer/unsafe_reuse`
 entry (check the baseline serializer for an example).
- Regenerate the report by running:

```
go run . --genreport
```

- **Include the updated report data in your PR**

## Data

The data being serialized is the following structure with randomly generated values:

```go
type A struct {
 Name string
 BirthDay time.Time
 Phone string
 Siblings int
 Spouse bool
 Money float64
}
```

go_serialization_benchmarks

Benchmarks of common basic operations for the Go language.

# gocostmodel

This package was inspired by Brian W. Kernighan and Rob Pike's book "The Practice of Programming" (Addison-Wesley, 1999). In Chapter 7 on performance, section 7.6 (Estimation):

> "It's hard to estimate ahead of time how fast a program will be [...]. It's easy, though, to create a *cost model* for a language or a system, which will give you at least a rough idea of how long important operations take."

This is a *cost model* that benchmarks common basic operations for the Go language. For it to be meaningful, it should be run on your target hardware, and the relative performance of operations is probably more important than the actual numbers. Examples of such results on a variety of hardware is available in the `bench/` folder.

There may be places where compiler optimizations render some benchmarks moot. If you find any such occurrences, please open an issue or send a pull request.

The names of the `*_test.go` source files should give an idea of the category of operations that are benchmarked.

## Example

Here is a short example of benchmark results, run on a Toshiba Chromebook 64-bit running Ubuntu (LXDE) with crouton:

```
BenchmarkConvIntToFloat64	1000000000	         2.16 ns/op	       0 B/op	       0 allocs/op
BenchmarkConvFloat64ToInt	2000000000	         1.26 ns/op	       0 B/op	       0 allocs/op
BenchmarkConvIntToFloat32	2000000000	         1.44 ns/op	       0 B/op	       0 allocs/op
BenchmarkConvFloat32ToInt	2000000000	         1.42 ns/op	       0 B/op	       0 allocs/op
BenchmarkConvStringToInt	30000000	        57.8 ns/op	       0 B/op	       0 allocs/op
BenchmarkConvIntToString	20000000	       108 ns/op	       1 B/op	       1 allocs/op
BenchmarkStringToByte	20000000	       110 ns/op	       8 B/op	       1 allocs/op
BenchmarkByteToString	20000000	        84.8 ns/op	       4 B/op	       1 allocs/op
BenchmarkFindSliceInt5	100000000	        13.0 ns/op	       0 B/op	       0 allocs/op
BenchmarkFindSliceStr5	20000000	        64.9 ns/op	       0 B/op	       0 allocs/op
BenchmarkFindSliceInt100	10000000	       236 ns/op	       0 B/op	       0 allocs/op
BenchmarkFindSliceStr100	 1000000	      1248 ns/op	       0 B/op	       0 allocs/op
BenchmarkFindMapInt5	100000000	        22.2 ns/op	       0 B/op	       0 allocs/op
BenchmarkFindMapStr5	20000000	        77.6 ns/op	       0 B/op	       0 allocs/op
BenchmarkFindMapInt100	30000000	        58.3 ns/op	       0 B/op	       0 allocs/op
BenchmarkFindMapStr100	20000000	        76.2 ns/op	       0 B/op	       0 allocs/op
BenchmarkFloat32Assign	2000000000	         0.99 ns/op	       0 B/op	       0 allocs/op
```

## Installation

    $ go get github.com/PuerkitoBio/gocostmodel

## Running the benchmarks

    $ go test -run zz -bench . [-benchmem]

## Contributing

Pull requests are welcome, but the benchmarks should be for common basic operations, in the spirit of the existing benchmarks.

## License

This source code is released under the [BSD 3-clause license][bsd].

[bsd]: http://opensource.org/licenses/BSD-3-Clause



gocostmodel

golang-benchmarks

Collection of benchmarks for popular Go database/SQL utilities.

golang-db-sql-benchmark
====================

A collection of benchmarks for popular Go database/SQL utilities

# Libraries under test

* [database/sql](https://golang.org/pkg/database/sql/) + [go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)
* [gocraft/dbr](https://github.com/gocraft/dbr)
* [Gorp](https://github.com/coopernurse/gorp)
* [Sqlx](https://github.com/jmoiron/sqlx)
* [Squirrel](https://github.com/lann/squirrel)

# database/sql SQL Execution Benchmarks:

* **BenchmarkPreparedStatementsNone** - Runs simple queries without query arguments, so database/sql doesn't need to create a prepared statement
* **BenchmarkPreparedStatementsThrowaway** - Runs queries with query arguments. database/sql must create and then throwaway a prepared statement each time
* **BenchmarkPreparedStatementsSingle** - Runs queries with query arguments, but creates and reuses the a single prepared statement


# Dbr/Sqlx/Gorp SQL Execution Benchmarks:
Each library under test has the same set of benchmarks, just replace `Dbr` in the examples with `Sqlx` or `Gorp`.
Each one is run with varying number of rows, N.

* **BenchmarkDbrSelectIntsN** - Select rows of integers into []int64's
* **BenchmarkDbrSelectAllN** - Select rows into structs using no query arguments
* **BenchmarkDbrSelectAllWithArgsN** - Select rows into structs using query arguments

# Dbr/Squrrel SQL Building Benchamrks
Test building (but not executing) various SQL statements

* **BenchmarkBuilderDbrSimple** - Simple SQL query with dbr
* **BenchmarkBuilderDbrComplex** - Complex SQL query with dbr
* **BenchmarkBuilderSquirrelSimple** - Simple SQL query with squirrel
* **BenchmarkBuilderSquirrelComplex** - Complex SQL query with squirrel

# Output

`godep go test -bench=. -benchmem 2>/dev/null | column -t` on @tyler-smith's 2.6 GHz i7 Macbook Pro:

```
BenchmarkPreparedStatementsNone 100000 23137 ns/op 612 B/op 20 allocs/op
BenchmarkPreparedStatementsThrowaway 20000 94086 ns/op 795 B/op 25 allocs/op
BenchmarkPreparedStatementsSingle 50000 33426 ns/op 652 B/op 23 allocs/op

BenchmarkDbrSelectInts1 100000 30098 ns/op 1198 B/op 22 allocs/op
BenchmarkDbrSelectInts100 20000 94477 ns/op 11645 B/op 426 allocs/op
BenchmarkDbrSelectInts1000 5000 627958 ns/op 100574 B/op 4041 allocs/op
BenchmarkDbrSelectInts10000 500 6207828 ns/op 1297929 B/op 40177 allocs/op
BenchmarkDbrSelectAll1 50000 35468 ns/op 1876 B/op 60 allocs/op
BenchmarkDbrSelectAll100 10000 157510 ns/op 29256 B/op 863 allocs/op
BenchmarkDbrSelectAll1000 2000 1436856 ns/op 272365 B/op 8078 allocs/op
BenchmarkDbrSelectAll10000 100 12927094 ns/op 2979829 B/op 80171 allocs/op
BenchmarkDbrSelectAllWithArgs1 50000 38959 ns/op 2590 B/op 72 allocs/op
BenchmarkDbrSelectAllWithArgs100 10000 162227 ns/op 29970 B/op 875 allocs/op
BenchmarkDbrSelectAllWithArgs1000 2000 1227154 ns/op 273765 B/op 8093 allocs/op
BenchmarkDbrSelectAllWithArgs10000 100 13342026 ns/op 3000164 B/op 80284 allocs/op

BenchmarkSqlxSelectInts1 20000 86128 ns/op 628 B/op 22 allocs/op
BenchmarkSqlxSelectInts100 10000 203886 ns/op 9480 B/op 426 allocs/op
BenchmarkSqlxSelectInts1000 2000 1289621 ns/op 83799 B/op 4038 allocs/op
BenchmarkSqlxSelectInts10000 100 15156243 ns/op 1134724 B/op 40139 allocs/op
BenchmarkSqlxSelectAll1 50000 32373 ns/op 959 B/op 25 allocs/op
BenchmarkSqlxSelectAll100 10000 177965 ns/op 28366 B/op 828 allocs/op
BenchmarkSqlxSelectAll1000 1000 1634705 ns/op 271352 B/op 8042 allocs/op
BenchmarkSqlxSelectAll10000 100 17483420 ns/op 2989883 B/op 80191 allocs/op
BenchmarkSqlxSelectAllWithArgs1 10000 140460 ns/op 1228 B/op 30 allocs/op
BenchmarkSqlxSelectAllWithArgs100 10000 250621 ns/op 23682 B/op 634 allocs/op
BenchmarkSqlxSelectAllWithArgs1000 2000 1548073 ns/op 229879 B/op 6049 allocs/op
BenchmarkSqlxSelectAllWithArgs10000 100 15441239 ns/op 2587009 B/op 60190 allocs/op

BenchmarkGorpSelectInts1 10000 134277 ns/op 432 B/op 15 allocs/op
BenchmarkGorpSelectAll1 50000 44086 ns/op 1983 B/op 75 allocs/op
BenchmarkGorpSelectAll100 10000 232527 ns/op 34258 B/op 978 allocs/op
BenchmarkGorpSelectAll1000 1000 1563675 ns/op 321846 B/op 9100 allocs/op
BenchmarkGorpSelectAll10000 100 16451807 ns/op 3472508 B/op 90250 allocs/op
BenchmarkGorpSelectAllWithArgs1 10000 131738 ns/op 2306 B/op 80 allocs/op
BenchmarkGorpSelectAllWithArgs100 10000 265323 ns/op 29948 B/op 785 allocs/op
BenchmarkGorpSelectAllWithArgs1000 2000 1423747 ns/op 279838 B/op 7103 allocs/op
BenchmarkGorpSelectAllWithArgs10000 100 14538568 ns/op 3072519 B/op 70263 allocs/op

BenchmarkBuilderDbrSimple 1000000 1699 ns/op 866 B/op 13 allocs/op
BenchmarkBuilderDbrComplex 500000 6193 ns/op 2190 B/op 37 allocs/op

BenchmarkBuilderSquirrelSimple 200000 8981 ns/op 2780 B/op 51 allocs/op
BenchmarkBuilderSquirrelComplex 50000 44721 ns/op 11707 B/op 259 allocs/op
```

# Run yourself

* Use the `godep` tool or manually install all libraries under test
* Create db: `mysql -e "create database golang_sql_benchmarks;"`
* Create schema: `mysql golang_sql_benchmarks < structure.sql`
* Run: `godep go test -bench=. -benchmem`
* You can set the MySQL DSN to use by setting the GOLANG_SQL_BENCHMARKS_DSN env var (defaults to root@unix(/var/run/mysqld/mysqld.sock)/golang_sql_benchmarks)

golang-sql-benchmark

Go micro-benchmarks for calculating the speed of language constructs.

== About ==

GoSpeed is a library of micro-benchmarks for Go which evolved from the GoLightly project. It's main
utility is for understanding and reasoning about the performance of Go programs.


== Usage ==

Download the repo to a convenient location in your $GOPATH

	git clone git@github.com:feyeleanor/gospeed.git

or install it to pkg/mod

	go get github.com/feyeleanor/GoSpeed

On macOS/Linux run the benchmarks with:

	go test -test.bench=".*"

or on Windows

	go test -bench .*

If you're running on older hardware such as an Intel Atom N270 netbook (which the early versions of
GoSpeed were developed on) then you may also need to add a custom timeout to complete benchmarks
which make significant use of channels as the default aggregate test timeout is ten minutes.

On macOS/Linux

	go test -test.bench=".*" -test.timeout="60m"

on Windows

	go test -bench .* -timeout 60m


gospeed

USAGE:

kvbench OPTIONS

OVERVIEW

kvbench is a simple benchmarking tool to evaluate the read performance
of a key/value store while writes are being applied in parallel.

DETAILS:

Running a benchmark consists of two steps:

(1) Generate a sample data file using the -o option.
 Use the -n, -b[01], -k[01], and -v[01] options to control the
 size of the sample. Use the -r <seed> option to change the
 pseudo-random data. Given identical inputs, an identical data
 file should be generated.

(2) Consume a sample data file and execute a benchmark using the
 -i option. Use the -d[01] options to control the inter-arrival
 rate of new row sets to be written to the collection. The -p
 option controls how often the benchmark will attempt to iterate
 over the keys.

You may specify both -o and -i options to generate the data and
then immediately run the benchmark.

OUTPUT OPTIONS

- -r n - pseudo-random seed
- -n n - total number of blocks to generate
- -b0 min - minimum number of records per block
- -b1 max - maximum number of records per block
- -k0 min - minimum length of key to generate
- -k1 max - maximum length of key to generate
- -v0 min - minimum length of value to generate
- -v1 max - maximum length of value to generate
-o dat - output path for data file

INPUT OPTIONS

- -r n - pseudo-random seed
- -d0 dur - minimum inter-arrival rate
- -d1 dur - maximum inter-arrival rate (not guaranteed)
- -p dur - poll db at this interval and print statistics
- -i dat - input path for data file
- -b bench - name of the benchmark to run (bolt, kv, kv-mu, leveldb, noop)
- -f path - path to the database

EXAMPLE

````
$ ./kvbench -n 100 -b0 0 -b1 4000 -k0 34 -k1 34 -v0 70 -v1 78 -o sample.dat
2014/04/16 19:57:46 writing data to sample.dat

$ ./kvbench -i sample.dat -b leveldb -f test/leveldb.db -d0 50ms -d1 175ms -p 10s
2014/04/16 19:57:51 reading data from sample.dat
2014/04/16 19:58:02 leveldb: 178046 ops in 296 ms: 601 ops/ms
2014/04/16 19:58:12 leveldb: 202517 ops in 315 ms: 642 ops/ms
2014/04/16 19:58:12 100 row sets arrived at an average inter-arrival rate of 116.046572ms
2014/04/16 19:58:12 leveldb: 202517 ops in 308 ms: 657 ops/ms

$ ./kvbench -i sample.dat -b bolt -f test/bolt.db -d0 50ms -d1 175ms -p 10s
2014/04/16 19:58:31 reading data from sample.dat
2014/04/16 19:58:41 bolt: 151830 ops in 141 ms: 1076 ops/ms
2014/04/16 19:58:51 bolt: 202517 ops in 210 ms: 964 ops/ms
2014/04/16 19:58:51 100 row sets arrived at an average inter-arrival rate of 134.606206ms
2014/04/16 19:58:51 bolt: 202517 ops in 168 ms: 1205 ops/ms

$ ./kvbench -i sample.dat -b kv -f test/kv.db -d0 50ms -d1 175ms -p 10s
2014/04/16 19:58:53 reading data from sample.dat
^C

$ date
Wed Apr 16 19:59:54 PDT 2014

$ ./kvbench -i sample.dat -b kv-mu -f test/kv-mu.db -d0 50ms -d1 175ms -p 10s
2014/04/16 20:00:01 reading data from sample.dat
2014/04/16 20:00:12 kv-mu: 18575 ops in 378 ms: 49 ops/ms
2014/04/16 20:00:24 kv-mu: 34127 ops in 588 ms: 58 ops/ms
2014/04/16 20:00:36 kv-mu: 50581 ops in 554 ms: 91 ops/ms
2014/04/16 20:00:49 kv-mu: 65373 ops in 1267 ms: 51 ops/ms
2014/04/16 20:01:00 kv-mu: 78226 ops in 1108 ms: 70 ops/ms
2014/04/16 20:01:12 kv-mu: 90815 ops in 1395 ms: 65 ops/ms
2014/04/16 20:01:25 kv-mu: 102478 ops in 2174 ms: 47 ops/ms
2014/04/16 20:01:38 kv-mu: 112228 ops in 1869 ms: 60 ops/ms
2014/04/16 20:01:51 kv-mu: 122214 ops in 2357 ms: 51 ops/ms
2014/04/16 20:02:06 kv-mu: 133389 ops in 1981 ms: 67 ops/ms
2014/04/16 20:02:18 kv-mu: 142738 ops in 1950 ms: 73 ops/ms
2014/04/16 20:02:30 kv-mu: 150676 ops in 1979 ms: 76 ops/ms
2014/04/16 20:02:46 kv-mu: 161695 ops in 2518 ms: 64 ops/ms
2014/04/16 20:03:00 kv-mu: 170495 ops in 2986 ms: 57 ops/ms
2014/04/16 20:03:16 kv-mu: 180825 ops in 3610 ms: 50 ops/ms
2014/04/16 20:03:33 kv-mu: 191325 ops in 3493 ms: 54 ops/ms
2014/04/16 20:03:48 kv-mu: 198455 ops in 4151 ms: 47 ops/ms
2014/04/16 20:04:01 kv-mu: 202517 ops in 3252 ms: 62 ops/ms
2014/04/16 20:04:01 100 row sets arrived at an average inter-arrival rate of 2.300565857s
2014/04/16 20:04:05 kv-mu: 202517 ops in 3538 ms: 57 ops/ms

$ ./kvbench -i sample.dat -b kv -f test/kv-2.db -d0 2s -d1 5s -p 10s
2014/04/16 20:04:40 reading data from sample.dat
2014/04/16 20:04:50 kv: 2936 ops in 39 ms: 75 ops/ms
2014/04/16 20:05:00 kv: 7593 ops in 110 ms: 69 ops/ms
2014/04/16 20:05:11 kv: 14963 ops in 287 ms: 52 ops/ms
2014/04/16 20:05:21 kv: 20289 ops in 274 ms: 74 ops/ms
2014/04/16 20:05:33 kv: 27723 ops in 2334 ms: 11 ops/ms
2014/04/16 20:05:44 kv: 34127 ops in 469 ms: 72 ops/ms
2014/04/16 20:05:54 kv: 40052 ops in 524 ms: 76 ops/ms
2014/04/16 20:06:05 kv: 44990 ops in 672 ms: 66 ops/ms
2014/04/16 20:06:16 kv: 51604 ops in 733 ms: 70 ops/ms
2014/04/16 20:06:27 kv: 58077 ops in 976 ms: 59 ops/ms
2014/04/16 20:06:39 kv: 69760 ops in 2488 ms: 28 ops/ms
2014/04/16 20:06:50 kv: 77416 ops in 1133 ms: 68 ops/ms
2014/04/16 20:07:02 kv: 82087 ops in 1714 ms: 47 ops/ms
2014/04/16 20:07:13 kv: 88134 ops in 1253 ms: 70 ops/ms
2014/04/16 20:07:27 kv: 97417 ops in 3505 ms: 27 ops/ms
2014/04/16 20:07:44 kv: 107525 ops in 7438 ms: 14 ops/ms
2014/04/16 20:08:00 kv: 118256 ops in 6163 ms: 19 ops/ms
2014/04/16 20:08:16 kv: 130346 ops in 5284 ms: 24 ops/ms
2014/04/16 20:08:36 kv: 143446 ops in 10035 ms: 14 ops/ms
2014/04/16 20:08:48 kv: 150676 ops in 2252 ms: 66 ops/ms
2014/04/16 20:09:04 kv: 158712 ops in 6425 ms: 24 ops/ms
2014/04/16 20:09:41 kv: 184613 ops in 26382 ms: 6 ops/ms
2014/04/16 20:10:00 kv: 197430 ops in 9778 ms: 20 ops/ms
2014/04/16 20:10:15 kv: 202517 ops in 4464 ms: 45 ops/ms
2014/04/16 20:10:15 100 row sets arrived at an average inter-arrival rate of 3.28728241s
2014/04/16 20:10:18 kv: 202517 ops in 3482 ms: 58 ops/ms
````

kvbench

# Skynet 1M concurrency microbenchmark

Creates an actor (goroutine, whatever), which spawns 10 new actors, each of them spawns 10 
more actors, etc. until one million actors are created on the final level. Then, each of them returns
back its ordinal number (from 0 to 999999), which are summed on the previous level and sent back upstream,
until reaching the root actor. (The answer should be 499999500000).

## Results (on my shitty Macbook 12" '2015, **Core M**, OS X): 

### Actors

- Scala/Akka: 6379 ms. 
- Erlang (non-HIPE): 4414 ms.
- Erlang (HIPE): 3999 ms.

### Coroutines / fibers / channels

- Haskell (GHC 7.10.3): 6181 ms.
- Go: 979 ms.
- Quasar fibers and channels (Java 8): TODO

### Futures / promises

- .NET Core: 650 ms.
- RxJava: 219 ms.

## Results (**i7-4770**, Win8.1): 

### Actors

- Scala/Akka: 4419 ms
- Erlang (non-HIPE): 1700 ms.

### Coroutines / fibers / channels

- Haskell (GHC 7.10.3): 2820 ms.
- Go: 629 ms.
- F# MailboxProcessor: 756ms. (should be faster?..)
- Quasar fibers and channels (Java 8): TODO

### Futures / promises

- .NET Core: Async (8 threads) 290 ms
- Node-bluebird (Promise) 285ms / 195ms (after warmup)
- .NET Full (TPL): 118 ms.

## Results (**i7-4771**, Ubuntu 15.10): 

- Scala/Akka: 1700-2700 ms
- Haskell (GHC 7.10.3): 41-44 ms
- Erlang (non-HIPE): 700-1100 ms
- Erlang (HIPE): 2100-3500 ms
- Go: 200-224 ms
- LuaJit: 297 ms

## How to run

### Scala/Akka

Install latest Scala and SBT. 

Go to `scala/`, then run `sbt compile run`.

### Java/Quasar

Install the Java 8 SDK.

Go to `java-quasar/`
`./gradlew`

### Go

Install latest Go compiler/runtime.

In `go/`, run `go run skynet.go`.

### Pony

Install latest Pony compiler.

In `pony/`, run `ponyc -b skynet && ./skynet`.

### Erlang

Install latest Erlang runtime.

In `erlang`, run `erl +P 2000000` (to raise process limit), then compile:

- For non-HIPE: `c(skynet).`
- For HIPE (if supported on your system): `hipe:c(skynet).`

Then, run:

`skynet:skynet(1000000,10).`

### .NET Core: 

Install latest version of .NET Core

Go to `dnx/`  
`dotnet restore` (first time)  
`dotnet run --configuration Release`

### Haskell

Install [Stack](http://haskellstack.org)

In `haskell/`, run `stack build && stack exec skynet +RTS -N`

### Node (bluebird)

Install node.js

in `node-bluebird/` run `npm install` then `node skynet`

### FSharp

Install FSharp Interactive

Run fsi skynet.fsx, or run fsi and paste the code in (runs faster this way)

### Crystal:

Install [latest version of Crystal](http://crystal-lang.org/docs/installation/index.html).

Go to `crystal/`
`crystal build skynet.cr --release`
`./skynet`

### .NET/TPL

Build the solution with VS2015. Windows only :(

=======
### Java

Install the Java 8 SDK.

Go to `java/`
`./gradlew :run`

### Rust (with [coroutine-rs](https://github.com/rustcc/coroutine-rs))

```bash
cd ./rust-coroutine
cargo build --release
cargo run --release
```

### LuaJIT

Install luajit

Run `luajit luajit/skynet.lua`

### Scheme/Guile Fibers

Install Guile, Guile fibers, and wisp; for example via `guix package -i guile guile-fibers guile-wisp`.

Go to `guile-fibers`
`./skynet.scm`


skynet

Compare various Image resize algorithms for the Go language.

speedtest-resize
================

Compare various Image resize algorithms for the Go language

I am writing a web gallery called gonagall in Go
(https://github.com/fawick/gonagall). For that, I need a efficient solution for
scaling and resizing a lot of images (mostly JPGs) to generate thumbnails and
bandwidth-friendly sized copies from high-resolution original photo files.

In this project I compare the speed of a few selected image resizing algorithms
with each other as well as with ImageMagick and GraphicsMagick. The competitors
are

- https://github.com/nfnt/resize, Pure golang image resizing, more precisely
 only Nearest-Neighbor interpolation that comes with that Go package.
- https://github.com/disintegration/gift Again, I use one of the fastest
 algorithms of the package. Here, it's called 'Box'
- https://github.com/disintegration/imaging Again, I use one of the fastest
 algorithms of the package. Here, it's called 'Box'
- https://github.com/anthonynsimon/bild A collection of parallel image
 processing algorithms in pure Go ('NearestNeighbor' algorithm)
- [ImageMagick convert](http://www.imagemagick.org/script/convert.php) with the options `-resize 150x150>`
- [ImageMagick convert](http://www.imagemagick.org/script/convert.php) with the
 options `-define "jpeg:size=300x300 -thumbnail 150x150>`. `-thumbnail` is
considered to be faster than resize, and the `-define` will reduce the size (in
terms of memory footprint) of the original image on reading.
- [GraphicsMagick convert](http://www.graphicsmagick.org/convert.html) with the
 options `-define "jpeg:size=300x300 -thumbnail 150x150>`.
- https://github.com/gographics/imagick Go wrapper for the MagickWand API,
 again the Box algorithm is used for the sake of comparing the results.
- https://github.com/lazywei/go-opencv Go binding for OpenCV, using the fastest
 algorithm.
- https://github.com/bamiaux/rez, pure go resizer, using bilinear interpolation
 in these tests
- https://github.com/DAddYE/vips, bindings for libvips
 (http://www.vips.ecs.soton.ac.uk/index.php?title=Libvips)
- https://github.com/daddye/trez, an image resizer build on top of OpenCV and
 jpeg-turbo
- https://camlistore.org/pkg/images/fastjpeg, package fastjpeg uses djpeg(1),
 from the Independent JPEG Group's (www.ijg.org) jpeg package, to quickly
 down-sample images on load
- External command `vipsthumbnail` with parameters `-s 150`
 (https://github.com/libvips/libvips)
- External command `epeg` with parameters `-m 150`
 (https://github.com/mattes/epeg)

### Installation

To run the tests `go get` the source and compile/run it:

 $ go get -u github.com/fawick/speedtest-resize -tags all
 $ cd $GOPATH/src/speedtest-resize
 $ go run main.go <jpg file folder>

Alternatively, call the go command (or the compiled binary) from the image
folder without supplying a parameter

 $ cd <jpg file folder>
 $ go run $GOPATH/src/speedtest-resize/main.go

A the package requires different 3rdparty libraries to be installed, you can
use build tags to control what libraries to use. The following build tags are
available:

| Tag | Description |
| ----------- | ------------------------------------------------------- |
| `opencv` | Include `lazywei/go-opencv` in the tests. |
| `imagick` | Include `gographics/imagick` in the tests. |
| `vips` | Include `DAddYE/vips in the tests`. |
| `fastjpeg` | Include `camlistore/fastjpeg in the tests`. |
| `all` | An alias for `opencv imagick fastjpeg vips`. |
| `nopure` | Don't include the Pure Golang packages |
| `noexec` | Don't run the tests that execute other programs. |

The default `go get` without any tags will try the packages that are pure go
and the external programs but not use any non-Go library.

### Benchmark

Im my test scenario all of these tools/packages are unleashed on a directory
containing JPG photo files, all of which have a resolution of 5616x3744 pixels
(aspect ratio 2:1, both landscape and portrait).

For each tool/package and for all files, the total time for loading the
original file, scaling the image to a thumbnail of 150x100 pixels, and writing
it to a new JPG file is measured. In the end, the total runtime for processing
the 10 first files and the average time per file is printed for each
tool/package.

The scenario is run on a Intel(R) Pentium(R) Dual T2390 @ 1.86GHz running
Ubuntu 14.04. Here are the results:

| Table | Time (avg.) | Size (avg.) | Pure Go |
|-----------------------|------------:|------------:|:-------:|
| vipsthumbnail | 0.120s | 0.065% | |
| ImageMagick_thumbnail | 0.326s | 0.242% | |
| vips | 0.339s | 0.100% | |
| magickwand_box | 1.148s | 0.538% | |
| ImageMagick_resize | 2.316s | 0.626% | |
| rez_bilinear | 2.913s | 0.053% | X |
| Nfnt_NearestNeighbor | 3.498s | 0.057% | X |
| imaging_box | 4.734s | 0.057% | X |
| gift_box | 4.746s | 0.057% | X |


--------

Yet another scenario ran by [lazywei](https://github.com/lazywei), 2.5GHz Intel Core i5, Mac OS X 10.9.1:

| Tables | Average time per file |
| -------------------- | ----------------------:|
| magickwand_box | 155.371531ms |
| imaging_Box | 463.459339ms |
| Nfnt_NearestNeighbor | 1.436507946s |
| OpenCv | 97.353041ms |

--------

Yet another scenario ran by [bamiaux](https://github.com/bamiaux), 3.3GHz Intel Core i5, win 7:

| Tables | Average time per file |
| -------------------- | ----------------------:|
| rez_bilinear | 148ms |
| imaging_Box | 243ms |
| Nfnt_NearestNeighbor | 233ms |

--------

A new scenario ran by [nono](https://github.com/nono), 3.4GHz Intel Core i7, Ubuntu 16.10:

| Table | Time (file avg.) | Size (file avg.) | Pure Go |
|--------------------------------|-----------------:|-----------------:|:-------:|
| ImageMagick_thumbnail | 0.057s | 0.361% | |
| vips | 0.070s | 0.260% | |
| epeg | 0.079s | 0.207% | |
| fastjpeg | 0.082s | 0.186% | |
| opencv | 0.110s | 0.597% | |
| vipsthumbnail | 0.115s | 0.441% | |
| GraphicsMagick_thumbnail | 0.172s | 0.427% | |
| magickwand_box | 0.190s | 0.575% | |
| T-REZ | 0.204s | 0.323% | |
| rez_bilinear | 0.349s | 0.140% | X |
| x_image_draw | 0.370s | 0.160% | X |
| imaging_box | 0.439s | 0.146% | X |
| gift_box | 0.440s | 0.146% | X |
| Nfnt_NearestNeighbor | 0.447s | 0.146% | X |
| bild_resize | 0.515s | 0.206% | X |
| ImageMagick_resize | 0.568s | 0.542% | |

--------

So, what is to learn from that? While all of the currently existing
pure-Go-language solutions do a pretty good job in generating good-looking
thumbnails, they are much slower than the veteran dedicated image processing
toolboxes. That is hardly surprising, given that both ImageMagick and
GraphicsMagick have been around for decades and have been optimized to work as
efficient as possible. Go and its image processing packages are still the new
kids on the block, and while they work pretty neat for the occasional tweak of
an image or two, I rather not use them as the default image processor in
[gonagall](http://github.com/fawick/gonagall) yet.

I was surprised to find that GraphicsMagick was slower than ImageMagick in my
test scenario, as I expected it to be exactly the other way around with
GraphicsMagick's fancy multi-processor algorithms.

While the imagick Wrapper is written in Go, it uses CGO bindings of the C
MagickWand API. It outperforms the pure-Go approaches (five times faster than
http://github.com/disintegration/imaging) but it still slower than calling
ImageMagick in an external process. Of the above 1.13 seconds, only around 275
millisecs were used for resizing and saving an individual file, while over 850
ms were used by simply loading the file. I wonder how much optimization can
still be done in the imagick loading routines.

Holy cow! `vipsthumbnail` __is__ blazing fast.

The Best Go Libraries For Benchmarks - Where to discover new Go libraries. (15)