Datastore connectivity for SQL, NoSQL, structured files

# Datastore Connectivity (dsc)

[![Datastore Connectivity library for Go.](https://goreportcard.com/badge/github.com/viant/dsc)](https://goreportcard.com/report/github.com/viant/dsc)
[![GoDoc](https://godoc.org/github.com/viant/dsc?status.svg)](https://godoc.org/github.com/viant/dsc)

This library is compatible with Go 1.10+

Please refer to [`CHANGELOG.md`](CHANGELOG.md) if you encounter breaking changes.

- [Motivation](#Motivation)
- [Usage](#Usage)
- [Prerequisites](#Prerequisites)
- [Installation](#Installation)
- [API Documentaion](#API-Documentation)
- [Tests](#Tests)
- [Examples](#Examples)
- [License](#License)
- [Credits and Acknowledgements](#Credits-and-Acknowledgements)



## Motivation

This library was developed as part of dsunit (Datastore unit testibility library) to provide unified access to SQL, noSQL, 
or any other store that deals with structured data in SQL-ish way.


## Usage:


The following is a very simple example of CRUD operations with dsc

```go
package main

import (
)


func main() {


	config := dsc.NewConfig("mysql", "[user]:[password]@[url]", "user:root,password:dev,url:tcp(127.0.0.1:3306)/mydb?parseTime=true")
	factory := NewManagerFactory()
	manager, err := factory.Create(config)
    if err != nil {
        panic(err.Error())
	}

    // manager := factory.CreateFromURL("file:///etc/myapp/datastore.json")
  
  
    interest := Interest{}
    
    success, err:= manager.ReadSingle(&interest, "SELECT id, name, expiry, category FROM interests WHERE id = ?", []interface{}{id},nil)
	if err != nil {
        panic(err.Error())
	}

    var interests = make([]Interest, 0)
    err:= manager.ReadAll(&interests, "SELECT id, name, expiry, category FROM interests", nil ,nil)
    if err != nil {
        panic(err.Error())
    }

    fmt.Printf("all interests: %v\n", interests)

    interests = []Interest {
        Interest{Name:"Abc", ExpiryTimeInSecond:3600, Category:"xyz"},
        Interest{Name:"Def", ExpiryTimeInSecond:3600, Category:"xyz"},
        Interest{Id:20, Name:"Ghi", ExpiryTimeInSecond:3600, Category:"xyz"},
    }


	inserted, updated, err:= manager.PersistAll(&interests, "interests", nil)
	if err != nil {
        panic(err.Error())
   	}
    deleted, err := manager.DeleteAll(&interests, "interests", nil)
    if err != nil {
        panic(err.Error())
   	}
 	fmt.Printf("Inserted %v, updated: %v, deleted: %v\n", inserted, updated, deleted)
  
}
```

More examples illustrating the use of the API are located in the
[`examples`](examples) directory.

Details about the API are available in the [`docs`](docs) directory.

<a name="Prerequisites"></a>
## Prerequisites

[Go](http://golang.org) version v1.5+ is required.

To install the latest stable version of Go, visit
[http://golang.org/dl/](http://golang.org/dl/)


Target


<a name="Installation"></a>
## Installation:

1. Install Go 1.5+ and setup your environment as [Documented](http://golang.org/doc/code.html#GOPATH) here.
2. Get the client in your ```GOPATH``` : ```go get github.com/viant/dsc```
 * To update the client library: ```go get -u github.com/viant/dsc```


### Some Hints:

 * To run a go program directly: ```go run <filename.go>```
 * to build:  ```go build -o <output> <filename.go>```



<a name="API-Documentation"></a>

## API Documentation

API documentation is available in the [`docs`](docs/README.md) directory.


## Tests

This library is packaged with a number of tests. Tests require Testify library.

Before running the tests, you need to update the dependencies:

    $ go get .

To run all the test cases with race detection:

    $ go test



<a name="Examples"></a>
## Examples

A simple CRUD applications is provided in the [`examples`](examples) directory.


## GoCover

[![GoCover](http://gocover.io/github.com/viant/dsc)](http://gocover.io/github.com/viant/dsc)


<a name="License"></a>
## License

The source code is made available under the terms of the Apache License, Version 2, as stated in the file `LICENSE`.

Individual files may be made available under their own specific license,
all compatible with Apache License, Version 2. Please see individual files for details.


<a name="Credits-and-Acknowledgements"></a>

##  Credits and Acknowledgements

**Library Author:** Adrian Witas

**Contributors:** Sudhakaran Dharmaraj


A simple key-value abstraction to store algebraic and linked-data data types at AWS storage services: AWS DynamoDB and AWS S3

dynamo

Transaction manager with multiple adapters (sql, sqlx, gorm, mongo, ...) controls transaction boundaries

# Go transaction manager

[![Go Reference](https://pkg.go.dev/badge/github.com/avito-tech/go-transaction-manager.svg)](https://pkg.go.dev/github.com/avito-tech/go-transaction-manager/trm/v2)
[![Test Status](https://github.com/avito-tech/go-transaction-manager/actions/workflows/main.yaml/badge.svg)](https://github.com/avito-tech/go-transaction-manager/actions?query=branch%3Amain)
[![Coverage Status](https://coveralls.io/repos/github/avito-tech/go-transaction-manager/badge.svg?branch=main)](https://coveralls.io/github/avito-tech/go-transaction-manager?branch=main)
[![Go Report Card](https://goreportcard.com/badge/github.com/avito-tech/go-transaction-manager)](https://goreportcard.com/report/github.com/avito-tech/go-transaction-manager/)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Transaction manager is an abstraction to coordinate database transaction boundaries.

Easiest way to get the perfect repository.

## Supported implementations

* [database/sql](https://pkg.go.dev/database/sql), [docs](https://pkg.go.dev/github.com/avito-tech/go-transaction-manager/drivers/sql/v2) (
  Go 1.13)
* [jmoiron/sqlx](https://github.com/jmoiron/sqlx), [docs](https://pkg.go.dev/github.com/avito-tech/go-transaction-manager/drivers/sqlx/v2) (
  Go 1.13)
* [gorm](https://github.com/go-gorm/gorm), [docs](https://pkg.go.dev/github.com/avito-tech/go-transaction-manager/drivers/gorm/v2) (
  Go 1.18)
* [mongo-go-driver](https://github.com/mongodb/mongo-go-driver), [docs](https://pkg.go.dev/github.com/avito-tech/go-transaction-manager/drivers/mongo/v2) (
  Go 1.13)
* [go-redis/redis](https://github.com/go-redis/redis), [docs](https://pkg.go.dev/github.com/avito-tech/go-transaction-manager/drivers/goredis8/v2) (
  Go 1.17)
* [pgx_v4](https://github.com/jackc/pgx/tree/v4), [docs](https://pkg.go.dev/github.com/avito-tech/go-transaction-manager/drivers/pgxv4/v2) (
  Go 1.16)
* [pgx_v5](https://github.com/jackc/pgx), [docs](https://pkg.go.dev/github.com/avito-tech/go-transaction-manager/drivers/pgxv5/v2) (
  Go 1.19)

## Installation

```bash
go get github.com/avito-tech/go-transaction-manager/trm/v2
```

To install some support database use `go get github.com/avito-tech/go-transaction-manager/drivers/{name}`.

For example `go get github.com/avito-tech/go-transaction-manager/drivers/sqlx/v2`.

### Backwards Compatibility

The library is compatible with the most recent two versions of Go.
Compatibility beyond that is not guaranteed.

The critical bugs are firstly solved for the most recent two Golang versions and then for older ones if it is simple.

#### Disclaimer: Keep your dependencies up to date, even indirect ones.

`go get -u && go mod tidy` helps you.

**Note**: The go-transaction-manager uses some old dependencies to support backwards compatibility for old versions of Go.

## Usage

**To use multiple transactions from different databases**, you need to set CtxKey in [Settings](trm/settings.go)
by [WithCtxKey](trm/settings/option.go) ([docs](https://pkg.go.dev/github.com/avito-tech/go-transaction-manager/trm/v2)).

**For nested transactions with different transaction managers**, you need to use [ChainedMW](trm/manager/chain.go) ([docs](https://pkg.go.dev/github.com/avito-tech/go-transaction-manager/trm/v2/manager)).

**To skip a transaction rollback due to an error, use [ErrSkip](manager.go#L20) or [Skippable](manager.go#L24)**

### Explanation of the approach [English](https://www.youtube.com/watch?v=aRsea6FFAyA), Russian [article](https://habr.com/ru/companies/avito/articles/727168/) and [youtube](https://www.youtube.com/watch?v=fcdckM5sUxA).

### Examples with an ideal *repository* and nested transactions.

* [database/sql](drivers/sql/example_test.go)
* [jmoiron/sqlx](drivers/sqlx/example_test.go)
* [gorm](drivers/gorm/example_test.go)
* [mongo-go-driver](drivers/mongo/example_test.go)
* [go-redis/redis](drivers/goredis8/example_test.go)
* [pgx_v4](drivers/pgxv4/example_test.go)
* [pgx_v5](drivers/pgxv5/example_test.go)

Below is an example how to start usage.

```go
package main

import (
	"context"
	"fmt"

	"github.com/jmoiron/sqlx"
	_ "github.com/mattn/go-sqlite3"

	trmsqlx "github.com/avito-tech/go-transaction-manager/drivers/sqlx/v2"
	"github.com/avito-tech/go-transaction-manager/trm/v2/manager"
)

func main() {
	db, err := sqlx.Open("sqlite3", "file:test?mode=memory")
	checkErr(err)
	defer db.Close()

	sqlStmt := `CREATE TABLE IF NOT EXISTS user (user_id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, username TEXT);`
	_, err = db.Exec(sqlStmt)
	checkErr(err, sqlStmt)

	r := newRepo(db, trmsqlx.DefaultCtxGetter)
	ctx := context.Background()
	trManager := manager.Must(trmsqlx.NewDefaultFactory(db))
	u := &user{Username: "username"}

	err = trManager.Do(ctx, func(ctx context.Context) error {
		checkErr(r.Save(ctx, u))

		// example of nested transactions
		return trManager.Do(ctx, func(ctx context.Context) error {
			u.Username = "new_username"
			return r.Save(ctx, u)
		})
	})
	checkErr(err)

	userFromDB, err := r.GetByID(ctx, u.ID)
	checkErr(err)

	fmt.Println(userFromDB)
}

func checkErr(err error, args ...interface{}) {
	if err != nil {
		panic(fmt.Sprint(append([]interface{}{err}, args...)...))
	}
}

type repo struct {
	db     *sqlx.DB
	getter *trmsqlx.CtxGetter
}

func newRepo(db *sqlx.DB, c *trmsqlx.CtxGetter) *repo {
	return &repo{db: db, getter: c}
}

type user struct {
	ID       int64  `db:"user_id"`
	Username string `db:"username"`
}

func (r *repo) GetByID(ctx context.Context, id int64) (*user, error) {
	query := "SELECT * FROM user WHERE user_id = ?;"
	u := user{}

	return &u, r.getter.DefaultTrOrDB(ctx, r.db).GetContext(ctx, &u, r.db.Rebind(query), id)
}

func (r *repo) Save(ctx context.Context, u *user) error {
	query := `UPDATE user SET username = :username WHERE user_id = :user_id;`
	if u.ID == 0 {
		query = `INSERT INTO user (username) VALUES (:username);`
	}

	res, err := sqlx.NamedExecContext(ctx, r.getter.DefaultTrOrDB(ctx, r.db), r.db.Rebind(query), u)
	if err != nil {
		return err
	} else if u.ID != 0 {
		return nil
	} else if u.ID, err = res.LastInsertId(); err != nil {
		return err
	}

	return err
}
```

## Benchmark

[Comparing](trm/internal/benchmark/with_or_without_trm/README.md) examples with and without trm.

## Contribution

### Requirements

- [golangci-lint](https://golangci-lint.run/welcome/install/)
- [make](https://www.gnu.org/software/make/#download)

### Local Running

* To install all dependencies use `make go.mod.tidy` or `make go.mod.vendor`.
* To run all tests use `make test` or `make test.with_real_db` for integration tests.

To run database by docker, there is [docker-compose.yaml](trm/drivers/test/docker-compose.yaml).
```bash
docker compose -f trm/drivers/test/docker-compose.yaml up
```

For full GitHub Actions run, you can use [act](https://github.com/nektos/act).

#### Running old go versions 

To stop Golang upgrading set environment variable `GOTOOLCHAIN=local` .

```sh
go install go1.16 # or older version
go1.16 install
```

Use `-mod=readonly` to prevent `go.mod` modification.

To run tests
```
go1.16 test -race -mod=readonly ./...
```

### How to bump up Golang version in CI/CD

1. Changes in [.github/workflows/main.yaml](.github/workflows/main.yaml).
   1. Add all old version of Go in `go-version:` for `tests-units` job.
   2. Update `go-version:` on current version of Go for `lint` and `tests-integration` jobs.
2. Update build tags by replacing `build go1.xx` on new version.


### Resolve problems with old version of dependencies

To build `go.mod` compatible for old version use `go mod tidy -compat=1.13` ([docs](https://go.dev/ref/mod#go-mod-tidy)).

However, `--compat` doesn't always work correct and we need to set some library versions manually.

1. `go get go.uber.org/multierr@v1.9.0` in [trm](trm), [sql](drivers/sql), [sqlx](drivers/sqlx).
2. `go get github.com/mattn/go-sqlite3@v1.14.14` in [trm](trm), [sql](drivers/sql), [sqlx](drivers/sqlx).
3. `go get github.com/stretchr/testify@v1.8.2` in [trm](trm), [sql](drivers/sql), [sqlx](drivers/sqlx), [goredis8](drivers/goredis8), [mongo](drivers/mongo).
4. `go get github.com/jackc/pgconn@v1.14.2` in [pgxv4](drivers/pgxv4). Golang version was bumped up from 1.12 to 1.17 in pgconn v1.14.3.
5. `go get golang.org/x/text@v0.13.0` in [pgxv4](drivers/pgxv4).

Also, you can change version of single dependency.

```bash
go mod edit -require=github.com/avito-tech/go-transaction-manager/trm/v2@v2.0.2
go mod tidy

```

## Authors

- Ilia Sergunin [github/maranqz](https://github.com/maranqz)


go-transaction-manager

Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)

gokv

Graph database with support for multiple backends

<div align="center">
  <a href="https://github.com/cayleygraph/cayley">
    <img width="200" src="https://github.com/cayleygraph/branding/raw/master/cayley_bottom.png" alt="Cayley">
  </a>
</div>

![Tests](https://github.com/cayleygraph/cayley/actions/workflows/build-and-release.yml/badge.svg)

Cayley is an open-source database for [Linked Data](https://www.w3.org/standards/semanticweb/data). It is inspired by the graph database behind Google's [Knowledge Graph](https://en.wikipedia.org/wiki/Knowledge_Graph) (formerly [Freebase](https://en.wikipedia.org/wiki/Freebase_(database))).

[![Get it from the Snap Store](https://snapcraft.io/static/images/badges/en/snap-store-white.svg)](https://snapcraft.io/cayley)

## [Documentation](https://cayley.gitbook.io/cayley/)

## Features

- Built-in query editor, visualizer and REPL
- Multiple query languages:
  - [Gizmo](./docs/gizmoapi.md): query language inspired by [Gremlin](https://tinkerpop.apache.org/gremlin.html)
  - [GraphQL](./docs/graphql.md)-inspired query language
  - [MQL](./docs/mql.md): simplified version for [Freebase](https://en.wikipedia.org/wiki/Freebase_(database)) fans
- Modular: easy to connect to your favorite programming languages and back-end stores
- Production ready: well tested and used by various companies for their production workloads
- Fast: optimized specifically for usage in applications

### Performance

Rough performance testing shows that, on 2014 consumer hardware and an average disk, 134m quads in LevelDB is no problem and a multi-hop intersection query -- films starring X and Y -- takes ~150ms.

## Community

- Website: [cayley.io](https://cayley.io)
- Slack: [cayleygraph.slack.com](https://cayleygraph.slack.com) -- Invite [here](https://cayley-slackin.herokuapp.com/)
- Discourse list: [discourse.cayley.io](https://discourse.cayley.io) (Also acts as mailing list, enable mailing list mode)
- Twitter: [@cayleygraph](https://twitter.com/cayleygraph)
- Involvement: [Contribute](./docs/contributing.md)


The Best Go Libraries For Database Drivers - Interfaces to Multiple Backends (5)

dsc

dynamo

go-transaction-manager

gokv

cayley