📚 igor - Awesome Go Library for Database

Go Gopher mascot for igor

Abstraction layer for PostgreSQL that supports advanced functionality and uses gorm-like syntax.

🏷️ Database
📂 Libraries for building and using SQL.
⭐ 125 stars
View on GitHub 🔗

Detailed Description of igor

igor

igor is an abstraction layer for PostgreSQL, written in Go. Igor syntax is (almost) compatible with GORM.

GoDoc Build Status

When to use igor

You should use igor when your DBMS is PostgreSQL and you want to place an abstraction layer on top of it and do CRUD operations in a smart, easy, secure and fast way.

Thus with igor you do not create a new schema. In general igor does not support DDL (you can do it with the Raw and Exec, but there are not method created ad-hoc for this purpose).

What igor does

  • Always uses prepared statements: no sql injection and good performance.
  • Supports transactions
  • Supports PostgreSQL JSON and JSONB types with igor.JSON
  • Supports PostgreSQL LISTEN/NOTIFY
  • Uses a GORM like syntax
  • Uses the same logic in insertion and update: handle default values in a coherent manner
  • Uses GORM models and conventions (partially, see Differences)
  • Exploits PostgreSQL RETURNING statement to update models fields with the updated values (even when changed on db side; e.g. when having a default value)
  • Automatically handle reserved keywords when used as a table name or fields. Does not quote every field (that's not recommended) but only the ones conflicting with a reserved keyword.

What igor is not

  • An ORM (and thus a complete GORM replacement):
    • Does not support associations
    • Does not support callbacks
    • Does not have any specific method for data migration and DDL operations
    • Does not support soft delete

Install

go get -u github.com/galeone/igor

GORM compatible

igor uses the same syntax of GORM. Thus in a great number of cases you can replace GORM with igor by only changing the import path.

Warning: igor is not a complete GORM replacement. See the Differences.

Model definition

Models are the same used in GORM.

The main differences are:

  • Igor does not handle associations. Thus, if you have a field that refers to another table, disable it with the annotation sql:"-" (see the code below).
  • Every model must implement the igor.DBTable interface. Therefore every model must have the method TableName() string, that returns the table name associated with the model.
  • Every model must explicit the primary key field (using the tag igor:"primary_key").
  • Since igor does not deal with DDL, sql:"type:<type>" is ignored.

Like:

type User struct {
	Counter uint64 `igor:"primary_key"`
    Username string
    Password string
    Name string
    Surname string
    Profile Profile `sql:"-"`
}

type (User) TableName() string {
    return "users"
}

Array support

igor supports PostgreSQL fields natively, without the need to use the pg.Array type - you can use just plain structs.

type NestMe struct {
	ID            int64 `igor:"primary_key"`
	SliceOfString []string
	SliceOfInt64  []int64
}

This structure maps this table definition:

CREATE TABLE nest_table(
    id bigserial not null PRIMARY KEY,
    slice_of_string text[] not null,
    slice_of_int64 bigint[] not null
)

Nested types support

igor allows you to embed types, and overwrite fields of the inner type. In particular, you can add the sql decorator (or change type, potentially).

type NestMe struct {
	ID            int64 `igor:"primary_key"`
	OverwriteMe   int64
	SliceOfString []string
	SliceOfInt64  []int64
}

type NestTable struct {
	NestMe
	OverwriteMe int64 `sql:"-"`
}
func (NestTable) TableName() string {
	return "nest_table"
}

The NestTable type disables the SQL generation for the field OverwriteMe that's present in the embedded type NestMe.

Methods

Connect

import "github.com/galeone/igor"

func main() {
  db, err := igor.Connect("user=galeone dbname=igor sslmode=disable")
}

Log

See: Logger.

Model

Model(DBModel) sets the table name for the current query

var logged bool
var counter uint64

db.Model(User{}).Select("login(?, ?) AS logged, counter", username, password).Where("LOWER(username) = ?", username).Scan(&logged, &counter);

it generates:

SELECT login($1, $2) AS logged, counter FROM users WHERE LOWER(username) = $3 ;

Joins

Joins append the join string to the current model

type Post struct {
	Hpid    uint64    `igor:"primary_key"`
	From    uint64
	To      uint64
	Pid     uint64    `sql:"default:0"`
	Message string
	Time    time.Time `sql:"default:(now() at time zone 'utc')"`
	Lang    string
	News    bool
	Closed  bool
}

type UserPost struct {
	Post
}

func (UserPost) TableName() string {
    return "posts"
}

users := new(User).TableName()
posts := new(UserPost).TableName()

var userPosts []UserPost
db.Model(UserPost{}).Order("hpid DESC").
    Joins("JOIN "+users+" ON "+users+".counter = "+posts+".to").
    Where(`"to" = ?`, user.Counter).Scan(&userPost)

it generates:

SELECT posts.hpid,posts."from",posts."to",posts.pid,posts.message,posts."time",posts.lang,posts.news,posts.closed
FROM posts
JOIN users ON users.counter = posts.to
WHERE "to" = $1

Table

Table appends the table string to FROM. It has the same behavior of Model, but passing the table name directly as a string

See example in Joins

CTE

CTE allows to define a Common Table Expression that precedes the query.

Warning: use it with the Table method.

var usernames []string
var ids []uint64 // fill them - not the usage of = any since this is converted to a pq.Array

db.CTE(`WITH full_users_id AS (
SELECT counter FROM users WHERE name = ? AND counter = any(?))`, "Paolo", ids).
Table("full_users_id as fui").
Select("username").
Joins("JOIN users ON fui.counter = users.counter").Scan(&usernames)

it generates:

WITH full_users_id AS (
	SELECT counter FROM users WHERE name = $1 AND counter = any($2)
)
SELECT username FROM full_users_id as fui JOIN users ON fui.counter = users.counter;

Select

Select sets the fields to retrieve. Appends fields to SELECT (See example in Model).

When select is not specified, every field is selected in the Model order (See example in Joins).

Warning: calling Select using parameters without type is allowed only if the stored procedure on the DBMS define the type.

Eg: if we have a function on PostgreSQL that accepts two parameters like

login(_username text, _pass text, OUT ret boolean) RETURNS boolean

we can call this function in that way

db.Select('login(?,?)', username, password)

But, if the DBMS can't infer the parameters (in every other case except the one previous mentioned), we must make parameters type explicit.

This is due to the use of prepared statements.

db.Select("?::int, ?::int, ?::int", 1, 2, 3)

Where

Where works with DBModels or strings.

When using a DBModel, if the primary key fields is not blank, the query will generate a where clause in the form:

Thus:

db.Model(UserPost{}).Where(&UserPost{Hpid: 1, From:1, To:1})

it generates:

SELECT posts.hpid,posts."from",posts."to",posts.pid,posts.message,posts."time",posts.lang,posts.news,posts.closed
FROM posts
WHERE posts.hpid = $1

Ignoring values that are not primary keys.

If the primary key field is blank, generates the where clause ANDing the conditions:

db.Model(UserPost{}).Where(&UserPost{From:1, To:1})

The conditions will be:

WHERE posts.from = $1 AND posts.to = $2

When using a string, you can use the ? as placeholder for parameters substitution. Thus

db.Model(UserPost{}).Where(`"to" = ?`, user.Counter)

it generates:

SELECT posts.hpid,posts."from",posts."to",posts.pid,posts.message,posts."time",posts.lang,posts.news,posts.closed
FROM posts
WHERE "to" = $1

Where supports slices as well:

db.Model(UserPost{}).Where(`"to" IN (?) OR "from" = ?`, []uint64{1,2,3,4,6}, 88)

it generates:

SELECT posts.hpid,posts."from",posts."to",posts.pid,posts.message,posts."time",posts.lang,posts.news,posts.closed
FROM posts
WHERE "to" IN ($1,$2,$3,$4,$5) OR "from" = $6

Create

Create INSERT a new row into the table specified by the DBModel.

Create handles default values using the following rules:

If a field is blank and has a default value and this default value is the Go Zero value for that field, igor does not generate the query part associated with the insertion of that fields (let the DBMS handle the default value generation).

If a field is blank and has a default value that's different from the Go Zero value for that filed, insert the specified default value.

Create exploits the RETURNING clause of PostgreSQL to fetch the new row and update the DBModel passed as argument.

In that way igor always have the up-to-date fields of DBModel.

post := &UserPost{
    From: 1,
    To: 1,
    Pid: 10,
    Message: "hi",
    Lang: "en",
}
db.Create(post)

it generates:

INSERT INTO posts("from","to",pid,message,lang) VALUES ($1,$2,$3,$4,$5)  RETURNING posts.hpid,posts."from",posts."to",posts.pid,posts.message,posts."time",posts.lang,posts.news,posts.closed;

The resulting row (the result of RETURNING) is used as a source for the Scan method, having the DBModel as argument.

Thus, in the example, the variable post.Time has the (now() at time zone 'utc') evaluation result value.

Delete

See Delete

Updates

Updates uses the same logic of Create (thus the default value handling is the same).

The only difference is that Updates UPDATE rows.

Update tries to infer the table name from the DBModel passed as argument if a Where clause has not been specified. Otherwise uses the Where clause to generate the WHERE part and the Model to generate the field = $n part.

var user User
db.First(&user, 1) // handle errors
user.Username = "username changed"

db.Updates(&user)

it generates:

UPDATE users SET users.username = "username changed" WHERE users.counter = 1 RETURNING users.counter,users.last,users.notify_story,users.private,users.lang,users.username,users.email,users.name,users.surname,users.gender,users.birth_date,users.board_lang,users.timezone,users.viewonline,users.registration_time

The RETURNING clause is handled in the same manner of Create.

Pluck

Pluck fills the slice with the query result. It calls Scan internally, thus the slice can be a slice of structures or a slice of simple types.

It panics if slice is not a slice or the query is not well formulated.

type Blacklist struct {
	From       uint64
	To         uint64
	Motivation string
	Time       time.Time `sql:"default:(now() at time zone 'utc')"`
	Counter    uint64    `igor:"primary_key"`
}

func (Blacklist) TableName() string {
	return "blacklist"
}

var blacklist []uint64
db.Model(Blacklist{}).Where(&Blacklist{From: user.Counter}).Pluck(`"to"`, &blacklist)

it generates

SELECT "to" FROM blacklist WHERE blacklist."from" = $1

Count

Count sets the query result to be count(*) and scan the result into value.

var count int
db.Model(Blacklist{}).Where(&Blacklist{From: user.Counter}).Count(&count

it generates:

SELECT COUNT(*) FROM blacklist WHERE blacklist."from" = $1

First

See First

Scan

See Scan and Find methods

Raw

Prepares and executes a raw query, the results is available for the Scan method.

See Scan and Find methods

Exec

Prepares and executes a raw query, the results is discarded. Useful when you don't need the query result or the operation have no result.

tx := db.Begin()
tx.Exec("DROP TABLE IF EXISTS users")
tx.Exec(`CREATE TABLE users (
	counter bigint NOT NULL,
	last timestamp without time zone DEFAULT timezone('utc'::text, now()) NOT NULL,
	notify_story jsonb,
	private boolean DEFAULT false NOT NULL,
	lang character varying(2) DEFAULT 'en'::character varying NOT NULL,
	username character varying(90) NOT NULL,
	password character varying(60) NOT NULL,
	name character varying(60) NOT NULL,
	surname character varying(60) NOT NULL,
	email character varying(350) NOT NULL,
	gender boolean NOT NULL,
	birth_date date NOT NULL,
	board_lang character varying(2) DEFAULT 'en'::character varying NOT NULL,
	timezone character varying(35) DEFAULT 'UTC'::character varying NOT NULL,
	viewonline boolean DEFAULT true NOT NULL,
	remote_addr inet DEFAULT '127.0.0.1'::inet NOT NULL,
	http_user_agent text DEFAULT ''::text NOT NULL,
	registration_time timestamp(0) with time zone DEFAULT now() NOT NULL
	)`)
tx.Commit()

Exec does not use prepared statements if there are no parameters to replace in the query. This make it possible to use a single call to Exec to execute multiple statements ;-terminated. e.g.

tx.Exec(`DROP TABLE IF EXISTS users;
        CREATE TABLE users (
        counter bigint NOT NULL,
        ...
    )`)
tx.Commit()

Where

Where builds the WHERE clause.

If a primary key is present in the struct passed as argument only that field is used.

user.Counter = 2
user.Name = "paolo"
db.Select("username").Where(&user)

it generates:

SELECT username FROM users WHERE users.counter = $1

because Counter is the primary key.

If the primary key is blank every non empty field is and-end.

user.Counter = 0 // 0 is a blank primary key

it generates

SELECT username FROM users WHERE users.name = $1

You can use a string to build the where clause and pass parameters if needed.

db.Model(User{}).Select("username").Where("counter IN (?) AND name ILIKE ?",[]uint64{1,2,4,5}, "nino")

it generates:

SELECT username FROM users WHERE counter in ($1,$2,$3,$4) AND name ILIKE $5

If a where condition can't be generated, Where panics

Limit

Limit sets the LIMIT value to the query

Offset

Offset sets the OFFSET value to the query

Order

Order sets the ORDER BY value to the query

DB

DB returns the current *sql.DB. It panics if called during a transaction

Begin

Begin initialize a transaction. It panics if begin has been already called.

Il returns a *igor.Database, thus you can use every other *Database method on the returned value.

tx := db.Begin()

Commit

Commit commits the transaction. It panics if the transaction is not started (you have to call Begin before)

tx.Create(&user)
tx.Commit()
// Now you can use the db variable again

Rollback

Rollback rollbacks the transaction. It panics if the transaction is not started (you have to call Begin before

if e := tx.Create(&user); e != nil {
    tx.Rollback()
} else {
    tx.Commit()
}
// Now you can use the db variable again

Listen

Listen executes LISTEN channel. Uses f to handle received notifications on channel.

if e := db.Listen("notification_channel", func(payload ...string) {
    if len(payload) > 0 {
        pl := strings.Join(payload, ",")
        fmt.Println("Received notification on channel notification_channel, having payload: " + pl)
    } else {
        fmt.Println("Received notification on channel notification_channel without payload")
    }
}); e != nil {
    // handle error
}

Unlisten

Unlisten executesUNLISTEN channel. Unregister function f, that was registered with Listen(channel ,f).

e := db.Unlisten("notification_channel")
// handle error

You can unlisten from every channel calling db.Unlisten("*") or using the method UnlistenAll

UnlistenAll

UnlistenAll executes UNLISTEN *. Thus do not receive any notification from any channel.

Notify

With Notify you can send a notification with or without payload on a channel.

e = db.Notify("notification_channel") // empty payload
e = db.Notify("notification_channel", "payload 1", "payload 2", "test")

When sending a payload, the strings are joined together. Therefore the payload sent with previous call to Notify is: payload 1, payload 2, test

Differences

Select and Where call order

In GORM, you can execute

db.Model(User{}).Select("username")

db.Select("username").Model(User{})

and achieve the same result.

In igor this is not possible. You must call Model before Select.

Thus always use:

db.Model(User{}).Select("username")

The reason is that igor generates queries in the form SELECT table.field1, table.filed2 FROM table [WHERE] RETURNING table.field1, table.filed2.

In order to avoid ambiguities when using Joins, the RETURNING part of the query must be in the form table.field1, table.filed2, ..., and table is the TableName() result of the DBModel passed as Model argument.

Models

Igor models are the same as GORM models (except that you have to use the igor tag field to define the primary key). The sql tag field is used to define default value and column value. Eg:

type Test struct {
    ID      uint64 `igor:"primary_key column:id_column"`
    Time    time.Time `sql:"default:(now() at time zone 'utc')"`
}

The other main difference is that igor models require the implementation of the DBModel interface.

In GORM, you can optionally define the TableName method on your Model. With igor this is mandatory.

This constraint gives to igor the ability to generate conditions (like the WHERE or INSERT or UPDATE part of the query) that have a counter part on DB size for sure.

If a type does not implement the DBModel interface your program will not compile (and thus you can easily find the error and fix it). Otherwise igor could generate a wrong query and we're trying to avoid that.

Open method

Since igor is PostgreSQL only, the gorm.Open method has been replaced with

Connect(connectionString string) (*Database, error)

Logger

There's no db.LogMode(bool) method in igor. If you want to log the prepared statements, you have to manually set a logger for igor.

logger := log.New(os.Stdout, "query-logger: ", log.LUTC)
db.Log(logger)

If you want to disable the logger, set it to nil

db.Log(nil)

Privacy: you'll never see the values of the variables, but only the prepared statement and the PostgreSQL placeholders. Respect your user privacy, do not log user input (like credentials).

Methods return value

In GORM, every method (even the ones that execute queries) returns a *DB.

In igor:

  • methods that execute queries returns error
  • methods that build the query returns *Database, thus you can chain the methods (GORM-like) and build the query.

Scan and Find methods

In GORM, Scan method is used to scan query results into a struct. The Find method is almost the same.

In igor:

  • Scan method executes the SELECT query. Thus return an error if Scan fails (see the previous section).

    Scan handle every type. You can scan query results in:

    • slice of struct .Scan(&sliceOfStruct)
    • single struct .Scan(&singleStruct)
    • single value .Scan(&integerType)
    • a comma separated list of values (because Scan is a variadic arguments function) .Scan(&firstInteger, &firstString, &secondInteger, &floatDestination)

  • Find method does not exists, is completely replaced by Scan.

Scan

In addiction to the previous section, there's another difference between GORM ad igor.

Scan method do not scans the selected fields into results using the selected fields name, but uses the order (to increase the performance).

Thus, having:

type Conversation struct {
	From   string    `json:"from"`
	Time   time.Time `json:"time"`
	ToRead bool      `json:"toRead"`
}

var convList []Conversation
err := Db().Raw(`SELECT DISTINCT otherid, MAX(times) as "time", to_read FROM (
    (SELECT MAX("time") AS times, "from" as otherid, to_read FROM pms WHERE "to" = ? GROUP BY "from", to_read)
    UNION
    (SELECT MAX("time") AS times, "to" as otherid, FALSE AS to_read FROM pms WHERE "from" = ? GROUP BY "to", to_read)
) AS tmp GROUP BY otherid, to_read ORDER BY to_read DESC, "time" DESC`, user.Counter, user.Counter).Scan(&convList)

Do not cause any problem, but if we change the SELECT clause, inverting the order, like

query := "SELECT DISTINCT otherid, to_read, MAX(times) as time " +
...

Scan will fail because it will try to Scan the boolean value in second position to_read, into the time.Time field of the Conversation structure.

Delete

In GORM, if you do not specify a primary key or a where clause (or if the value of the primary key is blank) the generated query will be

DELETE FROM <table>

That will delete everything from your table.

In igor this is not possible.

You must specify a Where clause or pass to Delete a non empty model that will be used to build the where clause.

db.Delete(&UserPost{}) // this panics

post := UserPost{
    Hpid: 10,
    From: 1,
}

db.Delete(&post)
//generates DELETE FROM posts WHERE hpid = $1, because hpid is a primary key

db.Where(&post).Delete(&UserPost{}) // ^ generates the same query

db.Delete(&UserPost{From:1,To:1})
// generates: DELETE FROM posts WHERE "from" = $1 AND "to" = $2

First

In GORM First is used to get the first record, with or without a second parameter that is the primary key value.

In igor this is not possible. First works only with 2 parameter.

  • DBModel: that's the model you want to fill
  • key interface{} that's the primary key value, that must be of the same type of the DBModel primary key.
var user User
db.First(&user, uint64(1))

db.First(&user, "1") // panics, because "1" is not of the same type of user.Counter (uint64)

it generates:

SELECT users.counter,users.last,users.notify_story,users.private,users.lang,users.username,users.email,users.name,users.surname,users.gender,users.birth_date,users.board_lang,users.timezone,users.viewonline,users.registration_time
FROM users
WHERE users.counter = $1

Other

Every other GORM method is not implemented.

JSON and JSONB support

Igor supports PostgreSQL JSON and JSONB types natively.

Just define the field in the DBModel with the type igor.JSON. After that, you can work with JSON in the following way:

user := createUser()

var ns igor.JSON = make(igor.JSON) // use it like a map[string]interface{}

ns["0"] = struct {
    From    uint64 `json:from`
    To      uint64 `json:to`
    Message string `json:message`
}{
    From:    1,
    To:      1,
    Message: "hi bob",
}
ns["numbers"] = 1
ns["test"] = 2

user.NotifyStory = ns

if e = db.Updates(&user); e != nil {
    t.Errorf("updates should work but got: %s\n", e.Error())
}

// To use JSON with json, use:
// printableJSON, _ := json.Marshal(user.NotifyStory)
// fmt.Printf("%s\n", printableJSON)

var nsNew igor.JSON
if e = db.Model(User{}).Select("notify_story").Where(&user).Scan(&nsNew); e != nil {
    t.Errorf("Problem scanning into igor.JSON: %s\n", e.Error())
}

LISTEN / NOTIFY support

PostgreSQL give us a beautiful method to avoid polling the DBMS, using a simple publish/subscribe model over database connections (read more on the docs).

Igor gives you the ability to generate notification and subscribe to notifications sent over a channel, using the methods Listen and Notify.

Below there's a working example:

count := 0
if e = db.Listen("notification_without_payload", func(payload ...string) {
    count++
    t.Log("Received notification on channel: notification_without_payload\n")
}); e != nil {
    t.Fatalf("Unable to listen on channel: %s\n", e.Error())
}

for i := 0; i < 4; i++ {
    if e = db.Notify("notification_without_payload"); e != nil {
        t.Fatalf("Unable to send notification: %s\n", e.Error())
    }
}

// wait some time to handle all notifications, because are asynchronous
time.Sleep(100 * time.Millisecond)
if count != 4 {
    t.Errorf("Expected to receive 4 notifications, but counted only: %d\n", count)
}

// listen on an opened channel should fail
if e = db.Listen("notification_without_payload", func(payload ...string) {}); e == nil {
    t.Errorf("Listen on an opened channel should fail, but succeeded\n")
}

// Handle payload

// listen on more channels, with payload
count = 0
if e = db.Listen("np", func(payload ...string) {
    count++
    t.Logf("channel np: received payload: %s\n", payload)
}); e != nil {
    t.Fatalf("Unable to listen on channel: %s\n", e.Error())
}

// test sending payload with notify
for i := 0; i < 4; i++ {
    if e = db.Notify("np", strconv.Itoa(i)+" payload"); e != nil {
        t.Fatalf("Unable to send notification with payload: %s\n", e.Error())
    }
}

// wait some time to handle all notifications
time.Sleep(100 * time.Millisecond)
if count != 4 {
    t.Errorf("Expected to receive 4 notifications, but counted only: %d\n", count)
}

// test unlisten
if e = db.Unlisten("notification_without_payload"); e != nil {
    t.Errorf("Unable to unlisten from notification_without_payload, got: %s\n", e.Error())
}

// test UnlistenAll
if e = db.UnlistenAll(); e != nil {
    t.Errorf("Unable to unlistenAll, got: %s\n", e.Error())
}

Contributing

Do you want to add some new method to improve GORM compatibility or add some new method to improve igor?

Feel free to contribute via Pull Request.

Testing

To test igor, you must create a igor user on PostgreSQL and make it own the igor database. On Archlinux, with postgres as the PostgreSQL superuser this can be achieved by:

createuser -U postgres igor
createdb -U postgres igor igor
psql -U postgres -d igor -c "GRANT USAGE, CREATE ON SCHEMA public TO igor;"

You can run tests with the usual command:

go test

License

Copyright 2016-2023 Paolo Galeone. All right reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

About the author

Feel free to contact me (you can find my email address and other ways to contact me in my GitHub profile page).

← Back to Database - Libraries for building and using SQL.