Handles address representation, validation and formatting

# address [![Build](https://github.com/bojanz/address/actions/workflows/build.yml/badge.svg)](https://github.com/bojanz/address/actions/workflows/build.yml) [![Coverage Status](https://coveralls.io/repos/github/bojanz/address/badge.svg?branch=master)](https://coveralls.io/github/bojanz/address?branch=master) [![Go Report Card](https://goreportcard.com/badge/github.com/bojanz/address)](https://goreportcard.com/report/github.com/bojanz/address) [![PkgGoDev](https://pkg.go.dev/badge/github.com/bojanz/address)](https://pkg.go.dev/github.com/bojanz/address)

Handles address representation, validation and formatting.

Inspired by Google's [libaddressinput](https://github.com/google/libaddressinput).

Backstory: https://bojanz.github.io/address-handling-go/

## Features

1. Address struct.
2. Address formats for ~200 countries.
3. Regions for ~50 countries, with local names where relevant (e.g: Okinawa / 沖縄県).
4. Country list, powered by CLDR v48.
5. HTML formatter.
6. HTTP handler for serving address formats and regions as JSON: only ~14kb gzipped!

## Address struct

Represents an address as commonly handled by web applications and APIs.

```go
type Address struct {
	Line1 string
	Line2 string
	Line3 string
	// Sublocality is the neighborhood/suburb/district.
	Sublocality string
	// Locality is the city/village/post town.
	Locality string
	// Region is the state/province/prefecture.
	// An ISO code is used when available.
	Region string
	// PostalCode is the postal/zip/pin code.
	PostalCode string
	// CountryCode is the two-letter code as defined by CLDR.
	CountryCode string
}
```

Recipient fields such as FirstName/LastName are not included since they are usually
present on the parent (Contact/Customer/User) struct. This allows the package
to avoid being opinionated about name handling.

There are three line fields, matching the HTML5 autocomplete spec and many shipping APIs.
This leaves enough space for specifying an organization and department, house or hotel
name, and other similar "care of" use cases. When mapping to an API that only has two address lines,
Line3 can be appended to Line2, separated by a comma.

## Address formats

The following information [is available](https://github.com/bojanz/address/blob/master/formats.go#L6):

- Which fields are used, and in which order.
- Which fields are required.
- Labels for the sublocality, locality, region and postal code fields.
- Regular expression pattern for validating postal codes.
- Regions and how to display them in an address.

Certain countries (e.g. China, Japan, Russia, Ukraine) have region names defined in both Latin and local scripts. The script is selected based on locale. For example, the "ru" locale will use Russian regions in Cyrilic, while "ru-Latn" and other locales will use the Latin version.

[Helpers](https://github.com/bojanz/address/blob/master/address.go#L61) are provided for validating required fields, regions, postal codes.

Format data was generated from Google's [Address Data](https://chromium-i18n.appspot.com/ssl-address) but isn't
automatically regenerated, to allow the community to submit their own corrections directly to the package.

## Countries

The country list is auto-generated from CLDR. 
Updating to the latest CLDR release is always one `go generate` away.

Most software uses the CLDR country list instead of the ISO one because the CLDR country names match their colloquial usage more closely (e.g. "Russia" instead of "Russian Federation"). 

To reduce the size of the included data, this package only includes country names in English.
Translated country names can be fetched on the frontend via [Intl.DisplayNames](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/DisplayNames). Alternatively, one can plug in [x/text/language/display](https://pkg.go.dev/golang.org/x/text/language/display) by setting a custom CountryMapper on the formatter.

## Formatter

Displays an address as HTML, using the country's address format.

The wrapper element ("p") and class ("address") can be configured.
The country name can be omitted, for the use case where all addresses belong to the same country. 

```go
addr := address.Address{
 Line1: "1098 Alta Ave",
 Locality: "Mountain View",
 Region: "CA",
 PostalCode: "94043",
 CountryCode: "US",
}
locale := address.NewLocale("en")
formatter := address.NewFormatter(locale)
output := formatter.Format(addr)
// Output:
// 
// 1098 Alta Ave 
// Mountain View, CA 94043 
// United States
// 

addr = address.Address{
 Line1: "幸福中路",
 Sublocality: "新城区",
 Locality: "西安市",
 Region: "SN",
 PostalCode: "710043",
 CountryCode: "CN",
}
locale := address.NewLocale("zh")
formatter := address.NewFormatter(locale)
formatter.NoCountry = true
formatter.WrapperElement = "div"
formatter.WrapperClass = "postal-address"
output := formatter.Format(addr)
// Output:
// <div class="postal-address" translate="no">
// 710043 
// 陕西省西安市新城区 
// 幸福中路
// </div>
```

address

A general purpose application that aligns text

# align
_A general purpose application that aligns text_

[![GoDoc](https://img.shields.io/badge/api-reference-blue.svg?style=flat-square)](https://godoc.org/github.com/Guitarbum722/align) 
[![Build Status](https://travis-ci.org/Guitarbum722/align.svg?branch=master)](https://travis-ci.org/Guitarbum722/align)
[![Go Report Card](https://goreportcard.com/badge/github.com/Guitarbum722/align)](https://goreportcard.com/report/github.com/Guitarbum722/align)
[![Coverage Status](https://coveralls.io/repos/github/Guitarbum722/align/badge.svg?branch=master)](https://coveralls.io/github/Guitarbum722/align?branch=master)

The focus of this application is to provide a fast, efficient, and useful tool for aligning text.
Its creation is the result of inspiration from several other amazing alignment tools, namely [`column`](http://man7.org/linux/man-pages/man1/column.1.html) or the Sublime Text plugin [`AlignTab`](https://github.com/randy3k/AlignTab).

See the [Wiki](https://github.com/Guitarbum722/align/wiki) for usage examples!

### Included

* A simple yet useful CLI with options to specify your delimiter, input and output files, etc.
* Align by any string as your delimiter or separator, not just a single character.
* If your separator string is contained within the data itself, it can be escaped by specifying a text qualifier.
* Right, Center, or Left justification of each field.

_Why?_

Sometimes, it's just easier to align a CSV (or delimited file) by its delimiter and view the columns in your plain text editor (which saves you from opening Excel!).

Another use is to align blocks of code by `=` or `=>`, etc.

### Install

```sh
$ go get github.com/Guitarbum722/align
$ make install

$ # build all binaries
$ make release
```

### Usage - CLI examples

```
Usage: align [-h] [-f] [-o] [-q] [-s] [-d] [-a] [-c] [-i] [-p]
Options:
 -h | --help help
 -f input file. If not specified, pipe input to stdin
 -o output file. (default: stdout)
 -q text qualifier (if applicable)
 -s delimiter (default: ',')
 -d output delimiter (defaults to the value of sep)
 -a <left>, <right>, <center> justification (default: left)
 -c output specific fields (default: all fields)
 -i override justification by column number (e.g. 2:center,5:right)
 -p extra padding surrounding delimiter
```

_Specify your input file, output file, delimiter._
*You can also pipe input to stdin (if the `-f` option is provided, it will take precedence over Stdin)*
If no `-o` option is provided, stdout will be used.

```sh
$ align -f input_file.csv -o output_file.csv

$ align -f input_file.csv -o 

$ cat awesome.csv | align
```

Do you have rows with a different number of fields? This might be more common with code, but `align` doesn't care!

```
$ echo "field1|field2\nValue1|Value2\nCoolValue1|CoolValue2|CoolValue3" | align -s \|
field1 | field2
Value1 | Value2
CoolValue1 | CoolValue2 | CoolValue3
```

Column filtering (specifiy output fields and optionally override the justification of the output fields). This might be useful if you would like to display a dollar amount or number field differently. The specified fields are indexed at 1.

```sh
# output fields 1,3,5 justified 'right'
$ cat file.csv | align -a right -c 1,3,5

# output fields 1,2,3,7,8 with default justification (left) except for field 7, which is right justified
$ cat file.csv | align -c 1,2,3,7,8 -i 1:right,7:center

#output all fields by default, with right justification, with overridden justification on certain columns
$ cat file.csv | align -a right -i 1:center,5:left
```

Support for worldwide characters.
```
first , last , middle , email
paul , danny , かど , や製油
```

It is perfectly acceptable to even use emojis as your input/output delimiters.
```
first 😮 last 😮 email
Hector 😮 Gonzalez 😮 h.g@nothing.com
```

Add additional padding if desired with the `-p` flag. Default is 1 space, and 0 will output with no additional padding. If the value supplied is less than 0, then the behavior will be as if it were set to 0 and no padding will be applied.
```
# padding of 4 spaces surrounding the delimiter.
align -p 4
```

### Contributions

If you have suggestions or discover a bug, please open an issue. If you think you can make the fix, please use the Fork / Pull Request on your feature branch approach.

align

Formats and parses numeric byte values (10K, 2M, 3G, etc.)

# Gommon [![GoDoc](http://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)](http://godoc.org/github.com/labstack/gommon) [![Coverage Status](http://img.shields.io/coveralls/labstack/gommon.svg?style=flat-square)](https://coveralls.io/r/labstack/gommon)

Common packages for Go
- [Bytes](https://github.com/labstack/gommon/tree/master/bytes) - Format/parse bytes.
- [Color](https://github.com/labstack/gommon/tree/master/color) - Style terminal text.
- [Log](https://github.com/labstack/gommon/tree/master/log) - Simple logging.


bytes

Fixed-width text formatting (encoder/decoder with reflection)

# fixedwidth [![GoDoc](https://godoc.org/github.com/ianlopshire/go-fixedwidth?status.svg)](http://godoc.org/github.com/ianlopshire/go-fixedwidth) [![Report card](https://goreportcard.com/badge/github.com/ianlopshire/go-fixedwidth)](https://goreportcard.com/report/github.com/ianlopshire/go-fixedwidth) [![Go Cover](http://gocover.io/_badge/github.com/ianlopshire/go-fixedwidth)](http://gocover.io/github.com/ianlopshire/go-fixedwidth)

Package fixedwidth provides encoding and decoding for fixed-width formatted Data.

`go get github.com/ianlopshire/go-fixedwidth`

## Usage

### Struct Tags

The struct tag schema schema used by fixedwidth is: `fixed:"{startPos},{endPos},[{alignment},[{padChar}]]"`[1](#f1).

The `startPos` and `endPos` arguments control the position within a line. `startPos` and `endPos` must both be positive integers greater than 0. Positions start at 1. The interval is inclusive. 

The `alignment` argument controls the alignment of the value within it's interval. The valid options are `default`[2](#f2), `right`, `left`, and `none`. The `alignment` is optional and can be omitted.

The `padChar` argument controls the character that will be used to pad any empty characters in the interval after writing the value. The default padding character is a space. The `padChar` is optional and can be omitted.

Fields without tags are ignored.

### Encode
```go
// define some data to encode
people := []struct {
 ID int `fixed:"1,5"`
 FirstName string `fixed:"6,15"`
 LastName string `fixed:"16,25"`
 Grade float64 `fixed:"26,30"`
 Age uint `fixed:"31,33"`
 Alive bool `fixed:"34,39"`
}{
 {1, "Ian", "Lopshire", 99.5, 20, true},
}

data, err := Marshal(people)
if err != nil {
 log.Fatal(err)
}
fmt.Printf("%s", data)
// Output:
// 1 Ian Lopshire 99.5020 true
```

### Decode
```go
// define the format
var people []struct {
 ID int `fixed:"1,5"`
 FirstName string `fixed:"6,15"`
 LastName string `fixed:"16,25"`
 Grade float64 `fixed:"26,30"`
 Age uint `fixed:"31,33"`
 Alive bool `fixed:"34,39"`
 Github bool `fixed:"40,41"`
}

// define some fixed-with data to parse
data := []byte("" +
 "1 Ian Lopshire 99.50 20 false f" + "\n" +
 "2 John Doe 89.50 21 true t" + "\n" +
 "3 Jane Doe 79.50 22 false F" + "\n" +
 "4 Ann Carraway 79.59 23 false T" + "\n")

err := Unmarshal(data, &people)
if err != nil {
 log.Fatal(err)
}

fmt.Printf("%+v\n", people[0])
fmt.Printf("%+v\n", people[1])
fmt.Printf("%+v\n", people[2])
fmt.Printf("%+v\n", people[3])
// Output:
//{ID:1 FirstName:Ian LastName:Lopshire Grade:99.5 Age:20 Alive:false Github:false}
//{ID:2 FirstName:John LastName:Doe Grade:89.5 Age:21 Alive:true Github:true}
//{ID:3 FirstName:Jane LastName:Doe Grade:79.5 Age:22 Alive:false Github:false}
//{ID:4 FirstName:Ann LastName:Carraway Grade:79.59 Age:23 Alive:false Github:true}
```

It is also possible to read data incrementally

```go
decoder := fixedwidth.NewDecoder(bytes.NewReader(data))
for {
 var element myStruct
 err := decoder.Decode(&element)
 if err == io.EOF {
 break
 }
 if err != nil {
 log.Fatal(err)
 }
 handle(element)
}
```

### UTF-8, Codepoints, and Multibyte Characters

fixedwidth supports encoding and decoding fixed-width data where indices are expressed in
unicode codepoints and not raw bytes. The data must be UTF-8 encoded.

```go
decoder := fixedwidth.NewDecoder(strings.NewReader(data))
decoder.SetUseCodepointIndices(true)
// Decode as usual now
```


```go
buff := new(bytes.Buffer)
encoder := fixedwidth.NewEncoder(buff)
encoder.SetUseCodepointIndices(true)
// Encode as usual now
```

### Alignment Behavior

| Alignment | Encoding | Decoding |
| --------- | -------- | -------- |
| `default` | Field is left aligned | The padding character is trimmed from both right and left of value |
| `left` | Field is left aligned | The padding character is trimmed from right of value |
| `right` | Field is right aligned | The padding character is trimmed from left of value |
| `none` | Field is left aligned | The padding character is not trimmed from value. Useful for nested structs. |

## Notes
1. `{}` indicates an argument. `[]` indicates and optional segment [^](#a1)
2. The `default` alignment is similar to `left` but has slightly different behavior required to maintain backwards compatibility [^](#a2) 

## Licence
MIT

go-fixedwidth

Formatters for time, numbers, and memory size to human readable format

go-humanize

Easily pretty-print your tabular data with Go

# Gotabulate - Easily pretty-print tabular data
[![GoDoc](https://godoc.org/github.com/bndr/gotabulate?status.svg)](https://godoc.org/github.com/bndr/gotabulate)
[![Build Status](https://travis-ci.org/bndr/gotabulate.svg?branch=master)](https://travis-ci.org/bndr/gotabulate)

## Summary

Go-Tabulate - Generic Go Library for easy pretty-printing of tabular data. 

## Installation

    go get github.com/bndr/gotabulate

## Description

Supported data types:
- 2D Array of `Int`, `Int64`, `Float64`, `String`, `interface{}`
- Map of `String`, `interface{}` (Keys will be used as header)

## Usage

```go
// Create Some Fake Rows
row_1 := []interface{}{"john", 20, "ready"}
row_2 := []interface{}{"bndr", 23, "ready"}

// Create an object from 2D interface array
t := gotabulate.Create([][]interface{}{row_1, row_2})

// Set the Headers (optional)
t.SetHeaders([]string{"age", "status"})

// Set the Empty String (optional)
t.SetEmptyString("None")

// Set Align (Optional)
t.SetAlign("right")

// Print the result: grid, or simple
fmt.Println(t.Render("grid"))

+---------+--------+-----------+
|         |    age |    status |
+=========+========+===========+
|    john |     20 |     ready |
+---------+--------+-----------+
|    bndr |     23 |     ready |
+---------+--------+-----------+
```

## Example with String

```go
// Some Strings
string_1 := []string{"TV", "1000$", "Sold"}
string_2 := []string{"PC", "50%", "on Hold"}

// Create Object
tabulate := gotabulate.Create([][]string{string_1, string_2})

// Set Headers
tabulate.SetHeaders([]string{"Type", "Cost", "Status"})

// Render
fmt.Println(tabulate.Render("simple"))

---------  ----------  ------------
    Type        Cost        Status
---------  ----------  ------------
      TV       1000$          Sold

      PC         50%       on Hold
---------  ----------  ------------
```

## Example with String Wrapping

```go
tabulate := gotabulate.Create([][]string{[]string{"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus laoreet vestibulum pretium. Nulla et ornare elit. Cum sociis natoque penatibus et magnis",
	"Vivamus laoreet vestibulum pretium. Nulla et ornare elit. Cum sociis natoque penatibus et magnis", "zzLorem ipsum", " test", "test"}, []string{"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus laoreet vestibulum pretium. Nulla et ornare elit. Cum sociis natoque penatibus et magnis",
	"Vivamus laoreet vestibulum pretium. Nulla et ornare elit. Cum sociis natoque penatibus et magnis", "zzLorem ipsum", " test", "test"}, STRING_ARRAY, []string{"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus laoreet vestibulum pretium. Nulla et ornare elit. Cum sociis natoque penatibus et magnis",
	"Vivamus laoreet vestibulum pretium. Nulla et ornare elit. Cum sociis natoque penatibus et magnis", "zzLorem ipsum", " test", "test"}, STRING_ARRAY})

tabulate.SetHeaders([]string{"Header 1", "header 2", "header 3", "header 4"})
// Set Max Cell Size
tabulate.SetMaxCellSize(16)

// Turn On String Wrapping
tabulate.SetWrapStrings(true)

// Render the table
fmt.Println(tabulate.Render("grid"))

+---------------------+---------------------+----------------+-------------+-------------+
|                     |            Header 1 |       header 2 |    header 3 |    header 4 |
+=====================+=====================+================+=============+=============+
|    Lorem ipsum dolo |    Vivamus laoreet  |    Lorem ipsum |        test |        test |
|    r sit amet, cons |    vestibulum preti |                |             |             |
|    ectetur adipisci |    um. Nulla et orn |                |             |             |
|    ng elit. Vivamus |    are elit. Cum so |                |             |             |
|     laoreet vestibu |    ciis natoque pen |                |             |             |
|    lum pretium. Nul |    atibus et magnis |                |             |             |
|    la et ornare eli |                     |                |             |             |
|    t. Cum sociis na |                     |                |             |             |
|    toque penatibus  |                     |                |             |             |
|           et magnis |                     |                |             |             |
+---------------------+---------------------+----------------+-------------+-------------+
|    Lorem ipsum dolo |    Vivamus laoreet  |    Lorem ipsum |        test |        test |
|    r sit amet, cons |    vestibulum preti |                |             |             |
|    ectetur adipisci |    um. Nulla et orn |                |             |             |
|    ng elit. Vivamus |    are elit. Cum so |                |             |             |
|     laoreet vestibu |    ciis natoque pen |                |             |             |
|    lum pretium. Nul |    atibus et magnis |                |             |             |
|    la et ornare eli |                     |                |             |             |
|    t. Cum sociis na |                     |                |             |             |
|    toque penatibus  |                     |                |             |             |
|           et magnis |                     |                |             |             |
+---------------------+---------------------+----------------+-------------+-------------+
|         test string |       test string 2 |           test |         row |        bndr |
+---------------------+---------------------+----------------+-------------+-------------+
|    Lorem ipsum dolo |    Vivamus laoreet  |    Lorem ipsum |        test |        test |
|    r sit amet, cons |    vestibulum preti |                |             |             |
|    ectetur adipisci |    um. Nulla et orn |                |             |             |
|    ng elit. Vivamus |    are elit. Cum so |                |             |             |
|     laoreet vestibu |    ciis natoque pen |                |             |             |
|    lum pretium. Nul |    atibus et magnis |                |             |             |
|    la et ornare eli |                     |                |             |             |
|    t. Cum sociis na |                     |                |             |             |
|    toque penatibus  |                     |                |             |             |
|           et magnis |                     |                |             |             |
+---------------------+---------------------+----------------+-------------+-------------+
|         test string |       test string 2 |           test |         row |        bndr |
+---------------------+---------------------+----------------+-------------+-------------+
```
## Examples

```go
t := gotabulate.Create([][]string{STRING_ARRAY, STRING_ARRAY})

t.SetHeaders(HEADERS) // If not headers are set, the first row will be used.

t.SetEmptyString("None") // Set what will be printed in the empty cell

rendered_string := t.Render("simple") // Render() will return a string

Simple Table
----------------------  ----------------------  ----------------------  -------------  -------------
             Header 1                Header 2                Header 3       Header 4       Header 5 
----------------------  ----------------------  ----------------------  -------------  -------------
          test string           test string 2                    test            row           bndr 

          test string           test string 2                    test            row           bndr 

    4th element empty       4th element empty       4th element empty           None           None 
----------------------  ----------------------  ----------------------  -------------  -------------

Grid Table (Align Right)
+-------------+-------------+-------------+-------------+-------------+
|    Header 1 |    Header 2 |    Header 3 |    Header 4 |    Header 5 |
+=============+=============+=============+=============+=============+
|       10.01 |      12.002 |      -123.5 |    20.00005 |        1.01 |
+-------------+-------------+-------------+-------------+-------------+
|       10.01 |      12.002 |      -123.5 |    20.00005 |        1.01 |
+-------------+-------------+-------------+-------------+-------------+
|       10.01 |      12.002 |      -123.5 |    20.00005 |        None |
+-------------+-------------+-------------+-------------+-------------+

Padded Headers:
+----------------------+----------------------+----------------------+-------------+-------------+
|                      |             Header 1 |             header 2 |    header 3 |    header 4 |
+======================+======================+======================+=============+=============+
|          test string |        test string 2 |                 test |         row |        bndr |
+----------------------+----------------------+----------------------+-------------+-------------+
|          test string |        test string 2 |                 test |         row |        bndr |
+----------------------+----------------------+----------------------+-------------+-------------+
|    4th element empty |    4th element empty |    4th element empty |        None |        None |
+----------------------+----------------------+----------------------+-------------+-------------+

Align Center:
+-------------+-------------+-------------+-------------+-------------+
|   Header 1  |   Header 2  |   Header 3  |   Header 4  |   Header 5  |
+=============+=============+=============+=============+=============+
|    10.01    |    12.002   |    -123.5   |   20.00005  |     1.01    |
+-------------+-------------+-------------+-------------+-------------+
|    10.01    |    12.002   |    -123.5   |   20.00005  |     1.01    |
+-------------+-------------+-------------+-------------+-------------+
|    10.01    |    12.002   |    -123.5   |   20.00005  |     1.01    |
+-------------+-------------+-------------+-------------+-------------+

Align Left:
+-------------+-------------+-------------+-------------+-------------+
| Header 1    | Header 2    | Header 3    | Header 4    | Header 5    |
+=============+=============+=============+=============+=============+
| 10.01       | 12.002      | -123.5      | 20.00005    | 1.01        |
+-------------+-------------+-------------+-------------+-------------+
| 10.01       | 12.002      | -123.5      | 20.00005    | 1.01        |
+-------------+-------------+-------------+-------------+-------------+
| 10.01       | 12.002      | -123.5      | 20.00005    | 1.01        |
+-------------+-------------+-------------+-------------+-------------+
```

### Status

Beta version. There may be edge cases that I have missed, so if your tables don't render properly please open up an issue. 

## Contribute

All Contributions are welcome. The todo list is on the bottom of this README. Feel free to send a pull request.

## License

Apache License 2.0

## TODO

- [ ] Add more examples
- [ ] Better Documentation
- [ ] Implement more data table formats
- [ ] Decimal point alignment for floats

## Acknowledgement

Inspired by Python package [tabulate](https://pypi.python.org/pypi/tabulate)


gotabulate

Convert data from SQL databases or document formats like CSV or Excel into formats such as JSON, Excel, CSV, HTML, Markdown, XML, and YAML

[![Go Reference](https://pkg.go.dev/badge/github.com/neilotoole/sq.svg)](https://pkg.go.dev/github.com/neilotoole/sq)
[![Go Report Card](https://goreportcard.com/badge/neilotoole/sq)](https://goreportcard.com/report/neilotoole/sq)
[![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/neilotoole/sq/blob/master/LICENSE)
![Main pipeline](https://github.com/neilotoole/sq/actions/workflows/main.yml/badge.svg)

# sq data wrangler

`sq` is a command line tool that provides jq-style access to
structured data sources: SQL databases, or document formats like CSV or Excel.
It is the lovechild of sql+jq.

![sq](.images/splash.png)

`sq` executes jq-like [queries](https://sq.io/docs/query), or database-native [SQL](https://sq.io/docs/cmd/sql/).
It can [join](https://sq.io/docs/query#cross-source-joins) across sources: join a CSV file to a Postgres table, or
MySQL with Excel.

`sq` outputs to a multitude of [formats](https://sq.io/docs/output#formats)
including [JSON](https://sq.io/docs/output#json),
[Excel](https://sq.io/docs/output#xlsx), [CSV](https://sq.io/docs/output#csv),
[HTML](https://sq.io/docs/output#html), [Markdown](https://sq.io/docs/output#markdown)
and [XML](https://sq.io/docs/output#xml), and can [insert](https://sq.io/docs/output#insert) query
results directly to a SQL database.

`sq` can also [inspect](https://sq.io/docs/inspect) sources to view metadata about the source structure (tables,
columns, size). You can use [`sq diff`](https://sq.io/docs/diff) to compare tables, or
entire databases. `sq` has commands for common database operations to
[copy](https://sq.io/docs/cmd/tbl-copy), [truncate](https://sq.io/docs/cmd/tbl-truncate),
and [drop](https://sq.io/docs/cmd/tbl-drop) tables.

Find out more at [sq.io](https://sq.io).

> [!TIP]
> The rest of this doc is mainly for `sq` end users and first-timers. Contributors (bug reports, feature requests, pull requests),
> see [CONTRIBUTING.md](./CONTRIBUTING.md).

## Drivers

`sq` knows how to deal with a data source type via a [driver](https://sq.io/docs/drivers)
implementation. To view the installed/supported drivers:

```shell
$ sq driver ls
DRIVER      DESCRIPTION
sqlite3     SQLite
postgres    PostgreSQL
sqlserver   Microsoft SQL Server / Azure SQL Edge
mysql       MySQL
clickhouse  ClickHouse
csv         Comma-Separated Values
tsv         Tab-Separated Values
json        JSON
jsona       JSON Array: LF-delimited JSON arrays
jsonl       JSON Lines: LF-delimited JSON objects
xlsx        Microsoft Excel XLSX
```

> [!NOTE]
> ClickHouse Driver support is currently in beta. Full details of support can be
> found in the [ClickHouse README](drivers/clickhouse/README.md).

## Install

### macOS

```shell
brew install sq
```

> [!IMPORTANT]
> `sq` is now a [core brew formula](https://formulae.brew.sh/formula/sq#default). Previously, `sq` was available via `brew install neilotoole/sq/sq`. If you have installed `sq` this way, you should uninstall it (`brew uninstall neilotoole/sq/sq`) before installing the new formula via `brew install sq`.

### Linux

```shell
/bin/sh -c "$(curl -fsSL https://sq.io/install.sh)"
```

### Windows

```shell
scoop bucket add sq https://github.com/neilotoole/sq
scoop install sq
```

### Go

```shell
go install github.com/neilotoole/sq
```

### Docker

The [`ghcr.io/neilotoole/sq`](https://github.com/neilotoole/sq/pkgs/container/sq)
image is preloaded with `sq` and a handful of related tools like `jq`.

#### Local

```shell
# Shell into a one-time container.
$ docker run -it ghcr.io/neilotoole/sq zsh

# Start detached (background) container named "sq-shell".
$ docker run -d --name sq-shell ghcr.io/neilotoole/sq
# Shell into that container.
$ docker exec -it sq-shell zsh
```

#### Kubernetes

Running `sq` in a Kubernetes environment is useful for DB migrations,
as well as general data wrangling.

```shell
# Start pod named "sq-shell".
$ kubectl run sq-shell --image ghcr.io/neilotoole/sq
# Shell into the pod.
$ kubectl exec -it sq-shell -- zsh
```

See other [install options](https://sq.io/docs/install/).

## Contributing

See [CONTRIBUTING.md](./CONTRIBUTING.md).

## Overview

Use `sq help` to see command help. Docs are over at [sq.io](https://sq.io).
Read the [overview](https://sq.io/docs/overview/), and
[tutorial](https://sq.io/docs/tutorial/). The [cookbook](https://sq.io/docs/cookbook/) has
recipes for common tasks, and the [query guide](https://sq.io/docs/query) covers `sq`'s query language.

The major concept is: `sq` operates on data sources, which are treated as SQL databases (even if the
source is really a CSV or XLSX file etc.).

In a nutshell, you [`sq add`](https://sq.io/docs/cmd/add) a source (giving it a [`handle`](https://sq.io/docs/concepts#handle)), and then execute commands against the
source.

### Sources

Initially there are no [sources](https://sq.io/docs/source).

```shell
$ sq ls

```

Let's [add](https://sq.io/docs/cmd/add) a source. First we'll add a [SQLite](https://sq.io/docs/drivers/sqlite)
database, but this could also be [Postgres](https://sq.io/docs/drivers/postgres),
[SQL Server](https://sq.io/docs/drivers/sqlserver) etc., or a document source such [Excel](https://sq.io/docs/drivers/xlsx) or
[CSV](https://sq.io/docs/drivers/csv).

Download the sample DB, and `sq add` the source.

```shell
$ wget https://sq.io/testdata/sakila.db

$ sq add ./sakila.db
@sakila  sqlite3  sakila.db

$ sq ls -v
HANDLE   ACTIVE  DRIVER   LOCATION                         OPTIONS
@sakila  active  sqlite3  sqlite3:///Users/demo/sakila.db

$ sq ping @sakila
@sakila       1ms  pong

$ sq src
@sakila  sqlite3  sakila.db
```

The [`sq ping`](https://sq.io/docs/cmd/ping) command simply pings the source
to verify that it's available.

[`sq src`](https://sq.io/docs/cmd/src) lists the [_active source_](https://sq.io/docs/source#active-source), which in our
case is `@sakila`.
You can change the active source using `sq src @other_src`.
When there's an active source specified, you can usually omit the handle from `sq` commands.
Thus you could instead do:

```shell
$ sq ping
@sakila  1ms  pong
```

> [!TIP]
> Document sources such as CSV or Excel can be added from the local filesystem, or
> from an HTTP URL.
>
> ```shell
> $ sq add https://acme.s3.amazonaws.com/sales.csv
> ```
>
> ![sq inspect remote](./.images/sq_inspect_remote_s3.png)
> See the [sources](https://sq.io/docs/source#download) docs for more.

### Query

Fundamentally, `sq` is for querying data. The jq-style syntax is covered in
detail in the [query guide](https://sq.io/docs/query).

![sq query where slq](./.images/sq_query_where_slq.png)

The above query selected some rows from the `actor` table. You could also
use [native SQL](https://sq.io/docs/cmd/sql), e.g.:

![sq query where sql](./.images/sq_query_where_sql.png)

But we're flying a bit blind here: how did we know about the `actor` table?

### Inspect

[`sq inspect`](https://sq.io/docs/inspect) is your friend.

![sq inspect](./.images/sq_inspect_source_text.png)

Use [`sq inspect -v`](https://sq.io/docs/cmd/inspect) to see more detail.
Or use [`-j`](https://sq.io/docs/output#json) to get JSON output:

![sq inspect -j](./.images/sq_inspect_sakila_sqlite_json.png)

Combine `sq inspect` with [jq](https://jqlang.github.io/jq/) for some useful capabilities.
Here's how to [list](https://sq.io/docs/cookbook#list-table-names)
all the table names in the active source:

```shell
$ sq inspect -j | jq -r '.tables[] | .name'
actor
address
category
city
country
customer
[...]
```

And here's how you
could [export](https://sq.io/docs/cookbook#export-all-table-data-to-csv) each table
to a CSV file:

```shell
$ sq inspect -j | jq -r '.tables[] | .name' | xargs -I % sq .% --csv --output %.csv
$ ls
actor.csv     city.csv      customer_list.csv  film_category.csv  inventory.csv  rental.csv                  staff.csv
address.csv   country.csv   film.csv           film_list.csv      language.csv   sales_by_film_category.csv  staff_list.csv
category.csv  customer.csv  film_actor.csv     film_text.csv      payment.csv    sales_by_store.csv          store.csv
```

Note that you can also inspect an individual table:

![sq inspect actor verbose](./.images/sq_inspect_actor_verbose.png)

Read more about [`sq inspect`](https://sq.io/docs/inspect).

### Diff

Use [`sq diff`](https://sq.io/docs/diff) to compare metadata, or row data, for sources, or individual tables.

The default behavior is to diff table schema and row counts. Table row data is not compared in this mode.

![sq diff](.images/sq_diff_src_default.png)

Use [`--data`](https://sq.io/docs/diff#--data) to compare row data.

![sq diff data](.images/sq_diff_table_data.png)

There are many more options available. See the [diff docs](https://sq.io/docs/diff).

### Insert query results

`sq` query results can be [output](https://sq.io/docs/output) in various formats
([`text`](https://sq.io/docs/output#text),
[`json`](https://sq.io/docs/output#json),
[`csv`](https://sq.io/docs/output#csv), etc.). Those results can also be "outputted"
as an [_insert_](https://sq.io/docs/output#insert) into a database table.

That is, you can use `sq` to insert results from a Postgres query into a MySQL table,
or copy an Excel worksheet into a SQLite table, or a push a CSV file into
a SQL Server table etc.

> [!TIP]
> If you want to copy a table inside the same (database) source,
> use [`sq tbl copy`](https://sq.io/docs/cmd/tbl-copy) instead, which uses the database's native table copy functionality.

Here we query a CSV file, and insert the results into a Postgres table.

![sq query insert inspect](./.images/sq_query_insert_inspect.png)

### Cross-source joins

`sq` can perform the usual [joins](https://sq.io/docs/query#joins). Here's how you would
join tables `actor`, `film_actor`, and `film`:

```shell
$ sq '.actor | join(.film_actor, .actor_id) | join(.film, .film_id) | .first_name, .last_name, .title'
```

But `sq` can also join across data sources. That is, you can join an Excel worksheet with a
Postgres table, or join a CSV file with MySQL, and so on.

This example joins a Postgres database, an Excel worksheet, and a CSV file.

![sq join multi source](./.images/sq_join_multi_source.png)

Read more about cross-source joins in the [query guide](https://sq.io/docs/query#joins).

### Table commands

`sq` provides several handy commands for working with tables:
[`tbl copy`](/docs/cmd/tbl-copy), [`tbl truncate`](/docs/cmd/tbl-truncate)
and [`tbl drop`](/docs/cmd/tbl-drop).
Note that these commands work directly
against SQL database sources, using their native SQL commands.

```shell
$ sq tbl copy .actor .actor_copy
Copied table: @sakila.actor --> @sakila.actor_copy (200 rows copied)

$ sq tbl truncate .actor_copy
Truncated 200 rows from @sakila.actor_copy

$ sq tbl drop .actor_copy
Dropped table @sakila.actor_copy
```

### UNIX pipes

For file-based sources (such as CSV or XLSX), you can `sq add` the source file,
but you can also pipe it:

```shell
$ cat ./example.xlsx | sq .Sheet1
```

Similarly, you can inspect:

```shell
$ cat ./example.xlsx | sq inspect
```

## Output formats

`sq` has many [output formats](https://sq.io/docs/output):

- `--text`: [Text](https://sq.io/docs/output#text)
- `--json`: [JSON](https://sq.io/docs/output#json)
- `--jsona`: [JSON Array](https://sq.io/docs/output#jsona)
- `--jsonl`: [JSON Lines](https://sq.io/docs/output#jsonl)
- `--csv` / `--tsv` : [CSV](https://sq.io/docs/output#csv) / [TSV](https://sq.io/docs/output#tsv)
- `--xlsx`: [XLSX](https://sq.io/docs/output#xlsx) (Microsoft Excel)
- `--html`: [HTML](https://sq.io/docs/output#html)
- `--xml`: [XML](https://sq.io/docs/output#xml)
- `--yaml`: [YAML](https://sq.io/docs/output#yaml)
- `--markdown`: [Markdown](https://sq.io/docs/output#markdown)
- `--raw`: [Raw](https://sq.io/docs/output#raw) (bytes)

## CHANGELOG

See [CHANGELOG.md](./CHANGELOG.md).

## Acknowledgements

- Thanks to [Diego Souza](https://github.com/diegosouza) for creating
  the [Arch Linux package](https://aur.archlinux.org/packages/sq-bin), and [`@icp`](https://github.com/icp1994)
  for creating the [Void Linux package](https://github.com/void-linux/void-packages/blob/master/srcpkgs/sq/template).
- Much inspiration is owed to [jq](https://jqlang.github.io/jq/).
- See [`go.mod`](https://github.com/neilotoole/sq/blob/master/go.mod) for a list of third-party
  packages.
- `sq` imports a bunch of [`usql`](https://github.com/xo/usql) functionality.
- Additionally, `sq` incorporates modified versions of:
 - [`olekukonko/tablewriter`](https://github.com/olekukonko/tablewriter)
 - [`segmentio/encoding`](https://github.com/segmentio/encoding) for JSON encoding.
- The [_Sakila_](https://dev.mysql.com/doc/sakila/en/) example databases were lifted
  from [jOOQ](https://github.com/jooq/jooq), which in turn owe their heritage to earlier work on
  Sakila.
- Date rendering via [`ncruces/go-strftime`](https://github.com/ncruces/go-strftime).
- A modified version [`dolmen-go/contextio`](https://github.com/dolmen-go/contextio) is
  incorporated into the codebase.
- [`djherbis/buffer`](https://github.com/djherbis/buffer) is used for caching.
- A forked version of [`nightlyone/lockfile`](https://github.com/nightlyone/lockfile) is incorporated.
- The human-friendly `text` log format handler is a fork of [`lmittmann/tint`](https://github.com/lmittmann/tint).

## Similar, related, or noteworthy projects

- [`usql`](https://github.com/xo/usql)
- [`textql`](https://github.com/dinedal/textql)
- [`golang-migrate`](https://github.com/golang-migrate/migrate)
- [`octosql`](https://github.com/cube2222/octosql)
- [`rq`](https://github.com/dflemstr/rq)
- [`miller`](https://github.com/johnkerl/miller)
- [`jsoncolor`](https://github.com/neilotoole/jsoncolor) is a JSON colorizer created for `sq`.
- [`streamcache`](https://github.com/neilotoole/streamcache) is a Go in-memory byte cache mechanism created for `sq`.
- [`fifomu`](https://github.com/neilotoole/fifomu) is a FIFO mutex, used by `streamcache`, and thus upstream in `sq`.
- [`tailbuf`](https://github.com/neilotoole/tailbuf) is a fixed-size object tail buffer created for `sq`.
- [`oncecache`](https://github.com/neilotoole/oncecache) is an in-memory object cache created for `sq`.


Wraps text at end of lines. Implementation of `textwrap` module from Python

# Text Wrap

This is a port of Python's "textwrap" module for Go. Well, sort of...

# Limitations

This modules (at least for now) is not wrapping on whitespaces and
right after hyphens in compound words, as it is customary in
English. That said, `break_on_hyphens` and `break_long_words` are not
yet supported.

Also `fix_sentence_endings` is not supported as well for now, which
doesn't work reliably in Python anyways (since it requires two spaces
and other conditions nobody cares of).

The implementation for hyphens support is planned however, while
`fix_sentence_endings` is not (but! your PRs are welcome and free to
implement it).

# Usage

The usage is quite similar as in Python:

```go
import (
	"fmt"
	"github.com/isbm/textwrap"
)

...

text := "Your very long text here"
wrapper := textwrap.NewTextWrap() // Defaults to 70
fmt.Println(wrapper.Fill(text))      // Returns string

// Get each line
for idx, line := range wrapper.Wrap(text) {
	fmt.Println(idx, line)
}

```

De-dent is also implemented and works exactly the same as in Python:

```go
multilineText := `
    There is some multiline text
  with different identation
      everywhere. So it will be
    aligned to the minimal.
`

// This will remove two leading spaces from each line
fmt.Println(wrapper.Dedent(multilineText))

```


# Configuration

You can setup wrapper object constructor the following way (given values are its
defaults, so you can change it to whatever you want):

```go
wrapper := textwrap.NewTextWrap().
	SetNewLine("\n").
	SetWidth(70),
	SetTabSpacesWidth(4).
	SetDropWhitespace(true).
	SetInitialIndent("").
	SetReplaceWhitespace(true)
```

Have fun.

# Bonus Functions

While it is possible to do it differently, this module also gives you
string whitespace trimming for _only_ leading whitespace (`TrimLeft`) or _only_
trailing (`TrimRight`), as contrary to `strings.TrimSpace` that trims
everything.

The whitespace is the same as defined in Python's `strings.whitespace`.



The Best Go Libraries For Text Processing - Formatters (8)