![afero logo-sm](https://cloud.githubusercontent.com/assets/173412/11490338/d50e16dc-97a5-11e5-8b12-019a300d0fcb.png)

A FileSystem Abstraction System for Go

[![Test](https://github.com/spf13/afero/actions/workflows/test.yml/badge.svg)](https://github.com/spf13/afero/actions/workflows/test.yml) [![GoDoc](https://godoc.org/github.com/spf13/afero?status.svg)](https://godoc.org/github.com/spf13/afero) [![Join the chat at https://gitter.im/spf13/afero](https://badges.gitter.im/Dev%20Chat.svg)](https://gitter.im/spf13/afero?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

# Overview

Afero is a filesystem framework providing a simple, uniform and universal API
interacting with any filesystem, as an abstraction layer providing interfaces,
types and methods. Afero has an exceptionally clean interface and simple design
without needless constructors or initialization methods.

Afero is also a library providing a base set of interoperable backend
filesystems that make it easy to work with afero while retaining all the power
and benefit of the os and ioutil packages.

Afero provides significant improvements over using the os package alone, most
notably the ability to create mock and testing filesystems without relying on the disk.

It is suitable for use in any situation where you would consider using the OS
package as it provides an additional abstraction that makes it easy to use a
memory backed file system during testing. It also adds support for the http
filesystem for full interoperability.


## Afero Features

* A single consistent API for accessing a variety of filesystems
* Interoperation between a variety of file system types
* A set of interfaces to encourage and enforce interoperability between backends
* An atomic cross platform memory backed file system
* Support for compositional (union) file systems by combining multiple file systems acting as one
* Specialized backends which modify existing filesystems (Read Only, Regexp filtered)
* A set of utility functions ported from io, ioutil & hugo to be afero aware
* Wrapper for go 1.16 filesystem abstraction `io/fs.FS`

# Using Afero

Afero is easy to use and easier to adopt.

A few different ways you could use Afero:

* Use the interfaces alone to define your own file system.
* Wrapper for the OS packages.
* Define different filesystems for different parts of your application.
* Use Afero for mock filesystems while testing

## Step 1: Install Afero

First use go get to install the latest version of the library.

 $ go get github.com/spf13/afero

Next include Afero in your application.
```go
import "github.com/spf13/afero"
```

## Step 2: Declare a backend

First define a package variable and set it to a pointer to a filesystem.
```go
var AppFs = afero.NewMemMapFs()

or

var AppFs = afero.NewOsFs()
```
It is important to note that if you repeat the composite literal you
will be using a completely new and isolated filesystem. In the case of
OsFs it will still use the same underlying filesystem but will reduce
the ability to drop in other filesystems as desired.

## Step 3: Use it like you would the OS package

Throughout your application use any function and method like you normally
would.

So if my application before had:
```go
os.Open("/tmp/foo")
```
We would replace it with:
```go
AppFs.Open("/tmp/foo")
```

`AppFs` being the variable we defined above.


## List of all available functions

File System Methods Available:
```go
Chmod(name string, mode os.FileMode) : error
Chown(name string, uid, gid int) : error
Chtimes(name string, atime time.Time, mtime time.Time) : error
Create(name string) : File, error
Mkdir(name string, perm os.FileMode) : error
MkdirAll(path string, perm os.FileMode) : error
Name() : string
Open(name string) : File, error
OpenFile(name string, flag int, perm os.FileMode) : File, error
Remove(name string) : error
RemoveAll(path string) : error
Rename(oldname, newname string) : error
Stat(name string) : os.FileInfo, error
```
File Interfaces and Methods Available:
```go
io.Closer
io.Reader
io.ReaderAt
io.Seeker
io.Writer
io.WriterAt

Name() : string
Readdir(count int) : []os.FileInfo, error
Readdirnames(n int) : []string, error
Stat() : os.FileInfo, error
Sync() : error
Truncate(size int64) : error
WriteString(s string) : ret int, err error
```
In some applications it may make sense to define a new package that
simply exports the file system variable for easy access from anywhere.

## Using Afero's utility functions

Afero provides a set of functions to make it easier to use the underlying file systems.
These functions have been primarily ported from io & ioutil with some developed for Hugo.

The afero utilities support all afero compatible backends.

The list of utilities includes:

```go
DirExists(path string) (bool, error)
Exists(path string) (bool, error)
FileContainsBytes(filename string, subslice []byte) (bool, error)
GetTempDir(subPath string) string
IsDir(path string) (bool, error)
IsEmpty(path string) (bool, error)
ReadDir(dirname string) ([]os.FileInfo, error)
ReadFile(filename string) ([]byte, error)
SafeWriteReader(path string, r io.Reader) (err error)
TempDir(dir, prefix string) (name string, err error)
TempFile(dir, prefix string) (f File, err error)
Walk(root string, walkFn filepath.WalkFunc) error
WriteFile(filename string, data []byte, perm os.FileMode) error
WriteReader(path string, r io.Reader) (err error)
```
For a complete list see [Afero's GoDoc](https://godoc.org/github.com/spf13/afero)

They are available under two different approaches to use. You can either call
them directly where the first parameter of each function will be the file
system, or you can declare a new `Afero`, a custom type used to bind these
functions as methods to a given filesystem.

### Calling utilities directly

```go
fs := new(afero.MemMapFs)
f, err := afero.TempFile(fs,"", "ioutil-test")

```

### Calling via Afero

```go
fs := afero.NewMemMapFs()
afs := &afero.Afero{Fs: fs}
f, err := afs.TempFile("", "ioutil-test")
```

## Using Afero for Testing

There is a large benefit to using a mock filesystem for testing. It has a
completely blank state every time it is initialized and can be easily
reproducible regardless of OS. You could create files to your heart’s content
and the file access would be fast while also saving you from all the annoying
issues with deleting temporary files, Windows file locking, etc. The MemMapFs
backend is perfect for testing.

* Much faster than performing I/O operations on disk
* Avoid security issues and permissions
* Far more control. 'rm -rf /' with confidence
* Test setup is far more easier to do
* No test cleanup needed

One way to accomplish this is to define a variable as mentioned above.
In your application this will be set to afero.NewOsFs() during testing you
can set it to afero.NewMemMapFs().

It wouldn't be uncommon to have each test initialize a blank slate memory
backend. To do this I would define my `appFS = afero.NewOsFs()` somewhere
appropriate in my application code. This approach ensures that Tests are order
independent, with no test relying on the state left by an earlier test.

Then in my tests I would initialize a new MemMapFs for each test:
```go
func TestExist(t *testing.T) {
	appFS := afero.NewMemMapFs()
	// create test files and directories
	appFS.MkdirAll("src/a", 0755)
	afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644)
	afero.WriteFile(appFS, "src/c", []byte("file c"), 0644)
	name := "src/c"
	_, err := appFS.Stat(name)
	if os.IsNotExist(err) {
		t.Errorf("file \"%s\" does not exist.\n", name)
	}
}
```

# Available Backends

## Operating System Native

### OsFs

The first is simply a wrapper around the native OS calls. This makes it
very easy to use as all of the calls are the same as the existing OS
calls. It also makes it trivial to have your code use the OS during
operation and a mock filesystem during testing or as needed.

```go
appfs := afero.NewOsFs()
appfs.MkdirAll("src/a", 0755)
```

## Memory Backed Storage

### MemMapFs

Afero also provides a fully atomic memory backed filesystem perfect for use in
mocking and to speed up unnecessary disk io when persistence isn’t
necessary. It is fully concurrent and will work within go routines
safely.

```go
mm := afero.NewMemMapFs()
mm.MkdirAll("src/a", 0755)
```

#### InMemoryFile

As part of MemMapFs, Afero also provides an atomic, fully concurrent memory
backed file implementation. This can be used in other memory backed file
systems with ease. Plans are to add a radix tree memory stored file
system using InMemoryFile.

## Network Interfaces

### SftpFs

Afero has experimental support for secure file transfer protocol (sftp). Which can
be used to perform file operations over a encrypted channel.

### GCSFs

Afero has experimental support for Google Cloud Storage (GCS). You can either set the
`GOOGLE_APPLICATION_CREDENTIALS_JSON` env variable to your JSON credentials or use `opts` in
`NewGcsFS` to configure access to your GCS bucket.

Some known limitations of the existing implementation:
* No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version.
* No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version.
* Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent.


## Filtering Backends

### BasePathFs

The BasePathFs restricts all operations to a given path within an Fs.
The given file name to the operations on this Fs will be prepended with
the base path before calling the source Fs.

```go
bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path")
```

### ReadOnlyFs

A thin wrapper around the source Fs providing a read only view.

```go
fs := afero.NewReadOnlyFs(afero.NewOsFs())
_, err := fs.Create("/file.txt")
// err = syscall.EPERM
```

# RegexpFs

A filtered view on file names, any file NOT matching
the passed regexp will be treated as non-existing.
Files not matching the regexp provided will not be created.
Directories are not filtered.

```go
fs := afero.NewRegexpFs(afero.NewMemMapFs(), regexp.MustCompile(`\.txt$`))
_, err := fs.Create("/file.html")
// err = syscall.ENOENT
```

### HttpFs

Afero provides an http compatible backend which can wrap any of the existing
backends.

The Http package requires a slightly specific version of Open which
returns an http.File type.

Afero provides an httpFs file system which satisfies this requirement.
Any Afero FileSystem can be used as an httpFs.

```go
httpFs := afero.NewHttpFs(<ExistingFS>)
fileserver := http.FileServer(httpFs.Dir(<PATH>))
http.Handle("/", fileserver)
```

## Composite Backends

Afero provides the ability have two filesystems (or more) act as a single
file system.

### CacheOnReadFs

The CacheOnReadFs will lazily make copies of any accessed files from the base
layer into the overlay. Subsequent reads will be pulled from the overlay
directly permitting the request is within the cache duration of when it was
created in the overlay.

If the base filesystem is writeable, any changes to files will be
done first to the base, then to the overlay layer. Write calls to open file
handles like `Write()` or `Truncate()` to the overlay first.

To writing files to the overlay only, you can use the overlay Fs directly (not
via the union Fs).

Cache files in the layer for the given time.Duration, a cache duration of 0
means "forever" meaning the file will not be re-requested from the base ever.

A read-only base will make the overlay also read-only but still copy files
from the base to the overlay when they're not present (or outdated) in the
caching layer.

```go
base := afero.NewOsFs()
layer := afero.NewMemMapFs()
ufs := afero.NewCacheOnReadFs(base, layer, 100 * time.Second)
```

### CopyOnWriteFs()

The CopyOnWriteFs is a read only base file system with a potentially
writeable layer on top.

Read operations will first look in the overlay and if not found there, will
serve the file from the base.

Changes to the file system will only be made in the overlay.

Any attempt to modify a file found only in the base will copy the file to the
overlay layer before modification (including opening a file with a writable
handle).

Removing and Renaming files present only in the base layer is not currently
permitted. If a file is present in the base layer and the overlay, only the
overlay will be removed/renamed.

```go
	base := afero.NewOsFs()
	roBase := afero.NewReadOnlyFs(base)
	ufs := afero.NewCopyOnWriteFs(roBase, afero.NewMemMapFs())

	fh, _ = ufs.Create("/home/test/file2.txt")
	fh.WriteString("This is a test")
	fh.Close()
```

In this example all write operations will only occur in memory (MemMapFs)
leaving the base filesystem (OsFs) untouched.


## Desired/possible backends

The following is a short list of possible backends we hope someone will
implement:

* SSH
* S3

# About the project

## What's in the name

Afero comes from the latin roots Ad-Facere.

**"Ad"** is a prefix meaning "to".

**"Facere"** is a form of the root "faciō" making "make or do".

The literal meaning of afero is "to make" or "to do" which seems very fitting
for a library that allows one to make files and directories and do things with them.

The English word that shares the same roots as Afero is "affair". Affair shares
the same concept but as a noun it means "something that is made or done" or "an
object of a particular type".

It's also nice that unlike some of my other libraries (hugo, cobra, viper) it
Googles very well.

## Release Notes

See the [Releases Page](https://github.com/spf13/afero/releases).

## Contributing

1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request

## Contributors

Names in no particular order:

* [spf13](https://github.com/spf13)
* [jaqx0r](https://github.com/jaqx0r)
* [mbertschler](https://github.com/mbertschler)
* [xor-gate](https://github.com/xor-gate)

## License

Afero is released under the Apache 2.0 license. See
[LICENSE.txt](https://github.com/spf13/afero/blob/master/LICENSE.txt)

afero

Abstract File Storage (mem, scp, zip, tar, cloud: s3, gs) for Go.

# afs - abstract file storage

[![GoReportCard](https://goreportcard.com/badge/github.com/viant/afs)](https://goreportcard.com/report/github.com/viant/afs)
[![GoDoc](https://godoc.org/github.com/viant/afs?status.svg)](https://godoc.org/github.com/viant/afs)
![goversion-image](https://img.shields.io/badge/Go-1.11+-00ADD8.svg)
<a href='https://github.com/jpoles1/gopherbadger' target='_blank'>![gopherbadger-tag-do-not-edit](https://img.shields.io/badge/Go%20Coverage-78%25-brightgreen.svg?longCache=true&style=flat)</a>


Please refer to [`CHANGELOG.md`](CHANGELOG.md) if you encounter breaking changes.

- [Motivation](#motivation)
- [Introduction](#introduction)
- [Usage](#usage)
- [Matchers](#matchers)
- [Content modifiers](#content-modifiers)
- [Streaming data](#streaming-data)
- [Options](#options)
- [Storage Implementations](#storage-implementations)
- [Testing mode](#testing-mode)
- [Storage Manager](#storage-managers)
- [GoCover](#gocover)
- [License](#license)
- [Credits and Acknowledgements](#credits-and-acknowledgements)

## Motivation

When dealing with various storage systems, like cloud storage, SCP, container or local file system, using shared API for typical storage operation provides an excellent simplification.
What's more, the ability to simulate storage-related errors like Auth or EOF allows you to test an app error handling.

## Introduction

This library uses a storage manager abstraction to provide an implementation for a specific storage system with following 

* **CRUD Operation:**

```go
List(ctx context.Context, URL string, options ...Option) ([]Object, error)

Walk(ctx context.Context, URL string, handler OnVisit, options ...Option) error

Open(ctx context.Context, object Object, options ...Option) (io.ReadCloser, error)

OpenURL(ctx context.Context, URL string, options ...Option) (io.ReadCloser, error)


Upload(ctx context.Context, URL string, mode os.FileMode, reader io.Reader, options ...Option) error

Create(ctx context.Context, URL string, mode os.FileMode, isDir bool, options ...Option) error

Delete(ctx context.Context, URL string, options ...Option) error
``` 

* **Batch uploader:**

```go
type Upload func(ctx context.Context, parent string, info os.FileInfo, reader io.Reader) error
 
Uploader(ctx context.Context, URL string, options ...Option) (Upload, io.Closer, error)
```

* **Utilities:**

```go

Copy(ctx context.Context, sourceURL, destURL string, options ...Option) error

Move(ctx context.Context, sourceURL, destURL string, options ...Option) error

NewWriter(ctx context.Context, URL string, mode os.FileMode, options ...storage.Option) (io.WriteCloser, error)

DownloadWithURL(ctx context.Context, URL string, options ...Option) ([]byte, error)

Download(ctx context.Context, object Object, options ...Option) ([]byte, error)

```


URL scheme is used to identify storage system, or alternatively relative/absolute path can be used for local file storage.
By default, all operations using the same baseURL share the same corresponding storage manager instance.
For example, instead supplying SCP auth details for all operations, auth option can be used only once.

```go

func main() {

 ctx := context.Background()
 {
 //auth with first call 
 fs := afs.New()
 defer fs.Close()
 keyAuth, err := scp.LocalhostKeyAuth("")
 if err != nil {
 log.Fatal(err)
 }
 reader1, err := fs.OpenURL(ctx, "scp://host1:22/myfolder/asset.txt", keyAuth)
 if err != nil {
 log.Fatal(err)
 }
 ...
 reader2, err := fs.OpenURL(ctx, "scp://host1:22/myfolder/asset.txt", keyAuth)
 }
 
 {
 //auth per baseURL 
 fs := afs.New()
 err = fs.Init(ctx, "scp://host1:22/", keyAuth)
 if err != nil {
 log.Fatal(err)
 }
 defer fs.Destroy("scp://host1:22/")
 reader, err := fs.OpenURL(ctx, "scp://host1:22/myfolder/asset.txt")
 }
}

```

## Usage

##### Downloading location content

```go
func main() {
	
 fs := afs.New()
 ctx := context.Background()
 objects, err := fs.List(ctx, "/tmp/folder")
 if err != nil {
 log.Fatal(err)
 }
 for _, object := range objects {
 fmt.Printf("%v %v\n", object.Name(), object.URL())
 if object.IsDir() {
 continue
 }
 reader, err := fs.Open(ctx, object)
 if err != nil {
 log.Fatal(err)
 }
 data, err := ioutil.ReadAll(reader)
 if err != nil {
 log.Fatal(err)
 }
 fmt.Printf("%s\n", data)
 }
}
```

##### Uploading Content

```go
func main() {
	
 fs := afs.New()
 ctx := context.Background()
 keyAuth, err := scp.LocalhostKeyAuth("")
 if err != nil {
 log.Fatal(err)
 }
 err = fs.Init(ctx, "scp://127.0.0.1:22/", keyAuth)
 if err != nil {
 log.Fatal(err)
 }	
 err = fs.Upload(ctx, "scp://127.0.0.1:22/folder/asset.txt", 0644, strings.NewReader("test me"))
 if err != nil {
 log.Fatal(err)
 }
 ok, err := fs.Exists(ctx, "scp://127.0.0.1:22/folder/asset.txt")
 if err != nil {
 log.Fatal(err)
 }
 fmt.Printf("has file: %v\n", ok)
 _ = fs.Delete(ctx, "scp://127.0.0.1:22/folder/asset.txt")
}
```
##### Uploading Content With Writer

```go
func main() {
	
 fs := afs.New()
 ctx := context.Background()
 keyAuth, err := scp.LocalhostKeyAuth("")
 if err != nil {
 log.Fatal(err)
 }
 err = fs.Init(ctx, "scp://127.0.0.1:22/", keyAuth)
 if err != nil {
 log.Fatal(err)
 }	
 writer = fs.NewWriter(ctx, "scp://127.0.0.1:22/folder/asset.txt", 0644)
 _, err := writer.Write([]byte("test me")))
 if err != nil {
 log.Fatal(err)
 }
 err = writer.Close()
 if err != nil {
 log.Fatal(err)
 }
 ok, err := fs.Exists(ctx, "scp://127.0.0.1:22/folder/asset.txt")
 if err != nil {
 log.Fatal(err)
 }
 fmt.Printf("has file: %v\n", ok)
 _ = fs.Delete(ctx, "scp://127.0.0.1:22/folder/asset.txt")
}
```



##### Data Copy

```go
func main() {

 fs := afs.New()
 ctx := context.Background()
 keyAuth, err := scp.LocalhostKeyAuth("")
 if err != nil {
 log.Fatal(err)
 }
 err = fs.Copy(ctx, "s3://mybucket/myfolder", "scp://127.0.0.1/tmp", option.NewSource(), option.NewDest(keyAuth))
 if err != nil {
 log.Fatal(err)
 }
}
```

##### Archiving content

```go

func main() {
	
 secretPath := path.Join(os.Getenv("HOME"), ".secret", "gcp-e2e.json")
 auth, err := gs.NewJwtConfig(option.NewLocation(secretPath))
 if err != nil {
 return
 }
 sourceURL := "mylocalPath/"
 destURL := "gs:mybucket/test.zip/zip://localhost/dir1"
 fs := afs.New()
 ctx := context.Background()
 err = fs.Copy(ctx, sourceURL, destURL, option.NewDest(auth))
 if err != nil {
 log.Fatal(err)
 }

}	
```


##### Archive Walker

Walker can be created for tar or zip archive.

```go
func main() {
	
 ctx := context.Background()
	fs := afs.New()
	walker := tar.NewWalker(s3afs.New())
	err := fs.Copy(ctx, "/tmp/test.tar", "s3:///dest/folder/test", walker)
	if err != nil {
		log.Fatal(err)
	}
```


##### Archive Uploader

Uploader can be created for tar or zip archive.

```go
func main() {
	
 ctx := context.Background()
	fs := afs.New()
	uploader := zip.NewBatchUploader(gsafs.New())
	err := fs.Copy(ctx, "gs:///tmp/test/data", "/tmp/data.zip", uploader)
	if err != nil {
		log.Fatal(err)
	}
}
```


##### Data Move

```go
func main() {
	
 fs := afs.New()
	ctx := context.Background()
	keyAuth, err := scp.LocalhostKeyAuth("")
	if err != nil {
		log.Fatal(err)
	}
	err = fs.Move(ctx, "/tmp/transient/app", "scp://127.0.0.1/tmp", option.NewSource(), option.NewDest(keyAuth))
	if err != nil {
		log.Fatal(err)
	}
}	
```

##### Batch Upload

```go
func main() {
	
 fs := afs.New()
	ctx := context.Background()
	upload, closer, err := fs.Uploader(ctx, "/tmp/clone")
	if err != nil {
		log.Fatal(err)
	}
	defer closer.Close()
	assets := []*asset.Resource{
		asset.NewFile("asset1.txt", []byte("test 1"), 0644),
		asset.NewFile("asset2.txt", []byte("test 2"), 0644),
		asset.NewDir("folder1", file.DefaultDirOsMode),
		asset.NewFile("folder1/asset1.txt", []byte("test 3"), 0644),
		asset.NewFile("folder1/asset2.txt", []byte("test 4"), 0644),
	}
	for _, asset := range assets {
		relative := ""
		var reader io.Reader
		if strings.Contains(asset.Name, "/") {
			relative, _ = path.Split(asset.Name)
		}
		if ! asset.Dir {
			reader = bytes.NewReader(asset.Data)
		}
		err = upload(ctx, relative, asset.Info(), reader)
		if err != nil {
			log.Fatal(err)
		}
	}
}
```
## Matchers

To filter source content you can use [Matcher](option/matcher.go) option. 
The following have been implemented.


**[Basic Matcher](matcher/basic.go)**

```go
func main() {
	
 matcher, err := NewBasic("/data", ".avro", nil)
 fs := afs.New()
 ctx := context.Background()
 err := fs.Copy(ctx, "/tmp/data", "s3://mybucket/data/", matcher.Match)
 if err != nil {
 log.Fatal(err)
 }
}
```

Exclusion

```go
func main() {
	
 matcher := matcher.Basic{Exclusion:".+/data/perf/\\d+/.+"}
 fs := afs.New()
 ctx := context.Background()
 err := fs.Copy(ctx, "/tmp/data", "s3://mybucket/data/", matcher.Match)
 if err != nil {
 log.Fatal(err)
 }
}
```

**[Filepath matcher](matcher/filepath.go)**

OS style filepath match, with the following terms:
- '*' matches any sequence of non-Separator characters
- '?' matches any single non-Separator character
- '[' [ '^' ] { character-range } ']'

```go

func main() {
	
 matcher := matcher.Filepath("*.avro")
 fs := afs.New()
 ctx := context.Background()
 err := fs.Copy(ctx, "/tmp/data", "gs://mybucket/data/", matcher)
 if err != nil {
 log.Fatal(err)
 }
}	
		

```

**[Ignore Matcher](matcher/ignore.go)**

Ignore matcher represents matcher that matches file that are not in the ignore rules.
The syntax of ignore borrows heavily from that of .gitignore; see https://git-scm.com/docs/gitignore or man gitignore for a full reference.


```go
func mian(){
	
	ignoreMatcher, err := matcher.NewIgnore([]string{"*.txt", ".ignore"})
 	//or matcher.NewIgnore(option.NewLocation(".cloudignore"))
	if err != nil {
		log.Fatal(err)
	}
	fs := afs.New()
	ctx := context.Background()
	objects, err := fs.List(ctx, "/tmp/folder", ignoreMatcher.Match)
	if err != nil {
		log.Fatal(err)
	}
	for _, object := range objects {
		fmt.Printf("%v %v\n", object.Name(), object.URL())
		if object.IsDir() {
			continue
		}
	}
}	
```



**[Modification Time Matcher](matcher/modification.go)**

Modification Time Matcher represents matcher that matches file that were modified either before or after specified time.

```go
func mian(){
	
	before, err := toolbox.TimeAt("2 days ago in UTC")
 if err != nil {
		log.Fatal(err)
	}	
	modTimeMatcher, err := matcher.NewModification(before, nil)
	if err != nil {
		log.Fatal(err)
	}
	fs := afs.New()
	ctx := context.Background()
	objects, err := fs.List(ctx, "/tmp/folder", modTimeMatcher.Match)
	if err != nil {
		log.Fatal(err)
	}
	for _, object := range objects {
		fmt.Printf("%v %v\n", object.Name(), object.URL())
		if object.IsDir() {
			continue
		}
	}
}	
```


## Content modifiers

To modify resource content on the fly you can use [Modifier](option/modifier.go) option.

```go
func main() {
	fs := afs.New()
	ctx := context.Background()
	sourceURL := "file:/tmp/app.war/zip://localhost/WEB-INF/classes/config.properties"
	destURL := "file:/tmp/app.war/zip://localhost/"
	err := fs.Copy(ctx, sourceURL, destURL, modifier.Replace(map[string]string{
		"${changeMe}": os.Getenv("USER"),
	}))
	if err != nil {
		log.Fatal(err)
	}
}
```

```go
package main

import (
	"context"
	"log"
	"github.com/viant/afs"
	"io"
	"fmt"
	"io/ioutil"
	"os"
	"strings"
)

func modifyContent(info os.FileInfo, reader io.ReadCloser) (closer io.ReadCloser, e error) {
 if strings.HasSuffix(info.Name() ,".info") {
 data, err := ioutil.ReadAll(reader)
 if err != nil {
 return nil, err
 }
 _ = reader.Close()
 expanded := strings.Replace(string(data), "$os.User", os.Getenv("USER"), 1)
 reader = ioutil.NopCloser(strings.NewReader(expanded))
 }
 return reader, nil
} 

func main() {

 fs := afs.New()
 reader ,err := fs.OpenURL(context.Background(), "s3://mybucket/meta.info", modifyContent)
 if err != nil {
 log.Fatal(err)	
 }
 
 defer reader.Close()
 content, err := ioutil.ReadAll(reader)
 if err != nil {
 log.Fatal(err)	
 }
 fmt.Printf("content: %s\n", content)
	
}
```


### Streaming data

Streaming data allows data reading and uploading in chunks with small memory footprint.


```go

 jwtConfig, err := gs.NewJwtConfig()
	if err != nil {
		log.Fatal(err)
	}

	ctx := context.Background()
	fs := afs.New()
	sourceURL := "gs://myBucket/path/myasset.gz"
	reader, err := fs.OpenURL(ctx, sourceURL, jwtConfig, option.NewStream(64*1024*1024, 0))
	if err != nil {
		log.Fatal(err)
	}
 
	_ = os.Setenv("AWS_SDK_LOAD_CONFIG", "true")
	destURL := "s3://myBucket/path/myasset.gz"
	err = fs.Upload(ctx, destURL, 0644, reader, &option.Checksum{Skip:true})
	if err != nil {
		log.Fatal(err)
	}

 // or
 writer = fs.NewWriter(ctx, destURL, 0644, &option.Checksum{Skip:true})
 _, err = io.Copy(writer, reader)
 if err != nil {
 log.Fatal(err)
 }
 err = writer.Close()
	if err != nil {
		log.Fatal(err)
	}
```



## Options


* **[Page](option/page.go)**

To control number and position of listed resources you can yse page option.

* **[Timeout](option/timeout.go)**

Provider specific timeout.

* **[BasicAuth](option/cred.go)** 

Provides user/password auth.

* **Source & Dest Options**

Groups options by source or destination options. This options work with Copy or Move operations.

```go

func main() {
	
 fs := afs.New()
 secretPath := path.Join(os.Getenv("HOME"), ".secret","gcp.json")
 jwtConfig, err := gs.NewJwtConfig(option.NewLocation(secretPath))
 if err != nil {
 	log.Fatal(err)
 }
 sourceOptions := option.NewSource(jwtConfig)
 authConfig, err := s3.NewAuthConfig(option.NewLocation("aws.json"))
 if err != nil {
 log.Fatal(err)
 }
 destOptions := option.NewDest(authConfig)
	err = fs.Copy(ctx, "gs://mybucket/data", "s3://mybucket/data", sourceOptions, destOptions)
}

```


* **[option.Checksum](option/checksum.go)** skip computing checksum if Skip is set, this option allows streaming upload in chunks
* **[option.Stream](option/stream.go)**: download reader reads data with specified stream PartSize 





Check out [storage manager](#storage-managers) for additional options. 

## Storage Implementations

- [File](file/README.md)
- [In Memory](mem/README.md)
- [SSH - SCP](scp/README.md)
- [HTTP](http/README.md)
- [Tar](tar/README.md)
- [Zip](zip/README.md)
- [GCP - GS](https://github.com/viant/afsc/tree/master/gs)
- [AWS - S3](https://github.com/viant/afsc/tree/master/s3)

## Testing fs

To unit test all storage operation all in memory you can use faker fs.

In addition you can use error options to test exception handling.

- **DownloadError**
```go

func mian() {
	fs := afs.NewFaker()
	ctx := context.Background()
	err := fs.Upload(ctx, "gs://myBucket/folder/asset.txt", 0, strings.NewReader("some data"), option.NewUploadError(io.EOF))
	if err != nil {
		log.Fatalf("expect upload error: %v", err)
	}
}
```

- **ReaderError**
```go

func mian() {
 fs := afs.NewFaker()
	ctx := context.Background()
	err := fs.Upload(ctx, "gs://myBucket/folder/asset.txt", 0, strings.NewReader("some data"), option.NewDownloadError(io.EOF))
	if err != nil {
		log.Fatal(err)
	}
	_, err = fs.OpenURL(ctx, "gs://myBucket/folder/asset.txt")
	if err != nil {
		log.Fatalf("expect download error: %v", err)
	}
}
```

- **UploadError**
```go

func mian() {
 fs := afs.NewFaker()
 ctx := context.Background()
 err := fs.Upload(ctx, "gs://myBucket/folder/asset.txt", 0, strings.NewReader("some data"), option.NewUploadError(io.EOF))
 if err != nil {
 log.Fatalf("expect upload error: %v", err)
 }
}
```


##### Code generation for static or in memory go file


Generate with mem storage

```
package main

import (
 "log"
 "github.com/viant/afs/parrot
)

func mian() {
 ctx := context.Background()
 err := parrot.GenerateWithMem(ctx, "pathToBinaryAsset", "gen.go", false)
 if err != nil {
 log.Fatal(err)
 }
}

```

Generate static data files

```
package main

import (
 "log"
 "github.com/viant/afs/parrot
)

func mian() {
 ctx := context.Background()
 err := parrot.Generate(ctx, "pathToBinaryAsset", "data/", false)
 if err != nil {
 log.Fatal(err)
 }
}

```


## Test setup utilities

Package [asset](asset) defines basic utilities to quickly manage asset related unit tests.

```go

func Test_XXX(t *testing.T) {

 var useCases = []struct {
		description string
		location string
		options []storage.Option
		assets []*asset.Resource
	}{

	}

	ctx := context.Background()
	for _, useCase := range useCases {
		fs := afs.New()
		mgr, err := afs.Manager(useCase.location, useCase.options...)
		if err != nil {
			log.Fatal(err)
		}
		err = asset.Create(mgr, useCase.location, useCase.assets)
		if err != nil {
			log.Fatal(err)
		}
		
		//... actual app logic

		actuals, err := asset.Load(mgr, useCase.location)
		if err != nil {
			log.Fatal(err)
		}
 for _, expect := range useCase.assets {
 actual, ok := actuals[expect.Name]
 if !assert.True(t, ok, useCase.description+": "+expect.Name+fmt.Sprintf(" - actuals: %v", actuals)) {
 continue
 }
 assert.EqualValues(t, expect.Name, actual.Name, useCase.description+" "+expect.Name)
 assert.EqualValues(t, expect.Mode, actual.Mode, useCase.description+" "+expect.Name)
 assert.EqualValues(t, expect.Dir, actual.Dir, useCase.description+" "+expect.Name)
 assert.EqualValues(t, expect.Data, actual.Data, useCase.description+" "+expect.Name)
 }

		_ = asset.Cleanup(mgr, useCase.location)

	}
}

```




## GoCover

[![GoCover](https://gocover.io/github.com/viant/afs)](https://gocover.io/github.com/viant/afs)


<a name="License"></a>
## License

The source code is made available under the terms of the Apache License, Version 2, as stated in the file `LICENSE`.

Individual files may be made available under their own specific license,
all compatible with Apache License, Version 2. Please see individual files for details.

<a name="Credits-and-Acknowledgements"></a>

## Credits and Acknowledgements

**Library Author:** Adrian Witas

A library to process http file uploads easily.

<div align="center">
	<pre> 
 ▄▄▄▄ ▄▄▄ ██▀███ ▄▄▄ ██ ▄█▀▄▄▄ 
▓█████▄ ▒████▄ ▓██ ▒ ██▒▒████▄ ██▄█▒▒████▄ 
▒██▒ ▄██▒██ ▀█▄ ▓██ ░▄█ ▒▒██ ▀█▄ ▓███▄░▒██ ▀█▄ 
▒██░█▀ ░██▄▄▄▄██ ▒██▀▀█▄ ░██▄▄▄▄██ ▓██ █▄░██▄▄▄▄██ 
░▓█ ▀█▓ ▓█ ▓██▒░██▓ ▒██▒ ▓█ ▓██▒▒██▒ █▄▓█ ▓██▒
░▒▓███▀▒ ▒▒ ▓▒█░░ ▒▓ ░▒▓░ ▒▒ ▓▒█░▒ ▒▒ ▓▒▒▒ ▓▒█░
▒░▒ ░ ▒ ▒▒ ░ ░▒ ░ ▒░ ▒ ▒▒ ░░ ░▒ ▒░ ▒ ▒▒ ░
 ░ ░ ░ ▒ ░░ ░ ░ ▒ ░ ░░ ░ ░ ▒ 
 ░ ░ ░ ░ ░ ░░ ░ ░ ░
 ░ 
	</pre>
 
[![Go Report Card](https://goreportcard.com/badge/github.com/xis/baraka)](https://goreportcard.com/report/github.com/xis/baraka)
[![codecov](https://codecov.io/gh/xis/baraka/branch/master/graph/badge.svg)](https://codecov.io/gh/xis/baraka)
[![Build Status](https://travis-ci.org/xis/baraka.svg?branch=master)](https://travis-ci.org/xis/baraka) 
[![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white&style=flat-square)](https://pkg.go.dev/github.com/xis/baraka/v2)
 
a tool for handling file uploads for http servers

makes it easier to make operations with files from the http request.
</div>

## Contents
 - [Install](#Install)
 - [Simple Usage](#Simple-Usage)
 - [Filtering Parts](#Filtering-Parts)
 - [More](#More)
 - [Contribute](#Contribute)
 - [License](#License)

## Install
```bash
go get github.com/xis/baraka/v2
```

## Simple Usage
```go
func main() {
	// create a parser
	parser := baraka.NewParser(baraka.ParserOptions{
		MaxFileSize: 5 << 20,
		MaxFileCount: 5,
		MaxParseCount: 5,
	})

	store := baraka.NewFilesystemStorage("./files")

	router := gin.Default()
	router.POST("/upload", func(c *gin.Context) {
		// parse
		request, err := parser.Parse(c.Request)
		if err != nil {
			fmt.Println(err)
		}

		// get the form
		images, err := request.GetForm("images")
		if err != nil {
			fmt.Println(err)
		}

		// save
		for key, image := range images {
			err = store.Save("images", "image_"+strconv.Itoa(key), image)
			if err != nil {
				fmt.Println(err)
			}
		}
	})
	router.Run()
}
```
You can use baraka with the other http server libraries, just pass the http.Request to the parser.Parse function.

## Filtering Parts
You can filter parts by their properties, like part's content type. Parser can inspect the part's bytes and detect the type of the part with the Inspector.

```go
// create a parser
parser := baraka.NewParser(baraka.ParserOptions{
	MaxFileSize: 5 << 20,
	MaxFileCount: 5,
	MaxParseCount: 5,
})

// give parser an inspector
parser.SetInspector(baraka.NewDefaultInspector(512))
// give parser a filter
parser.SetFilter(baraka.NewExtensionFilter(".jpg"))
```

Now parser will inspect the each part and it will just return the jpeg ones from the Parse function. You can make your own Inspector and Filter.

## Contribute
Pull requests are welcome. please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

## License
[MIT](https://choosealicense.com/licenses/mit/)

baraka

A file transfer system, support to manage files with http api, rpc call and ftp client.

### Bigfile ———— a file transfer system that supports http, rpc and ftp protocol [简体中文](https://learnku.com/docs/bigfile) ∙ [English](https://bigfile.site)
----


 <a href="https://travis-ci.org/bigfile/bigfile"><img src="https://travis-ci.org/bigfile/bigfile.svg?branch=master"/></a>
 <a href="https://codecov.io/gh/bigfile/bigfile"><img src="https://codecov.io/gh/bigfile/bigfile/branch/master/graph/badge.svg"/></a>
 <a href="https://godoc.org/github.com/bigfile/bigfile"><img src="https://godoc.org/github.com/bigfile/bigfile?status.svg"/></a>
 <a href="https://github.com/bigfile/bigfile"><img src="https://img.shields.io/badge/release-v1.0.11-blue"/></a>
 <a href="https://app.fossa.io/projects/git%2Bgithub.com%2Fbigfile%2Fbigfile?ref=badge_shield"><img src="https://app.fossa.io/api/projects/git%2Bgithub.com%2Fbigfile%2Fbigfile.svg?type=shield"/></a>
 <a href="https://goreportcard.com/report/github.com/bigfile/bigfile"><img src="https://goreportcard.com/badge/github.com/bigfile/bigfile"/></a>
 <a href="https://golangci.com/r/github.com/bigfile/bigfile"><img src="https://golangci.com/badges/github.com/bigfile/bigfile.svg"/></a>
 <a href="https://github.com/avelino/awesome-go"><img src="https://awesome.re/mentioned-badge.svg"/></a>



 <img src="https://bigfile.site/bigfile.png" />


**Bigfile** is a file transfer system, supports http, ftp and rpc protocol. Designed to provide a file management service and give developers more help. At the bottom, bigfile splits the file into small pieces of **1MB**, the same slice will only be stored once.

In fact, we built a file organization system based on the database. Here you can find familiar files and folders.

Since the rpc and http protocols are supported, those languages supported by [grpc](https://grpc.io/) and other languages can be quickly accessed.

More detailed documents can be found here

### Features

* Support HTTP(s) protocol

 * Support rate limit by ip
 * Support cors
 * Support to avoid replay attack
 * Support to validate parameter signature
 * Support Http Single Range Protocol

* Support FTP(s) protocol

* Support RPC protocol

* Support to deploy by [docker](https://hub.docker.com/r/bigfile/bigfile)

* Provide document with English and Chinese


### Install Bigfile

There are kinds of ways to install Bigfile, you can find more detailed documentation here [English](https://bigfile.site/start/) [简体中文](https://learnku.com/docs/bigfile)

### Start Bigfile

1. generate certificates

> bigfile rpc:make-cert

2. start server

> bigfile multi:server
>

This will print some information as follows:

 [2019/09/19 15:38:32.817] 56628 DEBUG bigfile http service listening on: https://0.0.0.0:10985
 [2019/09/19 15:38:32.818] 56628 DEBUG Go FTP Server listening on 2121
 [2019/09/19 15:38:32.819] 56628 DEBUG bigfile rpc service listening on: tcp://[::]:10986


### HTTP Example (Token Create)

```go
package main

import (
	"fmt"
	"io/ioutil"
	libHttp "net/http"
	"strings"
	"time"

	"github.com/bigfile/bigfile/databases/models"
	"github.com/bigfile/bigfile/http"
)

func main() {
	appUid := "42c4fcc1a620c9e97188f50b6f2ab199"
	appSecret := "f8f2ae1fe4f70b788254dcc991a35558"
	body := http.GetParamsSignBody(map[string]interface{}{
		"appUid": appUid,
		"nonce": models.RandomWithMD5(128),
		"path": "/images/png",
		"expiredAt": time.Now().AddDate(0, 0, 2).Unix(),
		"secret": models.RandomWithMD5(44),
		"availableTimes": -1,
		"readOnly": false,
	}, appSecret)
	request, err := libHttp.NewRequest(
		"POST", "https://127.0.0.1:10985/api/bigfile/token/create", strings.NewReader(body))
	if err != nil {
		fmt.Println(err)
		return
	}
	request.Header.Set("Content-Type", "application/x-www-form-urlencoded")
	resp, err := libHttp.DefaultClient.Do(request)
	if err != nil {
		fmt.Println(err)
		return
	}
	if bodyBytes, err := ioutil.ReadAll(resp.Body); err != nil {
		fmt.Println(err)
		return
	} else {
		fmt.Println(string(bodyBytes))
	}
}
```

### RPC Example (Token Create)

```go
package main

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"io/ioutil"

	"google.golang.org/grpc"
	"github.com/bigfile/bigfile/rpc"
	"google.golang.org/grpc/credentials"
)

func createConnection() (*grpc.ClientConn, error) {
	var (
		err error
		conn *grpc.ClientConn
		cert tls.Certificate
		certPool *x509.CertPool
		rootCertBytes []byte
	)
	if cert, err = tls.LoadX509KeyPair("client.pem", "client.key"); err != nil {
		return nil, err
	}

	certPool = x509.NewCertPool()
	if rootCertBytes, err = ioutil.ReadFile("ca.pem"); err != nil {
		return nil, err
	}

	if !certPool.AppendCertsFromPEM(rootCertBytes) {
		return nil, err
	}

	if conn, err = grpc.Dial("192.168.0.103:10986", grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{
		Certificates: []tls.Certificate{cert},
		RootCAs: certPool,
	}))); err != nil {
		return nil, err
	}
	return conn, err
}

func main() {
	var (
		err error
		conn *grpc.ClientConn
	)

	if conn, err = createConnection(); err != nil {
		fmt.Println(err)
		return
	}
	defer conn.Close()
	grpcClient := rpc.NewTokenCreateClient(conn)
 	fmt.Println(grpcClient.TokenCreate(context.TODO(), &rpc.TokenCreateRequest{
 		AppUid: "42c4fcc1a620c9e97188f50b6f2ab199",
 		AppSecret: "f8f2ae1fe4f70b788254dcc991a35558",
 }))
}
```

### FTP Example

![ftp-example](https://bigfile.site/ftp-connect-succesfully.png)

[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fbigfile%2Fbigfile.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fbigfile%2Fbigfile?ref=badge_large)
[![Get it from the Snap Store](https://snapcraft.io/static/images/badges/en/snap-store-white.svg)](https://snapcraft.io/bigfile)

bigfile

Compute message digest, like MD5, SHA256, SHA1, CRC or BLAKE2s, for large files.

checksum
==
[![PkgGoDev](https://pkg.go.dev/badge/github.com/codingsince1985/checksum)](https://pkg.go.dev/github.com/codingsince1985/checksum)
[![Go Report Card](https://goreportcard.com/badge/codingsince1985/checksum)](https://goreportcard.com/report/codingsince1985/checksum)
[![codecov](https://codecov.io/gh/codingsince1985/checksum/branch/master/graph/badge.svg?token=BO1wAQwc85)](https://codecov.io/gh/codingsince1985/checksum)

Compute message digest, like MD5, SHA256, SHA1, CRC or BLAKE2s-256 in Golang for potentially large files.

Usage
--
```go
package main

import (
	"fmt"
	"github.com/codingsince1985/checksum"
)

func main() {
	file := "/home/jerry/Downloads/ubuntu-20.04.2.0-desktop-amd64.iso"

	md5, _ := checksum.MD5sum(file)
	fmt.Println(md5)

	sha256, _ := checksum.SHA256sum(file)
	fmt.Println(sha256)

	sha1, _ := checksum.SHA1sum(file)
	fmt.Println(sha1)

	crc32, _ := checksum.CRC32(file)
	fmt.Println(crc32)

	blake2s256, _ := checksum.Blake2s256(file)
	fmt.Println(blake2s256)
}
```
License
==
checksum is distributed under the terms of the MIT license. See LICENSE for details.


checksum

# copy

[![Go Reference](https://pkg.go.dev/badge/github.com/otiai10/copy.svg)](https://pkg.go.dev/github.com/otiai10/copy)
[![Actions Status](https://github.com/otiai10/copy/workflows/Go/badge.svg)](https://github.com/otiai10/copy/actions)
[![codecov](https://codecov.io/gh/otiai10/copy/branch/main/graph/badge.svg)](https://codecov.io/gh/otiai10/copy)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://github.com/otiai10/copy/blob/main/LICENSE)
[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fotiai10%2Fcopy.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fotiai10%2Fcopy?ref=badge_shield)
[![CodeQL](https://github.com/otiai10/copy/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/otiai10/copy/actions/workflows/codeql-analysis.yml)
[![Go Report Card](https://goreportcard.com/badge/github.com/otiai10/copy)](https://goreportcard.com/report/github.com/otiai10/copy)
[![GitHub tag (latest SemVer)](https://img.shields.io/github/v/tag/otiai10/copy?sort=semver)](https://pkg.go.dev/github.com/otiai10/copy)
[![Docker Test](https://github.com/otiai10/copy/actions/workflows/docker-test.yml/badge.svg)](https://github.com/otiai10/copy/actions/workflows/docker-test.yml)
[![Vagrant Test](https://github.com/otiai10/copy/actions/workflows/vagrant-test.yml/badge.svg)](https://github.com/otiai10/copy/actions/workflows/vagrant-test.yml)
[![GopherJS](https://github.com/otiai10/copy/actions/workflows/gopherjs.yml/badge.svg)](https://github.com/otiai10/copy/actions/workflows/gopherjs.yml)
[![Go WASM](https://github.com/otiai10/copy/actions/workflows/wasm.yml/badge.svg)](https://github.com/otiai10/copy/actions/workflows/wasm.yml)

`copy` copies directories recursively.

# Example Usage

```go
package main

import (
	"fmt"
	cp "github.com/otiai10/copy"
)

func main() {
	err := cp.Copy("your/src", "your/dest")
	fmt.Println(err) // nil
}
```

# Advanced Usage

```go
// Options specifies optional actions on copying.
type Options struct {

	// OnSymlink can specify what to do on symlink
	OnSymlink func(src string) SymlinkAction

	// OnDirExists can specify what to do when there is a directory already existing in destination.
	OnDirExists func(src, dest string) DirExistsAction

	// OnError can let users decide how to handle errors (e.g., you can suppress specific error).
	OnError func(src, dest, string, err error) error

	// Skip can specify which files should be skipped
	Skip func(srcinfo os.FileInfo, src, dest string) (bool, error)

	// RenameDestination can rename destination.
	// If not set, nil, it does nothing.
	RenameDestination func(src, dest string) (string, error)

	// PermissionControl can control permission of
	// every entry.
	// When you want to add permission 0222, do like
	//
	//		PermissionControl = AddPermission(0222)
	//
	// or if you even don't want to touch permission,
	//
	//		PermissionControl = DoNothing
	//
	// By default, PermissionControl = PreservePermission
	PermissionControl PermissionControlFunc

	// Sync file after copy.
	// Useful in case when file must be on the disk
	// (in case crash happens, for example),
	// at the expense of some performance penalty
	Sync bool

	// Preserve the atime and the mtime of the entries
	// On linux we can preserve only up to 1 millisecond accuracy
	PreserveTimes bool

	// Preserve the uid and the gid of all entries.
	PreserveOwner bool

	// The byte size of the buffer to use for copying files.
	// If zero, the internal default buffer of 32KB is used.
	// See https://golang.org/pkg/io/#CopyBuffer for more information.
	CopyBufferSize uint

	// If you want to add some limitation on reading src file,
	// you can wrap the src and provide new reader,
	// such as `RateLimitReader` in the test case.
	WrapReader func(src io.Reader) io.Reader

	// If given, copy.Copy refers to this fs.FS instead of the OS filesystem.
	// e.g., You can use embed.FS to copy files from embedded filesystem.
	FS fs.FS

	// NumOfWorkers represents the number of workers used for
	// concurrent copying contents of directories.
	// If 0 or 1, it does not use goroutine for copying directories.
	// Please refer to https://pkg.go.dev/golang.org/x/sync/semaphore for more details.
	NumOfWorkers int64

	// PreferConcurrent is a function to determine whether or not
	// to use goroutine for copying contents of directories.
	// If PreferConcurrent is nil, which is default, it does concurrent
	// copying for all directories.
	// If NumOfWorkers is 0 or 1, this function will be ignored.
	PreferConcurrent func(srcdir, destdir string) (bool, error)
}
```

```go
// For example...
opt := Options{
	Skip: func(info os.FileInfo, src, dest string) (bool, error) {
		return strings.HasSuffix(src, ".git"), nil
	},
}
err := Copy("your/directory", "your/directory.copy", opt)
```

# Issues

- https://github.com/otiai10/copy/issues


## License
[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fotiai10%2Fcopy.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fotiai10%2Fcopy?ref=badge_large)


copy

File operations library which aims to mirror feature parity with [GNU cp](https://www.gnu.org/software/coreutils/manual/html_node/cp-invocation.html).

# flop

[![GoDoc](https://godoc.org/github.com/homedepot/flop?status.svg)](https://godoc.org/github.com/homedepot/flop)
[![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)
[![Go Report Card](https://goreportcard.com/badge/github.com/homedepot/flop)](https://goreportcard.com/report/github.com/homedepot/flop)

<img src="https://github.com/homedepot/flop/raw/master/doc/logo.png" width="200" title="design based on original gopher by Renee French">

----

flop aims to make copying files easier in Go, and is modeled after
[GNU cp](https://www.gnu.org/software/coreutils/manual/html_node/cp-invocation.html).
Most administrators and engineers interact with GNU utilities every day, so it makes sense to utilize
that knowledge and expectations for a library that does the same operation in code.

flop strategically diverges from cp where it is advantageous for the programmer to explicitly define the
behavior, like cp assuming that copying from a file path to a directory path means the file should be
created inside the directory. This behavior must be explicitly defined in flop by passing the option
AppendNameToPath, otherwise an error will be returned.

```BASH
go get -u github.com/homedepot/flop
```

## Usage

Basic file copy.

```go
err := flop.SimpleCopy("src_path", "dst_path")
handle(err)
```

Advanced file copy with [options](https://pkg.go.dev/github.com/homedepot/flop?tab=doc#Options).

```go
options := flop.Options{
 Recursive: true,
 MkdirAll: true,
}
err := flop.Copy("src_path", "dst_path", options)
handle(err)
```

## Logging

flop won't throw logs at you for no reason, but if you want to follow along with what's going on giving it a logger
can help expose the behavior, or aid in debugging if you are generous enough to contribute.

```go
// the logger just takes a string so format your favorite logger to accept one
import (
	"github.com/homedepot/flop"
	"github.com/rs/zerolog"
	zlog "github.com/rs/zerolog/log"
	llog "github.com/sirupsen/logrus"
)

func logDebug(msg string) {
	llog.WithFields(llog.Fields{
		"application": "stuffcopy",
	}).Info(msg)
}

func main() {
	zlog.Logger = zlog.Output(zerolog.ConsoleWriter{Out: os.Stderr})
	err := flop.Copy(src.Name(), dst.Name(), flop.Options{
		InfoLogFunc: zlog.Info().Msg, // Msg already accepts a string so we can just pass it directly
		DebugLogFunc: logDebug, // logrus Debug takes ...interface{} so we need to wrap it
	})
	handle(err)
}
```

flop

Disk usage analyzer with console interface.

# go DiskUsage()

<img src="./gdu.png" alt="Gdu " width="200" align="right">

[![Codecov](https://codecov.io/gh/dundee/gdu/branch/master/graph/badge.svg)](https://codecov.io/gh/dundee/gdu)
[![Go Report Card](https://goreportcard.com/badge/github.com/dundee/gdu)](https://goreportcard.com/report/github.com/dundee/gdu)
[![Maintainability](https://api.codeclimate.com/v1/badges/30d793274607f599e658/maintainability)](https://codeclimate.com/github/dundee/gdu/maintainability)
[![CodeScene Code Health](https://codescene.io/projects/13129/status-badges/code-health)](https://codescene.io/projects/13129)

Pretty fast disk usage analyzer written in Go.

Gdu is intended primarily for SSD disks where it can fully utilize parallel processing.
However HDDs work as well, but the performance gain is not so huge.

[![asciicast](https://asciinema.org/a/382738.svg)](https://asciinema.org/a/382738)

<a href="https://repology.org/project/gdu/versions">
 <img src="https://repology.org/badge/vertical-allrepos/gdu.svg" alt="Packaging status" align="right">
</a>

## Installation

Head for the [releases page](https://github.com/dundee/gdu/releases) and download the binary for your system.

Using curl:

 curl -L https://github.com/dundee/gdu/releases/latest/download/gdu_linux_amd64.tgz | tar xz
 chmod +x gdu_linux_amd64
 mv gdu_linux_amd64 /usr/bin/gdu

See the [installation page](./INSTALL.md) for other ways how to install Gdu to your system.

Or you can use Gdu directly via Docker:

 docker run --rm --init --interactive --tty --privileged --volume /:/mnt/root ghcr.io/dundee/gdu /mnt/root

## Usage

```
 gdu [flags] [directory_to_scan]

Flags:
 --config-file string Read config from file (default is $HOME/.gdu.yaml)
 -g, --const-gc Enable memory garbage collection during analysis with constant level set by GOGC
 --enable-profiling Enable collection of profiling data and provide it on http://localhost:6060/debug/pprof/
 -L, --follow-symlinks Follow symlinks for files, i.e. show the size of the file to which symlink points to (symlinks to directories are not followed)
 -h, --help help for gdu
 -i, --ignore-dirs strings Absolute paths to ignore (separated by comma) (default [/proc,/dev,/sys,/run])
 -I, --ignore-dirs-pattern strings Absolute path patterns to ignore (separated by comma)
 -X, --ignore-from string Read absolute path patterns to ignore from file
 -f, --input-file string Import analysis from JSON file
 -l, --log-file string Path to a logfile (default "/dev/null")
 -m, --max-cores int Set max cores that GDU will use. 8 cores available (default 8)
 -c, --no-color Do not use colorized output
 -x, --no-cross Do not cross filesystem boundaries
 --no-delete Do not allow deletions
 -H, --no-hidden Ignore hidden directories (beginning with dot)
 --no-mouse Do not use mouse
 --no-prefix Show sizes as raw numbers without any prefixes (SI or binary) in non-interactive mode
 -p, --no-progress Do not show progress in non-interactive mode
 -u, --no-unicode Do not use Unicode symbols (for size bar)
 -n, --non-interactive Do not run in interactive mode
 -o, --output-file string Export all info into file as JSON
 -r, --read-from-storage Read analysis data from persistent key-value storage
 --sequential Use sequential scanning (intended for rotating HDDs)
 -a, --show-apparent-size Show apparent size
 -d, --show-disks Show all mounted disks
 -C, --show-item-count Show number of items in directory
 -M, --show-mtime Show latest mtime of items in directory
 -B, --show-relative-size Show relative size
 --si Show sizes with decimal SI prefixes (kB, MB, GB) instead of binary prefixes (KiB, MiB, GiB)
 --storage-path string Path to persistent key-value storage directory (default is /tmp/badger) (default "/tmp/badger")
 -s, --summarize Show only a total in non-interactive mode
 --use-storage Use persistent key-value storage for analysis data (experimental)
 -v, --version Print version
 --write-config Write current configuration to file (default is $HOME/.gdu.yaml)

Basic list of actions in interactive mode (show help modal for more):
 ↑ or k Move cursor up
 ↓ or j Move cursor down
 → or Enter or l Go to highlighted directory
 ← or h Go to parent directory
 d Delete the selected file or directory
 e Empty the selected directory
 n Sort by name
 s Sort by size
 c Show number of items in directory
 ? Show help modal
```

## Examples

 gdu # analyze current dir
 gdu -a # show apparent size instead of disk usage
 gdu --no-delete # prevent write operations
 gdu <some_dir_to_analyze> # analyze given dir
 gdu -d # show all mounted disks
 gdu -l ./gdu.log <some_dir> # write errors to log file
 gdu -i /sys,/proc / # ignore some paths
 gdu -I '.*[abc]+' # ignore paths by regular pattern
 gdu -X ignore_file / # ignore paths by regular patterns from file
 gdu -c / # use only white/gray/black colors

 gdu -n / # only print stats, do not start interactive mode
 gdu -np / # do not show progress, useful when using its output in a script
 gdu -nps /some/dir # show only total usage for given dir
 gdu / > file # write stats to file, do not start interactive mode

 gdu -o- / | gzip -c >report.json.gz # write all info to JSON file for later analysis
 zcat report.json.gz | gdu -f- # read analysis from file

 GOGC=10 gdu -g --use-storage / # use persistent key-value storage for saving analysis data
 gdu -r / # read saved analysis data from persistent key-value storage

## Modes

Gdu has three modes: interactive (default), non-interactive and export.

Non-interactive mode is started automatically when TTY is not detected (using [go-isatty](https://github.com/mattn/go-isatty)), for example if the output is being piped to a file, or it can be started explicitly by using a flag.

Export mode (flag `-o`) outputs all usage data as JSON, which can be later opened using the `-f` flag.

Hard links are counted only once.

## File flags

Files and directories may be prefixed by a one-character
flag with following meaning:

* `!` An error occurred while reading this directory.

* `.` An error occurred while reading a subdirectory, size may be not correct.

* `@` File is symlink or socket.

* `H` Same file was already counted (hard link).

* `e` Directory is empty.

## Configuration file

Gdu can read (and write) YAML configuration file.

`$HOME/.config/gdu/gdu.yaml` and `$HOME/.gdu.yaml` are checked for the presense of the config file by default.

### Examples

* To configure gdu to permanently run in gray-scale color mode:

```
echo "no-color: true" >> ~/.gdu.yaml
```

* To set default sorting in configuration file:

```
sorting:
 by: name // size, name, itemCount, mtime
 order: desc
```

* To configure gdu to set CWD variable when browsing directories:

```
echo "change-cwd: true" >> ~/.gdu.yaml
```

* To save the current configuration

```
gdu --write-config
```

## Styling

There are wast ways how terminals can be colored.
Some gdu primitives (like basic text) addapt to different color schemas, but the selected/highlighted row does not.

If the default look is not sufficient, it can be changed in configuration file, e.g.:

```
style:
 selected-row:
 text-color: black
 background-color: "#ff0000"
```

## Deletion in background and in parallel (experimental)

Gdu can delete items in the background, thus not blocking the UI for additional work.
To enable:

```
echo "delete-in-background: true" >> ~/.gdu.yaml
```

Directory items can be also deleted in parallel, which might increase the speed of deletion.
To enable:

```
echo "delete-in-parallel: true" >> ~/.gdu.yaml
```

## Memory usage

### Automatic balancing

Gdu tries to balance performance and memory usage.

When less memory is used by gdu than the total free memory of the host,
then Garbage Collection is disabled during the analysis phase completely to gain maximum speed.

Otherwise GC is enabled.
The more memory is used and the less memory is free, the more often will the GC happen.

### Manual memory usage control

If you want manual control over Garbage Collection, you can use `--const-gc` / `-g` flag.
It will run Garbage Collection during the analysis phase with constant level of aggressiveness.
As a result, the analysis will be about 25% slower and will consume about 30% less memory.
To change the level, you can set the `GOGC` environment variable to specify how often the garbage collection will happen.
Lower value (than 100) means GC will run more often. Higher means less often. Negative number will stop GC.

Example running gdu with constant GC, but not so aggressive as default:

```
GOGC=200 gdu -g /
```

## Saving analysis data to persistent key-value storage (experimental)

Gdu can store the analysis data to persistent key-value storage instead of just memory.
Gdu will run much slower (approx 10x) but it should use much less memory (when using small GOGC as well).
Gdu can also reopen with the saved data.
Currently only BadgerDB is supported as the key-value storage (embedded).

```
GOGC=10 gdu -g --use-storage / # saves analysis data to key-value storage
gdu -r / # reads just saved data, does not run analysis again
```

## Running tests

 make install-dev-dependencies
 make test

## Profiling

Gdu can collect profiling data when the `--enable-profiling` flag is set.
The data are provided via embedded http server on URL `http://localhost:6060/debug/pprof/`.

You can then use e.g. `go tool pprof -web http://localhost:6060/debug/pprof/heap`
to open the heap profile as SVG image in your web browser.

## Benchmarks

Benchmarks were performed on 50G directory (100k directories, 400k files) on 500 GB SSD using [hyperfine](https://github.com/sharkdp/hyperfine).
See `benchmark` target in [Makefile](Makefile) for more info.

### Cold cache

Filesystem cache was cleared using `sync; echo 3 | sudo tee /proc/sys/vm/drop_caches`.

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `diskus ~` | 3.126 ± 0.020 | 3.087 | 3.155 | 1.00 |
| `gdu -npc ~` | 3.132 ± 0.019 | 3.111 | 3.173 | 1.00 ± 0.01 |
| `gdu -gnpc ~` | 3.136 ± 0.012 | 3.112 | 3.155 | 1.00 ± 0.01 |
| `pdu ~` | 3.657 ± 0.013 | 3.641 | 3.677 | 1.17 ± 0.01 |
| `dust -d0 ~` | 3.933 ± 0.144 | 3.849 | 4.213 | 1.26 ± 0.05 |
| `dua ~` | 3.994 ± 0.073 | 3.827 | 4.134 | 1.28 ± 0.02 |
| `gdu -npc --use-storage ~` | 12.812 ± 0.078 | 12.644 | 12.912 | 4.10 ± 0.04 |
| `du -hs ~` | 14.120 ± 0.213 | 13.969 | 14.703 | 4.52 ± 0.07 |
| `duc index ~` | 14.567 ± 0.080 | 14.385 | 14.657 | 4.66 ± 0.04 |
| `ncdu -0 -o /dev/null ~` | 14.963 ± 0.254 | 14.759 | 15.637 | 4.79 ± 0.09 |

### Warm cache

| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `pdu ~` | 226.6 ± 3.7 | 219.6 | 231.2 | 1.00 |
| `diskus ~` | 227.7 ± 5.2 | 221.6 | 239.9 | 1.00 ± 0.03 |
| `dust -d0 ~` | 400.1 ± 7.1 | 386.7 | 409.4 | 1.77 ± 0.04 |
| `dua ~` | 444.9 ± 2.4 | 442.4 | 448.9 | 1.96 ± 0.03 |
| `gdu -npc ~` | 451.3 ± 3.8 | 445.9 | 458.5 | 1.99 ± 0.04 |
| `gdu -gnpc ~` | 516.1 ± 6.7 | 503.1 | 527.5 | 2.28 ± 0.05 |
| `du -hs ~` | 905.0 ± 3.9 | 901.2 | 913.4 | 3.99 ± 0.07 |
| `duc index ~` | 1053.0 ± 5.1 | 1046.2 | 1064.1 | 4.65 ± 0.08 |
| `ncdu -0 -o /dev/null ~` | 1653.9 ± 5.7 | 1645.9 | 1663.0 | 7.30 ± 0.12 |
| `gdu -npc --use-storage ~` | 9754.9 ± 688.7 | 8403.8 | 10427.4 | 43.04 ± 3.12 |

## Alternatives

* [ncdu](https://dev.yorhel.nl/ncdu) - NCurses based tool written in pure `C` (LTS) or `zig` (Stable)
* [godu](https://github.com/viktomas/godu) - Analyzer with a carousel like user interface
* [dua](https://github.com/Byron/dua-cli) - Tool written in `Rust` with interface similar to gdu (and ncdu)
* [diskus](https://github.com/sharkdp/diskus) - Very simple but very fast tool written in `Rust`
* [duc](https://duc.zevv.nl/) - Collection of tools with many possibilities for inspecting and visualising disk usage
* [dust](https://github.com/bootandy/dust) - Tool written in `Rust` showing tree like structures of disk usage
* [pdu](https://github.com/KSXGitHub/parallel-disk-usage) - Tool written in `Rust` showing tree like structures of disk usage

## Notes

[HDD icon created by Nikita Golubev - Flaticon](https://www.flaticon.com/free-icons/hdd)

# go-csv-tag

Read csv file from Go using tags

[![godoc for artonge/go-csv-tag](https://godoc.org/github.com/artonge/go-csv-tag?status.svg)](http://godoc.org/github.com/artonge/go-csv-tag)

![Go](https://github.com/artonge/go-csv-tag/workflows/Go/badge.svg)
[![goreportcard for artonge/go-csv-tag](https://goreportcard.com/badge/github.com/artonge/go-csv-tag)](https://goreportcard.com/report/artonge/go-csv-tag)

[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)

**The project is in maintenance mode.**

It is kept compatible with changes in the Go ecosystem but no new features will be developed. PR could be accepted.

# Install

`go get github.com/artonge/go-csv-tag/v2`

# Example

## Load

The csv file:

```csv
name, ID, number
name1, 1, 1.2
name2, 2, 2.3
name3, 3, 3.4
```

Your Go code:

```go
type Demo struct {                                // A structure with tags
	Name string  `csv:"name"`
	ID   int     `csv:"ID"`
	Num  float64 `csv:"number"`
}

tab := []Demo{}                                   // Create the slice where to put the content
err  := csvtag.LoadFromPath(
	"file.csv",                                   // Path of the csv file
	&tab,                                         // A pointer to the create slice
	csvtag.CsvOptions{                            // Load your csv with optional options
		Separator: ';',                           // changes the values separator, default to ','
		Header: []string{"name", "ID", "number"}, // specify custom headers
		TagKey: "csv",                            // specify a custom tag key, default to 'csv'
})
```

You can also load the data from an io.Reader with:

```go
csvtag.LoadFromReader(youReader, &tab)
```

Or from a string with:

```go
csvtag.LoadFromString(yourString, &tab)
```

## Dump

Your Go code:

```go
type Demo struct {                         // A structure with tags
	Name string  `csv:"name"`
	ID   int     `csv:"ID"`
	Num  float64 `csv:"number"`
}

tab := []Demo{                             // Create the slice where to put the content
	Demo{
		Name: "some name",
		ID: 1,
		Num: 42.5,
	},
}

err := csvtag.DumpToFile(tab, "csv_file_name.csv")
```

You can also dump the data into an io.Writer with:

```go
err := csvtag.DumpToWriter(tab, yourIOWriter)
```

Or dump to a string with:

```go
str, err := csvtag.DumpToString(tab)
```

The csv file written:

```csv
name,ID,number
some name,1,42.5
```


go-csv-tag

[![GoDoc](https://godoc.org/github.com/hugocarreira/go-decent-copy?status.svg)](https://godoc.org/github.com/hugocarreira/go-decent-copy) [![Build Status](https://travis-ci.org/hugocarreira/go-decent-copy.svg?branch=master)](https://travis-ci.org/hugocarreira/go-decent-copy) [![Go Report Card](https://goreportcard.com/badge/github.com/hugocarreira/go-decent-copy)](https://goreportcard.com/report/github.com/hugocarreira/go-decent-copy) [![codecov](https://codecov.io/gh/hugocarreira/go-decent-copy/branch/master/graph/badge.svg)](https://codecov.io/gh/hugocarreira/go-decent-copy)

# Go-Decent-Copy

go-decent-copy provides a copy file for humans

## Usage

```go
package main

import "github.com/hugocarreira/go-decent-copy"

func main() {
    execPath, _ := os.Getwd()

    fileOrigin := fmt.Sprintf(execPath + "/testCopy.txt")
    fileDestiny := fmt.Sprintf(execPath + "/testCopied.txt")
    err := Copy(fileOrigin, fileDestiny)
    if err != nil {
        t.Fatalf("Error in copy file : %#v ", err.Error())
    }
}
```

## License

This project is licensed under the [MIT License](LICENSE).


go-decent-copy

Go bindings for ExifTool, the well-known library used to extract as much metadata as possible (EXIF, IPTC, ...) from files (pictures, PDF, office, ...).

# go-exiftool

[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)
[![Build Status](https://github.com/barasher/go-exiftool/workflows/go-exiftool-ci/badge.svg)](https://github.com/barasher/go-exiftool/actions)
[![go report card](https://goreportcard.com/badge/github.com/barasher/go-exiftool "go report card")](https://goreportcard.com/report/github.com/barasher/go-exiftool)
[![GoDoc](https://godoc.org/github.com/barasher/go-exiftool?status.svg)](https://godoc.org/github.com/barasher/go-exiftool)
[![codecov](https://codecov.io/gh/barasher/go-exiftool/branch/master/graph/badge.svg)](https://codecov.io/gh/barasher/go-exiftool)

**go-exiftool** is a golang library that wraps [ExifTool](https://www.sno.phy.queensu.ca/~phil/exiftool/).

[ExifTool](https://www.sno.phy.queensu.ca/~phil/exiftool/)'s purpose is to extract and update as much metadata as possible (EXIF, IPTC, XMP, GPS, ...) from [a lots of differents file types](https://www.sno.phy.queensu.ca/~phil/exiftool/#supported) (Office documents, pictures, movies, PDF, ...).

**go-exiftool** uses [ExifTool](https://www.sno.phy.queensu.ca/~phil/exiftool/)'s *`stay_open`* feature to [optimize performance](https://www.sno.phy.queensu.ca/~phil/exiftool/#performance).

## Requirements

**go-exiftool** needs [ExifTool](https://www.sno.phy.queensu.ca/~phil/exiftool/) to be installed.

- On Debian : `sudo apt-get install exiftool`

By default, `go-exiftool` binary will look for `exiftool` binary in $PATH, but another location can be specified (see `SetExiftoolBinaryPath` functional option).

## Usage

### Metadata extraction

```go
et, err := exiftool.NewExiftool()
if err != nil {
    fmt.Printf("Error when intializing: %v\n", err)
    return
}
defer et.Close()

fileInfos := et.ExtractMetadata("testdata/20190404_131804.jpg")

for _, fileInfo := range fileInfos {
    if fileInfo.Err != nil {
        fmt.Printf("Error concerning %v: %v\n", fileInfo.File, fileInfo.Err)
        continue
    }

    for k, v := range fileInfo.Fields {
        fmt.Printf("[%v] %v\n", k, v)
    }
}
```

Output :

```
[FOV] 69.4 deg
[Orientation] Rotate 90 CW
[ColorSpace] sRGB
[Compression] JPEG (old-style)
[YCbCrSubSampling] YCbCr4:2:2 (2 1)
[Aperture] 1.7
[ColorComponents] 3
[SubSecCreateDate] 2019:04:04 13:18:03.0937
[FileSize] 26 kB
[FileAccessDate] 2019:05:17 22:44:26+02:00
[DateTimeOriginal] 2019:04:04 13:18:03
[CreateDate] 2019:04:04 13:18:03
(...)
```

### Metadata update

See [example function ExampleExiftool_Write in exiftool_sample_test.go](exiftool_sample_test.go)

## Changelog

- v1.1.0 : initial release
- [v1.1.1](https://github.com/barasher/go-exiftool/milestone/2)
  - [add typed getters for filemetadata fields](https://github.com/barasher/go-exiftool/issues/2)
- [v1.1.3](https://github.com/barasher/go-exiftool/milestone/3)
  - [add Mac & Windows support](https://github.com/barasher/go-exiftool/pull/7) (thanks to @PROger4ever)
  - [increase errors readability](https://github.com/barasher/go-exiftool/pull/8) (thanks to @PROger4ever)
- [v1.2.0](https://github.com/barasher/go-exiftool/milestone/4)
  - [add stdout and stderr buffer configuration capabilities](https://github.com/barasher/go-exiftool/issues/6) (thank to @asannikov)
- [v1.3.0](https://github.com/barasher/go-exiftool/milestone/5)
  - Add functionnal option to set [ExifTool](https://www.sno.phy.queensu.ca/~phil/exiftool/)'s `-charset` parameter (thank to @PROger4ever)
- [v1.3.1](https://github.com/barasher/go-exiftool/milestone/6)
  - [fix line separator on macOS](https://github.com/barasher/go-exiftool/issues/16)
- [v1.3.2](https://github.com/barasher/go-exiftool/milestone/7)
  - [add Freebsd support](https://github.com/barasher/go-exiftool/pull/21) (thanks to @ghyman1)
- [v1.4.0](https://github.com/barasher/go-exiftool/milestone/8)
  - ["NoPrintConversion" functional option](https://github.com/barasher/go-exiftool/pull/24) (thanks to [Kjeldgaard](https://github.com/Kjeldgaard))
- [v1.5.0](https://github.com/barasher/go-exiftool/milestone/9)
  - "ExtractEmbedded" functional option
- [v1.6.0](https://github.com/barasher/go-exiftool/milestone/10)
  - "ExtractAllBinaryMetadata" functional option
- [v1.6.1](https://github.com/barasher/go-exiftool/milestone/11)
  - "SetExiftoolBinaryPath" functional option : specify where to find exiftool binary (default : $path)
- [v1.6.2](https://github.com/barasher/go-exiftool/milestone/12)
  - Several improvements (thanks to [Dale Hui](https://github.com/dhui))
- [v1.7.0](https://github.com/barasher/go-exiftool/milestone/13)
  - Add metadata writing capabilities (thanks to [Dale Hui](https://github.com/dhui))
- [v1.8.0](https://github.com/barasher/go-exiftool/milestone/14)
  - [Fix a bug](https://github.com/barasher/go-exiftool/issues/52) that was blocking `go-exiftool` when a folder was provided. Folder metadata extraction will now return a new sentinel error (ErrNotFile).
  - Add a new sentinel error (`ErrBufferTooSmall`) that is returned when `go-exiftool`'s buffer isn't big enough for a specifi file (countermeasure: initialize `go-exiftool` with the `Buffer` option)
  - [New option to specify a date format (`DateFormat`)](https://github.com/barasher/go-exiftool/pull/60) (thanks to [Andy Gorman](https://github.com/agorman))
  - [New option to output format for GPS coordinates (`CoordFormat`)](https://github.com/barasher/go-exiftool/pull/60) (thanks to [Andy Gorman](https://github.com/agorman))
- [v1.9.0](https://github.com/barasher/go-exiftool/milestone/15)
  - [New option to specify `-api` parameter to Exiftool](https://github.com/barasher/go-exiftool/issues/59) (thanks to [Blesmol](https://github.com/Blesmol))
- [v1.10.0](https://github.com/barasher/go-exiftool/milestone/16)
  - [New option to prefix metadata keys with group names (specifies `-G` parameter to Exiftool)](https://github.com/barasher/go-exiftool/issues/67) (thanks to [Andy Gorman](https://github.com/agorman))



go-exiftool

# go-gtfs
Load GTFS files in Go.

[![godoc for artonge/go-gtfs](https://godoc.org/github.com/artonge/go-gtfs?status.svg)](http://godoc.org/github.com/artonge/go-gtfs)

![Go](https://github.com/artonge/go-gtfs/workflows/Go/badge.svg)
[![goreportcard for artonge/go-gtfs](https://goreportcard.com/badge/github.com/artonge/go-gtfs)](https://goreportcard.com/report/artonge/go-gtfs)

[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)

**The project is in maintenance mode.**

It is kept compatible with changes in the Go ecosystem but no new features will be developed. PR could be accepted.
# Install
`go get github.com/artonge/go-gtfs`

# Examples
## Load one directory containing GTFS files:
```bash
path/to/gtfs_files
├── agency.txt
├── attributions.txt
├── calendar_dates.txt
├── calendar.txt
├── fare_attributes.txt
├── fare_rules.txt
├── feed_info.txt
├── frequencies.txt
├── levels.txt
├── pathways.txt
├── routes.txt
├── shapes.txt
├── stops.txt
├── stop_times.txt
├── transfers.txt
└── trips.txt
```
```go
g, err := gtfs.Load("path/to/gtfs_files", nil)
```

## Load a directory containing sub directories containing GTFS files:
```bash
path/to/gtfs_directories
├── gtfs1
│   ├── agency.txt
│   ├── attributions.txt
│   ├── calendar_dates.txt
│   ├── calendar.txt
│   ├── fare_attributes.txt
│   ├── fare_rules.txt
│   ├── feed_info.txt
│   ├── frequencies.txt
│   ├── levels.txt
│   ├── pathways.txt
│   ├── routes.txt
│   ├── shapes.txt
│   ├── stops.txt
│   ├── stop_times.txt
│   ├── transfers.txt
│   └── trips.txt
└── gtfs2
    ├── agency.txt
    ├── attributions.txt
    ├── calendar_dates.txt
    ├── calendar.txt
    ├── fare_attributes.txt
    ├── fare_rules.txt
    ├── feed_info.txt
    ├── frequencies.txt
    ├── levels.txt
    ├── pathways.txt
    ├── routes.txt
    ├── shapes.txt
    ├── stops.txt
    ├── stop_times.txt
    ├── transfers.txt
    └── trips.txt

```
```go
gs, err := gtfs.LoadSplitted("path/to/gtfs_directories", nil)
```

You can then access the data through the GTFS structure.
That structure contains arrays of approriate structures for each files.
```go
type GTFS struct {
	Path           string // The path to the containing directory
	Agency         Agency
	Agencies       []Agency
	Attributions   []Attribution
	Calendars      []Calendar
	CalendarDates  []CalendarDate
	FareAttributes []FareAttribute
	FareRules      []FareRule
	FeedInfos      []FeedInfo
	Frequencies    []Frequency
	Levels         []Level
	Routes         []Route
	Pathways       []Pathway
	Shapes         []Shape
	Stops          []Stop
	StopsTimes     []StopTime
	Trips          []Trip
	Transfers      []Transfer
}

type Route struct {
	ID        string `csv:"route_id"`
	AgencyID  string `csv:"agency_id"`
	ShortName string `csv:"route_short_name"`
	LongName  string `csv:"route_long_name"`
	Type      int    `csv:"route_type"`
	Desc      string `csv:"route_desc"`
	URL       string `csv:"route_url"`
	Color     string `csv:"route_color"`
	TextColor string `csv:"route_text_color"`
}

...
```

# Contributions
Pull requests are welcome ! :)


go-gtfs

A package to convert an HTML template to a PDF file.

[![PkgGoDev](https://pkg.go.dev/badge/github.com/SebastiaanKlippert/go-wkhtmltopdf)](https://pkg.go.dev/github.com/SebastiaanKlippert/go-wkhtmltopdf)
[![Go Report Card](https://goreportcard.com/badge/SebastiaanKlippert/go-wkhtmltopdf)](https://goreportcard.com/report/SebastiaanKlippert/go-wkhtmltopdf)
[![codebeat badge](https://codebeat.co/badges/a6bb7f66-7ae2-4de8-8b61-623ef68096c9)](https://codebeat.co/projects/github-com-sebastiaanklippert-go-wkhtmltopdf-master)
[![codecov](https://codecov.io/gh/SebastiaanKlippert/go-wkhtmltopdf/branch/master/graph/badge.svg)](https://codecov.io/gh/SebastiaanKlippert/go-wkhtmltopdf)

[![Build Status](https://github.com/SebastiaanKlippert/go-wkhtmltopdf/actions/workflows/ubuntu.yml/badge.svg?branch=master)](https://github.com/SebastiaanKlippert/go-wkhtmltopdf/actions/workflows/ubuntu.yml)
[![Build Status](https://github.com/SebastiaanKlippert/go-wkhtmltopdf/actions/workflows/macos.yml/badge.svg?branch=master)](https://github.com/SebastiaanKlippert/go-wkhtmltopdf/actions/workflows/macos.yml)

# go-wkhtmltopdf
Golang commandline wrapper for wkhtmltopdf

See http://wkhtmltopdf.org/index.html for wkhtmltopdf docs.

| :warning: WARNING |
|:---------------------------|
| wkhtmltopdf is no longer maintained and now archived on GitHub. See https://wkhtmltopdf.org/status.html |
| This go package is still maintained and will be for a while, but I recommend looking for alternatives for new projects. |


# What and why
We needed a way to generate PDF documents from Go. These vary from invoices with highly customizable lay-outs to reports with tables, graphs and images. In our opinion the best way to do this was by using HTML/CSS templates as source for our PDFs. Using CSS print media types and millimeters instead of pixel units we can generate very acurate PDF documents using wkhtmltopdf.

go-wkhtmltopdf is a pure Golang wrapper around the wkhtmltopdf command line utility.

It has all options typed out as struct members which makes it very easy to use if you use an IDE with
code completion and it has type safety for all options.
For example you can set general options like
```go
pdfg.Dpi.Set(600)
pdfg.NoCollate.Set(false)
pdfg.PageSize.Set(PageSizeA4)
pdfg.MarginBottom.Set(40)
``` 
The same goes for adding pages, settings page options, TOC options per page etc.

It takes care of setting the correct order of options as these can become very long with muliple pages where 
you have page and TOC options for each page.

Secondly it makes usage in server-type applications easier, every instance (PDF process) has its own output buffer 
which contains the PDF output and you can feed one input document from an io.Reader (using stdin in wkhtmltopdf).
You can combine any number of external HTML documents (HTTP(S) links) with at most one HTML document from stdin and set 
options for each input document.

Note: You can also ignore the internal buffer and let wkhtmltopdf write directly to disk if required for large files, or use the [SetOutput](https://godoc.org/github.com/SebastiaanKlippert/go-wkhtmltopdf#PDFGenerator.SetOutput) method to pass any `io.Writer`.

For us this is one of the easiest ways to generate PDF documents from Go(lang) and performance is very acceptable.

# Installation
go get or use a Go dependency manager of your liking.

```
go get -u github.com/SebastiaanKlippert/go-wkhtmltopdf
```

go-wkhtmltopdf finds the path to wkhtmltopdf by
* first looking in the current dir
* looking in the PATH and PATHEXT environment dirs
* using the WKHTMLTOPDF_PATH environment dir

**Warning**: Running executables from the current path is no longer possible in Go 1.19, see https://pkg.go.dev/os/exec@master#hdr-Executables_in_the_current_directory

If you need to set your own wkhtmltopdf path or want to change it during execution, you can call SetPath().

# Usage
See testfile ```wkhtmltopdf_test.go``` for more complex options, a common use case test is in ```simplesample_test.go``` 

```go
package wkhtmltopdf

import (
 "fmt"
 "log"
)

func ExampleNewPDFGenerator() {

 // Create new PDF generator
 pdfg, err := NewPDFGenerator()
 if err != nil {
 log.Fatal(err)
 }

 // Set global options
 pdfg.Dpi.Set(300)
 pdfg.Orientation.Set(OrientationLandscape)
 pdfg.Grayscale.Set(true)

 // Create a new input page from an URL
 page := NewPage("https://godoc.org/github.com/SebastiaanKlippert/go-wkhtmltopdf")

 // Set options for this page
 page.FooterRight.Set("[page]")
 page.FooterFontSize.Set(10)
 page.Zoom.Set(0.95)

 // Add to document
 pdfg.AddPage(page)

 // Create PDF document in internal buffer
 err = pdfg.Create()
 if err != nil {
 log.Fatal(err)
 }

 // Write buffer contents to file on disk
 err = pdfg.WriteFile("./simplesample.pdf")
 if err != nil {
 log.Fatal(err)
 }

 fmt.Println("Done")
 // Output: Done
}
```

As mentioned before, you can provide one document from stdin, this is done by using a [PageReader](https://godoc.org/github.com/SebastiaanKlippert/go-wkhtmltopdf#PageReader "GoDoc") object as input to AddPage. This is best constructed with [NewPageReader](https://godoc.org/github.com/SebastiaanKlippert/go-wkhtmltopdf#NewPageReader "GoDoc") and will accept any io.Reader so this can be used with files from disk (os.File) or memory (bytes.Buffer) etc. 
A simple example snippet:
```go
html := "<html>Hi</html>"
pdfgen.AddPage(NewPageReader(strings.NewReader(html)))
```

# Saving to and loading from JSON

The package now has the possibility to save the PDF Generator object as JSON and to create
a new PDF Generator from a JSON file.
All options and pages are saved in JSON, pages added using NewPageReader are read to memory before saving and then saved as Base64 encoded strings
in the JSON file.

This is useful to prepare a PDF file and generate the actual PDF elsewhere, for example on AWS Lambda.
To create PDF Generator on the client, where wkhtmltopdf might not be present, function `NewPDFPreparer` can be used.

Use `NewPDFPreparer` to create a PDF Generator object on the client and `NewPDFGeneratorFromJSON` to reconstruct it on the server.

```go 
// Client code
pdfg := NewPDFPreparer()
htmlfile, err := ioutil.ReadFile("testdata/htmlsimple.html")
if err != nil {
 log.Fatal(err)
}
 
pdfg.AddPage(NewPageReader(bytes.NewReader(htmlfile)))
pdfg.Dpi.Set(600)
 
// The contents of htmlsimple.html are saved as base64 string in the JSON file
jb, err := pdfg.ToJSON()
if err != nil {
 log.Fatal(err)
}
 
// Server code
pdfgFromJSON, err := NewPDFGeneratorFromJSON(bytes.NewReader(jb))
if err != nil {
 log.Fatal(err)
}
 
err = pdfgFromJSON.Create()
if err != nil {
 log.Fatal(err)
} 
```

For an example of running this in AWS Lambda see https://github.com/SebastiaanKlippert/go-wkhtmltopdf-lambda

# Speed 
The speed if pretty much determined by wkhtmltopdf itself, or if you use external source URLs, the time it takes to get and render the source HTML.

The go wrapper time is negligible with around 0.04ms for parsing an above average number of commandline options.

Benchmarks are included.

go-wkhtmltopdf

A cross-platform real-time file synchronization tool out of the box.

gofs

Simple and reliable package for file operations like copy/move/diff/list on files, directories and symbolic links.

# gut 🍱

Yet another collection of **G**o **u**tilities & **t**ools.  
A simple one. Just **go** with your **gut** feeling.

---

## Shortcuts

| Symbol            | 🍱 弁当                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Document**      | [![PkgGoDev](https://pkg.go.dev/badge/github.com/1set/gut)](https://pkg.go.dev/github.com/1set/gut)                                                                                                                                                                                                                                                                                                                                          |
| **License**       | [![License](https://img.shields.io/github/license/1set/gut)](https://github.com/1set/gut/blob/master/LICENSE)                                                                                                                                                                                                                                                                                                                                |
| **Build Status**  | [![GitHub Action - Ubuntu Build](https://github.com/1set/gut/workflows/Ubuntu/badge.svg)](https://github.com/1set/gut/actions?workflow=Ubuntu) [![GitHub Action - macOS Build](https://github.com/1set/gut/workflows/macOS/badge.svg)](https://github.com/1set/gut/actions?workflow=macOS) [![GitHub Action - Windows Build](https://github.com/1set/gut/workflows/Windows/badge.svg)](https://github.com/1set/gut/actions?workflow=Windows) |
| **Code Quality**  | [![Go Report Card](https://goreportcard.com/badge/github.com/1set/gut)](https://goreportcard.com/report/github.com/1set/gut) [![Codacy](https://api.codacy.com/project/badge/Grade/f70fcc271c3e4785a3dfb87739a44cd0)](https://www.codacy.com/manual/an9an63/gut) [![Code Climate](https://api.codeclimate.com/v1/badges/6217d80425ed0b096664/maintainability)](https://codeclimate.com/github/1set/gut/maintainability)                      |
| **Test Coverage** | [![Codecov](https://codecov.io/gh/1set/gut/branch/master/graph/badge.svg)](https://codecov.io/gh/1set/gut/branch/master)                                                                                                                                                                                                                                                                                                                     |

## Packages

[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)

- [`yos`](https://godoc.org/github.com/1set/gut/yos): yet another wrapper of platform-independent interface to operating system functionality (esp. file operations)
- [`yrand`](https://godoc.org/github.com/1set/gut/yrand): yet another wrapper of **crypto/rand** with func like **math/rand**
- [`yhash`](https://godoc.org/github.com/1set/gut/yhash): yet another collection of methods to calculate checksum of file/string/bytes
- [`ystring`](https://godoc.org/github.com/1set/gut/ystring): yet another collection of methods to manipulate UTF-8 encoded strings


gut/yos

A tiny cross-platform Go library to hide/unhide files and directories.

# higgs

A tiny cross-platform Go library to hide or unhide files and directories.

**Supported OSs:** All unix based OSs (Tested on Ubuntu, MacOS), Windows

[![unix](https://github.com/dastoori/higgs/actions/workflows/unix.yml/badge.svg)](https://github.com/dastoori/higgs/actions/workflows/unix.yml)
[![windows](https://github.com/dastoori/higgs/actions/workflows/windows.yml/badge.svg)](https://github.com/dastoori/higgs/actions/workflows/windows.yml)
[![codecov](https://codecov.io/gh/dastoori/higgs/branch/master/graph/badge.svg?token=T1AJXSWI3F)](https://codecov.io/gh/dastoori/higgs)
[![GitHub release](https://img.shields.io/github/v/release/dastoori/higgs?sort=semver)](https://github.com/dastoori/higgs/releases) 
[![Go Reference](https://pkg.go.dev/badge/github.com/dastoori/higgs.svg)](https://pkg.go.dev/github.com/dastoori/higgs)
![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/dastoori/higgs)
[![Go Report Card](https://goreportcard.com/badge/github.com/dastoori/higgs)](https://goreportcard.com/report/github.com/dastoori/higgs)
[![GitHub](https://img.shields.io/github/license/dastoori/higgs)](https://github.com/dastoori/higgs/blob/master/LICENSE)
[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go) 

## Installation

```sh
$ go get github.com/dastoori/higgs
```

## Usage

```go
package main

import (
	"fmt"
	"github.com/dastoori/higgs"
)

func main() {
	path := "foo.txt"

	// Hiding a file
	newPath, err := higgs.Hide(path)
	// NOTE: On Unix after hiding the file the file name
	// will be changed to `.foo.txt` and `newPath` contains
	// the new file name

	if err != nil {
		fmt.Println(err)
	}

	// Unhiding a file
	newPath, err = higgs.Unhide(newPath)
	
	if err != nil {
		fmt.Println(err)
	}

	// Setting unix overwrite option (disable by default)
	fh := NewFileHide(".bar.txt", UnixOverwriteOption(true))

	// NOTE: On Unix if a `bar.txt` file exists, it will be
	// overwritten after unhiding `.bar.txt`
	err := fh.Unhide()
	
	if err != nil {
		fmt.Println(err)
	}

	// NOTE: `fh.Path` contains the new file name
}
```

## License

MIT

higgs

A package for reading and creating ISO9660 disk images

## iso9660
[![Go Reference](https://pkg.go.dev/badge/github.com/kdomanski/iso9660.svg)](https://pkg.go.dev/github.com/kdomanski/iso9660)
[![codecov](https://codecov.io/gh/kdomanski/iso9660/branch/master/graph/badge.svg?token=14MJSZYZ24)](https://codecov.io/gh/kdomanski/iso9660)
[![Go Report Card](https://goreportcard.com/badge/github.com/kdomanski/iso9660)](https://goreportcard.com/report/github.com/kdomanski/iso9660)

A package for reading and creating ISO9660

Joliet extension is **NOT** supported.

Experimental support for reading Rock Ridge extension is currently in the works.
If you are experiencing issues, please use the v0.3 release, which ignores Rock Ridge.

## References for the format:
- [ECMA-119 1st edition (December 1986)](https://www.ecma-international.org/wp-content/uploads/ECMA-119_1st_edition_december_1986.pdf) ([Web Archive link](http://web.archive.org/web/20210122025258/https://www.ecma-international.org/wp-content/uploads/ECMA-119_1st_edition_december_1986.pdf))
- [ECMA-119 2nd edition (December 1987)](https://www.ecma-international.org/wp-content/uploads/ECMA-119_2nd_edition_december_1987.pdf) ([Web Archive link](http://web.archive.org/web/20210418211711/https://www.ecma-international.org/wp-content/uploads/ECMA-119_2nd_edition_december_1987.pdf))
- [ECMA-119 3rd edition (December 2017)](https://www.ecma-international.org/wp-content/uploads/ECMA-119_3rd_edition_december_2017.pdf) ([Web Archive link](http://web.archive.org/web/20210527165925/https://www.ecma-international.org/wp-content/uploads/ECMA-119_3rd_edition_december_2017.pdf))
- [ECMA-119 4th edition (June 2019)](https://www.ecma-international.org/wp-content/uploads/ECMA-119_4th_edition_june_2019.pdf) ([Web Archive link](https://www.ecma-international.org/wp-content/uploads/ECMA-119_4th_edition_june_2019.pdf))
- [Rock Ridge Interchange Protocol](http://www.nextcomputers.org/NeXTfiles/Projects/CD-ROM/Rock_Ridge_Interchange_Protocol.pdf) ([Web Archive link](http://web.archive.org/web/20071017082049/http://www.nextcomputers.org/NeXTfiles/Projects/CD-ROM/Rock_Ridge_Interchange_Protocol.pdf))
- [System Use Sharing Protocol v1.12](http://aminet.net/package/docs/misc/RRIP)

## Examples

### Extracting an ISO

```go
package main

import (
  "log"

  "github.com/kdomanski/iso9660/util"
)

func main() {
  f, err := os.Open("/home/user/myImage.iso")
  if err != nil {
    log.Fatalf("failed to open file: %s", err)
  }
  defer f.Close()

  if err = util.ExtractImageToDirectory(f, "/home/user/target_dir"); err != nil {
    log.Fatalf("failed to extract image: %s", err)
  }
}
```

### Creating an ISO

```go
package main

import (
  "log"
  "os"

  "github.com/kdomanski/iso9660"
)

func main() {
  writer, err := iso9660.NewWriter()
  if err != nil {
    log.Fatalf("failed to create writer: %s", err)
  }
  defer writer.Cleanup()

  f, err := os.Open("/home/user/myFile.txt")
  if err != nil {
    log.Fatalf("failed to open file: %s", err)
  }
  defer f.Close()

  err = writer.AddFile(f, "folder/MYFILE.TXT")
  if err != nil {
    log.Fatalf("failed to add file: %s", err)
  }

  outputFile, err := os.OpenFile("/home/user/output.iso", os.O_WRONLY | os.O_TRUNC | os.O_CREATE, 0644)
  if err != nil {
    log.Fatalf("failed to create file: %s", err)
  }

  err = writer.WriteTo(outputFile, "testvol")
  if err != nil {
    log.Fatalf("failed to write ISO image: %s", err)
  }

  err = outputFile.Close()
  if err != nil {
    log.Fatalf("failed to close output file: %s", err)
  }
}
```

### Recursively create an ISO image from the given directories

```go
package main

import (
  "fmt"
  "log"
  "os"
  "path/filepath"
  "strings"

  "github.com/kdomanski/iso9660"
)

func main() {
  writer, err := iso9660.NewWriter()
  if err != nil {
    log.Fatalf("failed to create writer: %s", err)
  }
  defer writer.Cleanup()

  isoFile, err := os.OpenFile("C:/output.iso", os.O_WRONLY|os.O_TRUNC|os.O_CREATE, 0644)
  if err != nil {
    log.Fatalf("failed to create file: %s", err)
  }
  defer isoFile.Close()

  prefix := "F:\\" // the prefix to remove in the output iso file
  sourceFolders := []string{"F:\\test1", "F:\\test2"} // the given directories to create an ISO file from

  for _, folderName := range sourceFolders {
    folderPath := strings.Join([]string{prefix, folderName}, "/")

    walk_err := filepath.Walk(folderPath, func(path string, info os.FileInfo, err error) error {
      if err != nil {
        log.Fatalf("walk: %s", err)
        return err
      }
      if info.IsDir() {
        return nil
      }
      outputPath := strings.TrimPrefix(path, prefix) // remove the source drive name
      fmt.Printf("Adding file: %s\n", outputPath)

      fileToAdd, err := os.Open(path)
      if err != nil {
        log.Fatalf("failed to open file: %s", err)
      }
      defer fileToAdd.Close()

      err = writer.AddFile(fileToAdd, outputPath)
      if err != nil {
        log.Fatalf("failed to add file: %s", err)
      }
      return nil
    })
    if walk_err != nil {
      log.Fatalf("%s", walk_err)
    }
  }

  err = writer.WriteTo(isoFile, "Test")
  if err != nil {
    log.Fatalf("failed to write ISO image: %s", err)
  }
}
```


iso9660

File system event notification library with simple API, similar to os/signal.

notify [![GoDoc](https://godoc.org/github.com/rjeczalik/notify?status.svg)](https://godoc.org/github.com/rjeczalik/notify) [![Build Status](https://img.shields.io/travis/rjeczalik/notify/master.svg)](https://travis-ci.org/rjeczalik/notify "inotify + FSEvents + kqueue") [![Build status](https://img.shields.io/appveyor/ci/rjeczalik/notify-246.svg)](https://ci.appveyor.com/project/rjeczalik/notify-246 "ReadDirectoryChangesW") [![Coverage Status](https://img.shields.io/coveralls/rjeczalik/notify/master.svg)](https://coveralls.io/r/rjeczalik/notify?branch=master)
======

Filesystem event notification library on steroids.

*Documentation*

[godoc.org/github.com/rjeczalik/notify](https://godoc.org/github.com/rjeczalik/notify)

*Installation*

```
~ $ go get -u github.com/rjeczalik/notify
```

*Projects using notify*

- [github.com/rjeczalik/cmd/notify](https://godoc.org/github.com/rjeczalik/cmd/notify)
- [github.com/cortesi/devd](https://github.com/cortesi/devd)
- [github.com/cortesi/modd](https://github.com/cortesi/modd)
- [github.com/syncthing/syncthing](https://github.com/syncthing/syncthing)
- [github.com/OrlovEvgeny/TinyJPG](https://github.com/OrlovEvgeny/TinyJPG)
- [github.com/mitranim/gow](https://github.com/mitranim/gow)


notify

Load Open Packaging Conventions (OPC) files for Go.

# opc

[![PkgGoDev](https://pkg.go.dev/badge/github.com/qmuntal/opc)](https://pkg.go.dev/github.com/qmuntal/opc)
[![Build Status](https://github.com/qmuntal/opc/actions/workflows/test.yml/badge.svg)](https://github.com/qmuntal/opc/actions/workflows/test.yml)
[![Go Report Card](https://goreportcard.com/badge/github.com/qmuntal/opc)](https://goreportcard.com/report/github.com/qmuntal/opc)
[![codecov](https://coveralls.io/repos/github/qmuntal/opc/badge.svg)](https://coveralls.io/github/qmuntal/opc?branch=master)
[![codeclimate](https://codeclimate.com/github/qmuntal/opc/badges/gpa.svg)](https://codeclimate.com/github/qmuntal/opc)
[![License](https://img.shields.io/badge/License-BSD%202--Clause-orange.svg)](https://opensource.org/licenses/BSD-2-Clause)
[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)  

Package opc implements the ISO/IEC 29500-2, also known as the [Open Packaging Convention](https://en.wikipedia.org/wiki/Open_Packaging_Conventions).

The Open Packaging specification describes an abstract model and physical format conventions for the use of XML, Unicode, ZIP, and other openly available technologies and specifications to organize the content and resources of a document within a package.

The OPC is the foundation technology for many new file formats: .docx, .pptx, .xlsx, .3mf, .dwfx, ...

## Features
- [x] Package reader and writer
- [x] Package core properties and relationships
- [x] Part relationships
- [x] ZIP mapping
- [x] Package, relationships and parts validation against specs
- [ ] Part interleaved pieces
- [ ] Digital signatures

## Examples
### Write
```go
// Create a file to write our archive to.
f, _ := os.Create("example.xlsx")

// Create a new OPC archive.
w := opc.NewWriter(f)

// Create a new OPC part.
name := opc.NormalizePartName("docs\\readme.txt")
part, _ := w.Create(name, "text/plain")

// Write content to the part.
part.Write([]byte("This archive contains some text files."))

// Make sure to check the error on Close.
w.Close()
```

### Read
```go
r, _ := opc.OpenReader("testdata/test.xlsx")
defer r.Close()

// Iterate through the files in the archive,
// printing some of their contents.
for _, f := range r.Files {
  fmt.Printf("Contents of %s with type %s :\n", f.Name, f.ContentType)
  rc, _ := f.Open()
  io.CopyN(os.Stdout, rc, 68)
  rc.Close()
  fmt.Println()
}
```


Read and write [parquet](https://parquet.apache.org) files.

# Parquet

Parquet generates a parquet reader and writer based on a struct.  The struct
can be defined by you or it can be generated by reading an existing parquet file.

We (Parsyl) will respond to pull requests and issues to the best of our
abilities.  However, sometimes we will have higher priorities and the response
might not be immediate.

NOTE: If you generate the code based on a parquet file there are quite a few
limitations.  The PageType of each PageHeader must be DATA_PAGE and the Codec
(defined in ColumnMetaData) must be PLAIN or SNAPPY. Also, the parquet file's
schema must consist of the currently [supported types](#supported-types).  But
wait, there's more!  Some of the encodings, like DELTA_BINARY_PACKED, BIT_PACKED,
PLAIN_DICTIONARY, and DELTA_BYTE_ARRAY are also not supported.  I would guess
there are other parquet options that will cause problems since there are so many
possibilities.

## Installation
    
    go get -u github.com/parsyl/parquet/...

This will also install parquet's only two dependencies: thift and snappy

## Usage

First define a struct for the data to be written to parquet:

```go
type Person struct {
  	ID  int32  `parquet:"id"`
	Age *int32 `parquet:"age"`
}
```

Next, add a go:generate comment somewhere (in this example all code lives
in main.go):

```go
// go:generate parquetgen -input main.go -type Person -package main
```

Generate the code for the reader and writer:

```console
$ go generate
```

A new file (parquet.go) has now been written that defines ParquetWriter
and ParquetReader.  Next, make use of the writer and reader:

```go
package main

import (
    "bytes"
    "encoding/json"
)

func main() {
    var buf bytes.Buffer
    w, err := NewParquetWriter(&buf)
    if err != nil {
        log.Fatal(err)
    }

    w.Add(Person{ID: 1, Age: getAge(30)})
    w.Add(Person{ID: 2})

    // Each call to write creates a new parquet row group.
    if err := w.Write(); err != nil {
        log.Fatal(err)
    }

    // Close must be called when you are done.  It writes
    // the parquet metadata at the end of the file.
    if err := w.Close(); err != nil {
        log.Fatal(err)
    }

    r, err := NewParquetReader(bytes.NewReader(buf.Bytes()))
    if err != nil {
        log.Fatal(err)
    }

    enc := json.NewEncoder(os.Stdout)
    for r.Next() {
        var p Person
        r.Scan(&p)
        enc.Encode(p)
    }

    if err := r.Error(); err != nil {
        log.Fatal(err)
    }
}

func getAge(a int32) *int32 { return &a }
```

NewParquetWriter has a couple of optional arguments available: MaxPageSize,
Uncompressed, and Snappy.  For example, the following sets the page size (number
of rows in a page before a new one is created) and sets the page data compression
to snappy:

```go
w, err := NewParquetWriter(&buf, MaxPageSize(10000), Snappy)
```

See [this](./_examples/people) for a complete example of how to generate the code
based on an existing struct.

See [this](./_examples/via_parquet) for a complete example of how to generate the code
based on an existing parquet file.

## Supported Types 

The struct used to define the parquet data can have the following types:

```
int32
uint32
int64
uint64
float32
float64
string
bool
```

Each of these types may be a pointer to indicate that the data is optional.  The
struct can also embed another struct:

```go
type Being struct {
	ID  int32  `parquet:"id"`
	Age *int32 `parquet:"age"`
}

type Person struct {
	Being
	Username string `parquet:"username"`
}
```

Nested and repeated structs are supported too:

```go
type Being struct {
	ID  int32  `parquet:"id"`
	Age *int32 `parquet:"age"`
}

type Person struct {
	Being    Being
	Username string `parquet:"username"`
	Friends  []Being
}
```

If you want a field to be excluded from parquet you can tag
it with a dash or make it unexported like so:

```go
type Being struct {
  	ID  int32  `parquet:"id"`
	Password string`parquet:"-"` //will not be written to parquet
	age int32                    //will not be written to parquet
}
```

## Parquetgen

Parquetgen is the command that go generate should call in
order to generate the code for your custom type.  It also can
print the page headers and file metadata from a parquet file:

```console
$ parquetgen --help
Usage of parquetgen:
  -ignore
        ignore unsupported fields in -type, otherwise log.Fatal is called when an unsupported type is encountered (default true)
  -import string
        import statement of -type if it doesn't live in -package
  -input string
        path to the go file that defines -type
  -metadata
        print the metadata of a parquet file (-parquet) and exit
  -output string
        name of the file that is produced, defaults to parquet.go (default "parquet.go")
  -package string
        package of the generated code
  -pageheaders
        print the page headers of a parquet file (-parquet) and exit (also prints the metadata)
  -parquet string
        path to a parquet file (if you are generating code based on an existing parquet file or printing the file metadata or page headers)
  -struct-output string
        name of the file that is produced, defaults to parquet.go (default "generated_struct.go")
  -type string
        name of the struct that will used for writing and reading
```


parquet

Treat paths as their own type instead of using strings.

# pathtype

Treat paths as their own type instead of using strings. This small package wraps functions from the standard library to create a new Path type.

## Example

### Code

```go
package main

import (
	"fmt"
	"log"

	pt "github.com/jonchun/pathtype"
)

type path = pt.Path

func main() {
	myFile := path("myfile.txt")
	exampleFile := path("example/example.txt")
	fmt.Println(exampleFile.Dir())
	fmt.Println(exampleFile.Dir().Join(myFile))

	res, err := exampleFile.Dir().Join(myFile).Dir().Abs()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(res)

	fmt.Println("=========================")
	listBase(res)
	fmt.Println("=========================")
	listExt(res)
}

// list all Base for files in p
func listBase(p path) {
	if glob, err := p.Glob("*"); err != nil {
		log.Fatal(err)
	} else {
		for _, match := range glob {
			fmt.Println(match.Base())
		}
	}
}

// list all extensions for files in p
func listExt(p path) {
	if glob, err := p.Glob("*"); err != nil {
		log.Fatal(err)
	} else {
		for _, match := range glob {
			fmt.Println(match.Ext())
		}
	}
}
```

### Output

```
example
example/myfile.txt
/home/jonchun/example_module/example
=========================
1.log
2.log
example.txt
=========================
.log
.log
.txt
```

See [GoDoc](https://godoc.org/github.com/jonchun/pathtype) for documentation, but it should be pretty self-explanatory.

## TODO

- Update examples for all the methods and have 1-to-1 coverage of the official examples/docs.


pathtype

# pdfcpu: a Go PDF processor and CLI

[![Test](https://github.com/pdfcpu/pdfcpu/workflows/Test/badge.svg)](https://github.com/pdfcpu/pdfcpu/actions)
[![Coverage Status](https://coveralls.io/repos/github/pdfcpu/pdfcpu/badge.svg?branch=master)](https://coveralls.io/github/pdfcpu/pdfcpu?branch=master)
[![GoDoc](https://godoc.org/github.com/pdfcpu/pdfcpu?status.svg)](https://pkg.go.dev/github.com/pdfcpu/pdfcpu)
[![Go Report Card](https://goreportcard.com/badge/github.com/pdfcpu/pdfcpu)](https://goreportcard.com/report/github.com/pdfcpu/pdfcpu)
[![Hex.pm](https://img.shields.io/hexpm/l/plug.svg)](https://opensource.org/licenses/Apache-2.0)
[![Latest release](https://img.shields.io/github/release/pdfcpu/pdfcpu.svg)](https://github.com/pdfcpu/pdfcpu/releases)
[![](https://img.shields.io/static/v1?label=Sponsor&message=%E2%9D%A4&logo=GitHub&color=%23fe8e86)](https://github.com/sponsors/hhrutter)

<a href="https://pdfcpu.io"><img src="resources/logoSmall.png" width="150"></a>
<a href="https://pdfa.org"><img src="resources/pdfa.png" width="75"></a>

pdfcpu is a PDF processing library written in [Go](http://golang.org) that supports encryption and offers both an API and a command-line interface (CLI). It is compatible with all PDF versions with basic support and ongoing improvement for PDF 2.0 (ISO-32000-2).


## Motivation

This is an effort to build a comprehensive PDF processing library from the ground up written in Go. Over time pdfcpu aims to support the standard range of PDF processing features and also any interesting use cases that may present themselves along the way.


 <kbd><a href="https://pdfcpu.io/generate/grid"><img src="resources/gridpdf.png" height="150"></a></kbd>&nbsp;
 <kbd><a href="https://pdfcpu.io/core/watermark"><img src="resources/wmi1abs.png" height="150"></a></kbd>&nbsp;
 <kbd><a href="https://pdfcpu.io/generate/nup"><img src="resources/nup9pdf.png" height="150"></a></kbd>&nbsp;
 <kbd><a href="https://pdfcpu.io/fonts/fonts"><img src="resources/cjkv.png" height="150"></a></kbd>&nbsp;
 <kbd><a href="https://pdfcpu.io/core/stamp"><img src="resources/4exp.png" height="150"></a></kbd>&nbsp;
 <kbd><a href="https://pdfcpu.io/form/form"><img src="resources/form.png" height="150"></a></kbd> 
 <kbd><a href="https://pdfcpu.io/generate/create"><img src="resources/table.png" height="100"></a></kbd>&nbsp;
 <kbd><a href="https://pdfcpu.io/core/stamp"><img src="resources/sti.png" height="150"></a></kbd>&nbsp;
 <kbd><img src="resources/hold3.png" height="150"></kbd>&nbsp;
 <kbd><a href="https://pdfcpu.io/core/watermark"><img src="resources/wmi4.png" height="150"></a></kbd>&nbsp;
 <kbd><a href="https://pdfcpu.io/generate/create"><img src="resources/imagebox.png" height="100"></a></kbd>&nbsp; 
 <kbd><a href="https://pdfcpu.io/generate/booklet"><img src="resources/book2A4p1.png" height="150"></a></kbd>
 <kbd><a href="https://pdfcpu.io/core/stamp"><img src="resources/stp.png" height="150"></a></kbd>&nbsp;
 <kbd><a href="https://pdfcpu.io/generate/grid"><img src="resources/gridimg.png" height="150"></a></kbd>
 <kbd><a href="https://pdfcpu.io/core/stamp"><img src="resources/stRoundBorder.png" height="150"></a></kbd>
 <kbd><a href="https://pdfcpu.io/generate/booklet"><img src="resources/book4A4p1.png" height="150"></a></kbd>


## Focus

The primary emphasis is on providing robust assistance for batch processing and scripting through a comprehensive command-line interface.
Simultaneously, pdfcpu aims to simplify the integration of PDF processing into your Go-based backend system by offering a versatile set of commands.

## Command Set

* [annotations](https://pdfcpu.io/annot/annot)
* [attachments](https://pdfcpu.io/attach/attach)
* [booklet](https://pdfcpu.io/generate/booklet)
* [bookmarks](https://pdfcpu.io/bookmarks/bookmarks)
* [boxes](https://pdfcpu.io/boxes/boxes)
* [change owner password](https://pdfcpu.io/encrypt/change_opw)
* [change user password](https://pdfcpu.io/encrypt/change_upw)
* [collect](https://pdfcpu.io/core/collect)
* [create](https://pdfcpu.io/generate/create)
* [crop](https://pdfcpu.io/core/crop)
* [cut](https://pdfcpu.io/generate/cut)
* [decrypt](https://pdfcpu.io/encrypt/decryptPDF)
* [encrypt](https://pdfcpu.io/encrypt/encryptPDF)
* [extract](https://pdfcpu.io/extract/extract)
* [fonts](https://pdfcpu.io/fonts/fonts)
* [form](https://pdfcpu.io/form/form)
* [grid](https://pdfcpu.io/generate/grid)
* [images](https://pdfcpu.io/images/images)
* [import](https://pdfcpu.io/generate/import)
* [info](https://pdfcpu.io/info)
* [keywords](https://pdfcpu.io/keywords/keywords)
* [merge](https://pdfcpu.io/core/merge)
* [ndown](https://pdfcpu.io/generate/ndown)
* [nup](https://pdfcpu.io/generate/nup)
* [optimize](https://pdfcpu.io/core/optimize)
* [pagelayout](https://pdfcpu.io/pagelayout/pagelayout)
* [pagemode](https://pdfcpu.io/pagemode/pagemode)
* [pages](https://pdfcpu.io/pages/pages)
* [permissions](https://pdfcpu.io/encrypt/perm_add)
* [portfolio](https://pdfcpu.io/portfolio/portfolio)
* [poster](https://pdfcpu.io/generate/poster)
* [properties](https://pdfcpu.io/properties/properties)
* [resize](https://pdfcpu.io/core/resize)
* [rotate](https://pdfcpu.io/core/rotate)
* [split](https://pdfcpu.io/core/split)
* [stamp](https://pdfcpu.io/core/stamp)
* [trim](https://pdfcpu.io/core/trim)
* [validate](https://pdfcpu.io/core/validate)
* [viewerpref](https://pdfcpu.io/viewerpref/viewerpref)
* [watermark](https://pdfcpu.io/core/watermark)
* [zoom](https://pdfcpu.io/core/zoom)

## Documentation

* [pdfcpu.io](https://pdfcpu.io)
* [API tests](https://github.com/pdfcpu/pdfcpu/tree/master/pkg/api/test)
* [API samples](https://github.com/pdfcpu/pdfcpu/tree/master/pkg/samples) 
* CLI usage: `$ pdfcpu help cmd`

### GoDoc

* [pdfcpu package](https://pkg.go.dev/github.com/pdfcpu/pdfcpu)
* [pdfcpu API](https://pkg.go.dev/github.com/pdfcpu/pdfcpu/pkg/api)
* [pdfcpu CLI](https://pkg.go.dev/github.com/pdfcpu/pdfcpu/pkg/cli)

## Reminder

* Always make sure your work is based on the latest commit! 
* pdfcpu is still *Alpha* - bugfixes are committed on the fly and will be mentioned in the next release notes. 
* Follow [pdfcpu](https://twitter.com/pdfcpu) for news and release announcements.
* For quick questions or discussions get in touch on the [Gopher Slack](https://invite.slack.golangbridge.org/) in the #pdfcpu channel.


## Demo Screencast

(using older version with a smaller command set)

[![asciicast](resources/demo.png)](https://asciinema.org/a/P5jaAo9kgZXKj2iSA1OqIdLAU)

## Installation

### Download
Get the latest binary [here](https://github.com/pdfcpu/pdfcpu/releases).


### Using Go Modules

```
$ git clone https://github.com/pdfcpu/pdfcpu
$ cd pdfcpu/cmd/pdfcpu
$ go install
$ pdfcpu version
```
or directly through Go install:

```
$ go install github.com/pdfcpu/pdfcpu/cmd/pdfcpu@latest
```

### Using Homebrew (macOS)
```
$ brew install pdfcpu
$ pdfcpu version
```

### Using DNF/YUM (Fedora)
```
$ sudo dnf install golang-github-pdfcpu
$ pdfcpu version
```

### Run in a Docker container

```shell
$ docker build -t pdfcpu .
# mount current host folder into container as /app to process files in the local host folder
$ docker run -it -v "$(pwd)":/app pdfcpu validate a.pdf
```

## Contributing

### What

* Please [create](https://github.com/pdfcpu/pdfcpu/issues/new/choose) an issue if you find a bug or want to propose a change.
* Feature requests - always welcome!
* Bug fixes - always welcome!
* PRs - let's [discuss](https://github.com/pdfcpu/pdfcpu/discussions) first or [create](https://github.com/pdfcpu/pdfcpu/issues/new/choose) an issue.
* pdfcpu is stable but still *Alpha* and occasionally undergoing heavy changes.

### How

* The pdfcpu [discussion board](https://github.com/pdfcpu/pdfcpu/discussions) is open! Please engage in any form helpful for the community.
* If you want to report a bug please attach the *very verbose* (`pdfcpu cmd -vv ...`) output and ideally a test PDF that you can share.
* Always make sure your contribution is based on the latest commit.
* Please sign your commits.

### Reporting Crashes

Unfortunately crashes do happen :(
For the majority of the cases this is due to a diverse pool of PDF Writers out there and millions of PDF files using different versions waiting to be processed by pdfcpu. Sometimes these PDFs were written more than 20(!) years ago. Often there is an issue with validation - sometimes a bug in the parser. Many times even using relaxed validation with pdfcpu does not work. In these cases we need to extend relaxed validation and for this we are relying on your help. By reporting crashes you are helping to improve the stability of pdfcpu. If you happen to crash on any pdfcpu operation be it on the command line or in your Go backend these are the steps to report this:

Regardless of the pdfcpu operation, please start using the pdfcpu command line to validate your file:

``` sh
$ pdfcpu validate -v &> crash.log
```

 or to produce very verbose output

 ``` sh
 $ pdfcpu validate -vv &> crash.log
 ```

will produce what's needed to investigate a crash. Then open an issue and post `crash.log` or its contents. Ideally post a test PDF you can share to reproduce this. You can also email to hhrutter@gmail.com or if you prefer Slack you can get in touch on the Gopher slack #pdfcpu channel.

If processing your PDF with pdfcpu crashes during validation and can be opened by Adobe Reader and Mac Preview chances are we can extend relaxed validation and provide a fix. If the file in question cannot be opened by both Adobe Reader and Mac Preview we cannot help you!

## Contributors

Thanks 💚 goes to these wonderful people:


||||||||
| :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| [<img src="https://avatars1.githubusercontent.com/u/11322155?v=4" width="100px"/> Horst Rutter](https://github.com/hhrutter) | [<img src="https://avatars0.githubusercontent.com/u/5140211?v=4" width="100px"/> haldyr](https://github.com/haldyr) | [<img src="https://avatars3.githubusercontent.com/u/20608155?v=4" width="100px"/> Vyacheslav](https://github.com/SimePel) | [<img src="https://avatars1.githubusercontent.com/u/617459?v=4" width="100px"/> Erik Unger](https://github.com/ungerik) | [<img src="https://avatars1.githubusercontent.com/u/13079058?v=4" width="100px"/> Richard Wilkes](https://github.com/richardwilkes) | [<img src="https://avatars1.githubusercontent.com/u/16303386?s=400&v=4" width="100px"/> minenok-tutu](https://github.com/minenok-tutu) | [<img src="https://avatars0.githubusercontent.com/u/1965445?s=400&v=4" width="100px"/> Mateusz Burniak](https://github.com/matbur) |
| [<img src="https://avatars2.githubusercontent.com/u/1175110?s=400&v=4" width="100px"/> Dmitry Harnitski](https://github.com/dharnitski) | [<img src="https://avatars0.githubusercontent.com/u/1074083?s=400&v=4" width="100px"/> ryarnyah](https://github.com/ryarnyah) | [<img src="https://avatars0.githubusercontent.com/u/13267?s=400&v=4" width="100px"/> Sam Giffney](https://github.com/s01ipsist) | [<img src="https://avatars3.githubusercontent.com/u/32948066?s=400&v=4" width="100px"/> Carlos Eduardo Witte](https://github.com/cewitte) | [<img src="https://avatars1.githubusercontent.com/u/2374948?s=400&u=a36e5f8da8dc1c102bc4d283f25e4c61cae7f985&v=4" width="100px"/> minusworld](https://github.com/minusworld) | [<img src="https://avatars0.githubusercontent.com/u/18538487?s=400&u=b9e628dfc60f672a887be2ed04a791195829943e&v=4" width="100px"/> Witold Konior](https://github.com/jozuenoon) | [<img src="https://avatars0.githubusercontent.com/u/630151?s=400&v=4" width="100px"/> joonas.fi](https://github.com/joonas-fi) |
| [<img src="https://avatars3.githubusercontent.com/u/10349817?s=400&u=93bacb23bd2909d5b6c5b644a8d4cdd947422ee1&v=4" width="100px"/> Henrik Reinstädtler](https://github.com/henrixapp) | [<img src="https://avatars1.githubusercontent.com/u/72016286?s=400&v=4" width="100px"/> VMorozov-wh](https://github.com/VMorozov-wh) | [<img src="https://avatars0.githubusercontent.com/u/31929422?s=400&v=4" width="100px"/> Benoit KUGLER](https://github.com/benoitkugler) | [<img src="https://avatars.githubusercontent.com/u/704919?s=400&v=4" width="100px"/> Adam Greenhall](https://github.com/adamgreenhall) | [<img src="https://avatars.githubusercontent.com/u/5201812?s=400&u=8a0a9fca4560be71d4923299ddebf877854eea54&v=4" width="100px"/> moritamori](https://github.com/moritamori) | [<img src="https://avatars.githubusercontent.com/u/41904529?s=400&u=044396494285ad806e86d1936c390b3071ce57c0&v=4" width="100px"/> JanBaryla](https://github.com/JanBaryla) | [<img src="https://avatars.githubusercontent.com/u/43145244?s=400&u=89a689f1a854ce0f57ae2a0333c82bfdc5723bb9&v=4" width="100px"/> TheDiscordian](https://github.com/TheDiscordian) |
| [<img src="https://avatars.githubusercontent.com/u/15472552?v=4" width="100px"/> Rafael Garcia Argente](https://github.com/rgargente) | [<img src="https://avatars.githubusercontent.com/u/710057?v=4" width="100px"/> truyet](https://github.com/truyet) | [<img src="https://avatars.githubusercontent.com/u/5031217?v=4" width="100px"/> Christian Nicola](https://github.com/christiannicola) | [<img src="https://avatars.githubusercontent.com/u/3233970?v=4" width="100px"/> Benjamin Krill](https://github.com/kben) | [<img src="https://avatars.githubusercontent.com/u/26521615?v=4" width="100px"/> Peter Wyatt](https://github.com/petervwyatt) | [<img src="https://avatars.githubusercontent.com/u/3142701?v=4" width="100px"/> Kroum Tzanev](https://github.com/kpym) | [<img src="https://avatars.githubusercontent.com/u/992878?v=4" width="100px"/> Stefan Huber](https://github.com/signalwerk) |
| [<img src="https://avatars.githubusercontent.com/u/59667587?v=4" width="100px"/> Juan Iscar](https://github.com/juaismar) | [<img src="https://avatars.githubusercontent.com/u/20135478?v=4" width="100px"/> Eng Zer Jun](https://github.com/Juneezee) | [<img src="https://avatars.githubusercontent.com/u/28459131?v=4" width="100px"/> Dmitry Ivanov](https://github.com/hant0508)|[<img src="https://avatars.githubusercontent.com/u/16866547?v=4" width="100px"/> Rene Kaufmann](https://github.com/HeavyHorst)|[<img src="https://avatars.githubusercontent.com/u/26827864?v=4" width="100px"/> Christian Heusel](https://github.com/christian-heusel) | [<img src="https://avatars.githubusercontent.com/u/305673?v=4" width="100px"/> Chris](https://github.com/freshteapot) | [<img src="https://avatars.githubusercontent.com/u/2892794?v=4" width="100px"/> Lukasz Czaplinski](https://github.com/scoiatael) |
[<img src="https://avatars.githubusercontent.com/u/49206635?v=4" width="100px"/> Joel Silva Schutz](https://github.com/joelschutz) | [<img src="https://avatars.githubusercontent.com/u/28219076?v=4" width="100px"/> semvis123](https://github.com/semvis123) | [<img src="https://avatars.githubusercontent.com/u/8717479?v=4" width="100px"/> guangwu](https://github.com/testwill) | [<img src="https://avatars.githubusercontent.com/u/4014912?v=4" width="100px"/> Yoshiki Nakagawa](https://github.com/yyoshiki41) | [<img src="https://avatars.githubusercontent.com/u/432860?v=4" width="100px"/> Steve van Loben Sels](https://github.com/stevevls) | [<img src="https://avatars.githubusercontent.com/u/6083533?v=4" width="100px"/> Yaofu](https://github.com/mygityf) | [<img src="https://avatars.githubusercontent.com/u/15724278?v=4" width="100px"/> vsenko](https://github.com/vsenko) |
[<img src="https://avatars.githubusercontent.com/u/16507?v=4" width="100px"/> Alexis Hildebrandt](https://github.com/afh) | [<img src="https://avatars.githubusercontent.com/u/1395040?v=4" width="100px"/> Sivukhin Nikita](https://github.com/sivukhin) | [<img src="https://avatars.githubusercontent.com/u/247730?v=4" width="100px"/> Joachim Bauch](https://github.com/fancycode) | [<img src="https://avatars.githubusercontent.com/u/127291996?v=4" width="100px"/> kalimit](https://github.com/kalimit) | [<img src="https://avatars.githubusercontent.com/u/5080535?v=4" width="100px"/> Andreas Erhard](https://github.com/xelan) | [<img src="https://avatars.githubusercontent.com/u/32378535?v=4" width="100px"/> Matsumoto Toshi](https://github.com/toshi1127) | [<img src="https://avatars.githubusercontent.com/u/440634?v=4" width="100px"/> Carl Wilson](https://github.com/carlwilson) |
[<img src="https://avatars.githubusercontent.com/u/9918222?v=4" width="100px"/> LNAhri](https://github.com/LNAhri) | | | | | | 














## Code of Conduct

Please note that this project is released with a Contributor [Code of Conduct](CODE_OF_CONDUCT.md). By participating in this project you agree to abide by its terms.

## Disclaimer

Usage of pdfcpu assumes you know about and respect all copyrights of any PDF content you may be processing. This applies to the PDF files as such, their content and in particular all embedded resources like font files or images. Credit goes to [Renee French](https://instagram.com/reneefrench) for creating our beloved Gopher.

## License

Apache-2.0

pdfcpu

Package to allow one to concurrently go through a filesystem with ease.

# skywalker [![GoDoc](https://godoc.org/github.com/dixonwille/skywalker?status.svg)](https://godoc.org/github.com/dixonwille/skywalker) [![Build Status](https://travis-ci.org/dixonwille/skywalker.svg?branch=master)](https://travis-ci.org/dixonwille/skywalker) [![Build status](https://ci.appveyor.com/api/projects/status/d1h7lpf0pv546amh?svg=true)](https://ci.appveyor.com/project/dixonwille/skywalker) [![codecov](https://codecov.io/gh/dixonwille/skywalker/branch/master/graph/badge.svg)](https://codecov.io/gh/dixonwille/skywalker) [![Go Report Card](https://goreportcard.com/badge/github.com/dixonwille/skywalker)](https://goreportcard.com/report/github.com/dixonwille/skywalker)

Skywalker is a package to allow one to concurrently go through a filesystem with ease.

## Features

- Concurrency
- BlackList filtering
- WhiteList filtering
- Filter by Directory
- Filter by Extension
- Glob Filtering (provided by [gobwas/glob](https://github.com/gobwas/glob))

> For matching to work properly across platforms. Please use `/`. In [gobwas/glob](https://github.com/gobwas/glob) the `\` is an escape character (so you can escape `*`, `?`, etc...) making it difficult to know if you want to escape a character or go into directory.

## Example

```go
package main

import (
    "fmt"
    "sort"
    "strings"
    "sync"

    "github.com/dixonwille/skywalker"
)

type ExampleWorker struct {
    *sync.Mutex
    found []string
}

func (ew *ExampleWorker) Work(path string) {
    //This is where the necessary work should be done.
    //This will get concurrently so make sure it is thread safe if you need info across threads.
    ew.Lock()
    defer ew.Unlock()
    ew.found = append(ew.found, path)
}

func main() {
    //Following two functions are only to create and destroy data for the example
    defer teardownData()
    standupData()

    ew := new(ExampleWorker)
    ew.Mutex = new(sync.Mutex)

    //root is the root directory of the data that was stood up above
    sw := skywalker.New(root, ew)
    sw.DirListType = skywalker.LTBlacklist
    sw.DirList = []string{"sub"}
    sw.ExtListType = skywalker.LTWhitelist
    sw.ExtList = []string{".pdf"}
    err := sw.Walk()
    if err != nil {
        fmt.Println(err)
        return
    }
    sort.Sort(sort.StringSlice(ew.found))
    for _, f := range ew.found {
        fmt.Println(strings.Replace(f, sw.Root, "", 1))
    }
}
```


skywalker

Go library for Gina Trapani's [_todo.txt_](http://todotxt.org/) files, supports parsing and manipulating of task lists in the [_todo.txt_ format](https://github.com/todotxt/todo.txt).

# todotxt

Yet another a Go library for Gina Trapani's todo.txt files. ✅

[![PkgGoDev](https://pkg.go.dev/badge/github.com/1set/todotxt)](https://pkg.go.dev/github.com/1set/todotxt)
[![License](https://img.shields.io/github/license/1set/todotxt)](https://github.com/1set/todotxt/blob/master/LICENSE)
[![GitHub Action Workflow](https://github.com/1set/todotxt/workflows/build/badge.svg)](https://github.com/1set/todotxt/actions?workflow=build)
[![Go Report Card](https://goreportcard.com/badge/github.com/1set/todotxt)](https://goreportcard.com/report/github.com/1set/todotxt)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/448bb14650984e21a7c41bcac221a1ba)](https://www.codacy.com/gh/1set/todotxt/dashboard)
[![Codecov](https://codecov.io/gh/1set/todotxt/branch/master/graph/badge.svg)](https://codecov.io/gh/1set/todotxt)

## Features

Based on [**go-todotxt**](https://github.com/JamesClonk/go-todotxt) from [Fabio Berchtold](https://github.com/JamesClonk) with:

- [x] Go mod support
- [x] Segments for task string
- [x] Task due today is not overdue
- [x] Negative `Due()` for overdue tasks
- [x] Support multiple options for sorting and filtering
- [x] More sorting options: by ID, text, context, project
- [x] Preset filters

## Usage

A quick start example:

```go
import (
	todo "github.com/1set/todotxt"
)

// ...

if tasklist, err := todo.LoadFromPath("todo.txt"); err != nil {
    log.Fatal(err)
} else {
    tasks := tasklist.Filter(todo.FilterNotCompleted).Filter(todo.FilterDueToday, todo.FilterHasPriority)
    _ = tasks.Sort(todo.SortPriorityAsc, todo.SortProjectAsc)
    for i, t := range tasks {
        fmt.Println(t.Todo)
        // oh really?
        tasks[i].Complete()
    }
    if err = tasks.WriteToPath("today-todo.txt"); err != nil {
        log.Fatal(err)
    }
}
```

For more examples and details, please check the [Go Doc](https://pkg.go.dev/github.com/1set/todotxt).

## Credits

- Original [**go-todotxt**](https://github.com/JamesClonk/go-todotxt) authored by [Fabio Berchtold](https://github.com/JamesClonk)
- Cool idea about [**todo.txt**](https://github.com/todotxt/todo.txt) from [Gina Trapani](http://todotxt.org/)


todotxt

A pluggable, extensible, and opinionated set of filesystem functionality for Go across a number of filesystem types such as os, S3, and GCS.

# vfs

---

[![GitHub tag](https://img.shields.io/github/tag/c2fo/vfs.svg?style=flat)](https://github.com/c2fo/vfs/releases)
[![Build Status](https://travis-ci.org/C2FO/vfs.svg?branch=master)](https://travis-ci.org/C2FO/vfs)
[![GoDoc](https://pkg.go.dev/badge/github.com/c2fo/vfs/v6?utm_source=godoc)](https://pkg.go.dev/github.com/c2fo/vfs/v6)
[![codecov](https://codecov.io/gh/c2fo/vfs/branch/master/graph/badge.svg)](https://codecov.io/gh/c2fo/vfs)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](License.md)
[![Go Report Card](https://goreportcard.com/badge/github.com/c2fo/vfs)](https://goreportcard.com/report/github.com/c2fo/vfs)
[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)

Package vfs provides a pluggable, extensible, and opinionated set of file system
functionality for Go across a number of file system types such as os, Azure, S3, GCS
and SFTP.


### Philosophy

When building our platform, initially we wrote a library that was something to
the effect of

```go
 if config.DISK == "S3" {
 	 // do some s3 file system operation
 } else if config.DISK == "mock" {
 // fake something
 } else {
 // do some native os.xxx operation
 }
```

Not only was ugly but because the behaviors of each "file system" were 
different and we had to constantly alter the file locations and pass a bucket string (even
if the file system didn't know what a bucket was).

We found a handful of third-party libraries that were interesting but none of
them had everything we needed/wanted. Of particular inspiration was
https://github.com/spf13/afero in its composition of the super-powerful stdlib
[io.*](https://godoc.org/io) interfaces. Unfortunately, it didn't support Google Cloud Storage and there
was still a lot of passing around of strings and structs. Few, if any, of the
vfs-like libraries provided interfaces to easily and confidently create new
file system backends.

###### What we needed/wanted was the following(and more):

* self-contained set of structs that could be passed around like a file/dir handle
* the struct would represent an existing or nonexistent file/dir
* provide common (and only common) functionality across all file system so that after initialization, we don't care
 what the underlying file system is and can therefore write our code agnostically/portably
* use [io.*](https://godoc.org/io) interfaces such as [io.Reader](https://godoc.org/io#Reader) and [io.Writer](https://godoc.org/io#Writer) without needing to call a separate function
* extensibility to easily add other needed file systems like Microsoft Azure Cloud File Storage
* prefer native atomic functions when possible (ie S3 to S3 moving would use the native move api call rather than
 copy-delete)
* a uniform way of addressing files regardless of file system. This is why we use complete URI's in vfssimple
* [fmt.Stringer](https://godoc.org/fmt#Stringer) interface so that the file struct passed to a log message (or other Stringer use) would show the URI
* mockable file system
* pluggability so that third-party implementations of our interfaces could be used


### Install

Pre 1.17:
```
go get -u github.com/c2fo/vfs/v6
```

Post 1.17:
```
go install github.com/c2fo/vfs/v6
```

### Upgrading

#### Upgrading from v5 to v6
With v6.0.0, sftp.Options struct changed to to accept an array of Key Exchange algorithms rather than a string. To update, change the syntax of the auth commands.
```
 "keyExchanges":"diffie-hellman-group-a256"
```
becomes
```
 "keyExchanges":["diffie-hellman-group-a256"]
```

### Usage

We provide [vfssimple](docs/vfssimple.md) as basic way of initializing file system backends (see each
implementations's docs about authentication). [vfssimple](docs/vfssimple.md) pulls in every c2fo/vfs
backend. If you need to reduce the backend requirements (and app memory
footprint) or add a third party backend, you'll need to implement your own
"factory". See [backend](docs/backend.md) doc for more info.

You can then use those file systems to initialize locations which you'll be
referencing frequently, or initialize files directly

```go
 osFile, err := vfssimple.NewFile("file:///path/to/file.txt")
 s3File, err := vfssimple.NewFile("s3://bucket/prefix/file.txt")

 osLocation, err := vfssimple.NewLocation("file:///tmp/")
 s3Location, err := vfssimple.NewLocation("s3://bucket/")

 osTmpFile, err := osLocation.NewFile("anotherFile.txt") // file at /tmp/anotherFile.txt
```

You can perform a number of actions without any consideration for the system's api or implementation details.

```go
 osFileExists, err := osFile.Exists() // true, nil
 s3FileExists, err := s3File.Exists() // false, nil
 err = osFile.CopyToFile(s3File) // nil
 s3FileExists, err = s3File.Exists() // true, nil

 movedOsFile, err := osFile.MoveToLocation(osLocation)
 osFileExists, err = osFile.Exists() // false, nil (move actions delete the original file)
 movedOsFileExists, err := movedOsFile.Exists() // true, nil

 s3FileUri := s3File.URI() // s3://bucket/prefix/file.txt
 s3FileName := s3File.Name() // file.txt
 s3FilePath := s3File.Path() // /prefix/file.txt
```

File's [io.*](https://godoc.org/io) interfaces may be used directly:

```go
 reader := strings.NewReader("Clear is better than clever")
 gsFile, err := vfssimple.NewFile("gs://somebucket/path/to/file.txt")

 byteCount, err := io.Copy(gsFile, reader)
 err := gsFile.Close()
```

Note: [io.Copy()](https://godoc.org/io#Copy) doesn't strictly define what happens if a reader is empty. This is complicated because io.Copy
will first delegate actual copying in the following:
 1. if the io.Reader also implements io.WriterTo, WriteTo() will do the copy
 2. if the io.Writer also implements io.ReaderFrom, ReadFrom() will do the copy
 3. finally, if neither 1 or 2, io.Copy will do it's own buffered copy

In case 3, and most implementations of cases 1 and 2, if reader is empty, Write() never gets called. What that means for
vfs is there is no way for us to ensure that an empty file does or doesn't get written on an io.Copy(). For instance 
OS always creates a file, regardless of calling Write() whereas S3 must Write() and Close().

As such, vfs cannot guarantee copy behavior except in our own CopyToFile, MoveToFile, CopyToLocation, and MoveToLocation
functions. If you need to ensure a file gets copied/moved with io.Copy(), you must do so yourself OR use vfs's [utils.TouchCopy](docs/utils.md)


### Third-party Backends

 * none so far

Feel free to send a pull request if you want to add your backend to the list.

### See also:
* [vfscp](docs/vfscp.md)
* [vfssimple](docs/vfssimple.md)
* [backend](docs/backend.md)
 * [os backend](docs/os.md)
 * [gs backend](docs/gs.md)
 * [s3 backend](docs/s3.md)
 * [in-memory backend](docs/mem.md)
 * [sftp backend](docs/sftp.md)
 * [ftp backend](docs/ftp.md)
 * [azure backend](docs/azure.md) 
* [utils](docs/utils.md)

### Ideas

Things to add:
* Provide better List() functionality with more abstracted filtering and paging (iterator?) Return File structs vs URIs?
* Add better/any context.Context() support


### Contributors

Brought to you by the Enterprise Pipeline team at C2FO:

* John Judd - john.judd@c2fo.com

* Dustin Schnee - dustin.schnee@c2fo.com

* Katie Hrenchir Shields - katie.shields@c2fo.com

* Grant Higgins - grant.higgins@c2fo.com

* Pooja Dhondge - pooja.dhondge@c2fo.com

* Chris Roush – chris.roush@c2fo.com

* Anthony Ross - anthony.ross@c2fo.com

https://github.com/c2fo/

### Contributing

 1. Fork it (<https://github.com/c2fo/vfs/fork>)
 2. Create your feature branch (`git checkout -b feature/fooBar`)
 3. Commit your changes (`git commit -am 'Add some fooBar'`)
 4. Push to the branch (`git push origin feature/fooBar`)
 5. Create a new Pull Request


### License

Distributed under the MIT license. See `http://github.com/c2fo/vfs/License.md
for more information.


### Definitions

###### absolute path
- A path is said to be absolute if it provides the entire context
need to find a file, including the file system root. An absolute path must
begin with a slash and may include . and .. directories.

###### file path
- A file path ends with a filename and therefore may not end with a slash. It may be relative or absolute.

###### location path
- A location/directory path must end with a slash. It may be relative or absolute.

###### relative path
- A relative path is a way to locate a directory or file relative to
another directory. A relative path may not begin with a slash but may include .
and .. directories.

###### URI
- A Uniform Resource Identifier (URI) is a string of characters that
unambiguously identifies a particular resource. To guarantee uniformity, all
URIs follow a predefined set of syntax rules, but also maintain extensibility
through a separately defined hierarchical naming scheme (e.g. http://).

## Interfaces

#### type File

```go
type File interface {
	io.Closer
	io.Reader
	io.Seeker
	io.Writer
	fmt.Stringer

	// Exists returns boolean if the file exists on the file system. Returns an error, if any.
	Exists() (bool, error)

	// Location returns the vfs.Location for the File.
	Location() Location

	// CopyToLocation will copy the current file to the provided location.
	//
	// * Upon success, a vfs.File, representing the file at the new location, will be returned.
	// * In the case of an error, nil is returned for the file.
	// * CopyToLocation should use native functions when possible within the same scheme.
	// * If the file already exists at the location, the contents will be overwritten with the current file's contents.
	// * CopyToLocation will Close both the source and target Files which therefore can't be appended to without first
	// calling Seek() to move the cursor to the end of the file.
	CopyToLocation(location Location) (File, error)

	// CopyToFile will copy the current file to the provided file instance.
	//
	// * In the case of an error, nil is returned for the file.
	// * CopyToLocation should use native functions when possible within the same scheme.
	// * If the file already exists, the contents will be overwritten with the current file's contents.
	// * CopyToFile will Close both the source and target Files which therefore can't be appended to without first
	// calling Seek() to move the cursor to the end of the file.
	CopyToFile(file File) error

	// MoveToLocation will move the current file to the provided location.
	//
	// * If the file already exists at the location, the contents will be overwritten with the current file's contents.
	// * If the location does not exist, an attempt will be made to create it.
	// * Upon success, a vfs.File, representing the file at the new location, will be returned.
	// * In the case of an error, nil is returned for the file.
	// * When moving within the same Scheme, native move/rename should be used where possible.
	// * If the file already exists, the contents will be overwritten with the current file's contents.
	// * MoveToLocation will Close both the source and target Files which therefore can't be appended to without first
	// calling Seek() to move the cursor to the end of the file.
	MoveToLocation(location Location) (File, error)

	// MoveToFile will move the current file to the provided file instance.
	//
	// * If the file already exists, the contents will be overwritten with the current file's contents.
	// * The current instance of the file will be removed.
	// * MoveToFile will Close both the source and target Files which therefore can't be appended to without first
	// calling Seek() to move the cursor to the end of the file.
	MoveToFile(file File) error

	// Delete unlinks the File on the file system.
	Delete() error

	// LastModified returns the timestamp the file was last modified (as *time.Time).
	LastModified() (*time.Time, error)

	// Size returns the size of the file in bytes.
	Size() (uint64, error)

	// Path returns absolute path, including filename, ie /some/path/to/file.txt
	//
	// If the directory portion of a file is desired, call
	// someFile.Location().Path()
	Path() string

	// Name returns the base name of the file path.
	//
	// For file:///some/path/to/file.txt, it would return file.txt
	Name() string
	
	// Touch creates a zero-length file on the vfs.File if no File exists. Update File's last modified timestamp.
 	// Returns error if unable to touch File.
 Touch() error

	// URI returns the fully qualified absolute URI for the File. IE, s3://bucket/some/path/to/file.txt
	URI() string
}
```

File represents a file on a file system. A File may or may not actually exist on
the file system.

#### type FileSystem

```go
type FileSystem interface {
	// NewFile initializes a File on the specified volume at path 'absFilePath'.
	//
	// * Accepts volume and an absolute file path.
	// * Upon success, a vfs.File, representing the file's new path (location path + file relative path), will be returned.
	// * On error, nil is returned for the file.
	// * Note that not all file systems will have a "volume" and will therefore be "":
	// file:///path/to/file has a volume of "" and name /path/to/file
	// whereas
	// s3://mybucket/path/to/file has a volume of "mybucket and name /path/to/file
	// results in /tmp/dir1/newerdir/file.txt for the final vfs.File path.
	// * The file may or may not already exist.
	NewFile(volume string, absFilePath string) (File, error)

	// NewLocation initializes a Location on the specified volume with the given path.
	//
	// * Accepts volume and an absolute location path.
	// * The file may or may not already exist. Note that on key-store file systems like S3 or GCS, paths never truly exist.
	// * On error, nil is returned for the location.
	//
	// See NewFile for note on volume.
	NewLocation(volume string, absLocPath string) (Location, error)

	// Name returns the name of the FileSystem ie: Amazon S3, os, Google Cloud Storage, etc.
	Name() string	

	// Scheme returns the uri scheme used by the FileSystem: s3, file, gs, etc.
	Scheme() string

	// Retry will return the retry function to be used by any file system.
	Retry() Retry
}
```

FileSystem represents a file system with any authentication accounted for.

#### type Location

```go
type Location interface {
	// String returns the fully qualified absolute URI for the Location. IE, file://bucket/some/path/
	fmt.Stringer

	// List returns a slice of strings representing the base names of the files found at the Location.
	//
	// * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	// * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	// be checked first.
	List() ([]string, error)

	// ListByPrefix returns a slice of strings representing the base names of the files found in Location whose filenames
	// match the given prefix.
	//
	// * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	// * "relative" prefixes are allowed, ie, listByPrefix from "/some/path/" with prefix "to/somepattern" is the same as
	// location "/some/path/to/" with prefix of "somepattern"
	// * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	// be checked first.
	ListByPrefix(prefix string) ([]string, error)

	// ListByRegex returns a slice of strings representing the base names of the files found in Location that matched the
	// given regular expression.
	//
	// * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	// * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	// be checked first.
	ListByRegex(regex *regexp.Regexp) ([]string, error)

	// Volume returns the volume as string. In URI parlance, volume equates to authority.
	// For example s3://mybucket/path/to/file.txt, volume would return "mybucket".
	//
	// Note: Some file systems may not have a volume and will return "".
	Volume() string

	// Path returns absolute location path, ie /some/path/to/. An absolute path must be resolved to it's shortest path:
	// see path.Clean
	Path() string

	// Exists returns boolean if the location exists on the file system. Returns an error if any.
	Exists() (bool, error)

	// NewLocation is an initializer for a new Location relative to the existing one.
	//
	// Given location:
	// loc := fs.NewLocation(:s3://mybucket/some/path/to/")
	// calling:
	// newLoc := loc.NewLocation("../../")
	// would return a new vfs.Location representing:
	// s3://mybucket/some/
	//
	// * Accepts a relative location path.
	NewLocation(relLocPath string) (Location, error)

	// Given location:
	// 	 loc := fs.NewLocation("file:///some/path/to/")
	// calling:
	// loc.ChangeDir("../../")
	// would update the current location instance to
	// file:///some/.
	//
	// * ChangeDir accepts a relative location path.
	ChangeDir(relLocPath string) error

	//FileSystem returns the underlying vfs.FileSystem struct for Location.
	FileSystem() FileSystem

	// NewFile will instantiate a vfs.File instance at or relative to the current location's path.
	//
	// * Accepts a relative file path.
	// * In the case of an error, nil is returned for the file.
	// * Resultant File path will be the shortest path name equivalent of combining the Location path and relative path, if any.
	// ie, /tmp/dir1/ as location and relFilePath "newdir/./../newerdir/file.txt"
	// results in /tmp/dir1/newerdir/file.txt for the final vfs.File path.
	// * Upon success, a vfs.File, representing the file's new path (location path + file relative path), will be returned.
	// * The file may or may not already exist.
	NewFile(relFilePath string) (File, error)

	// DeleteFile deletes the file of the given name at the location.
	//
	// This is meant to be a short cut for instantiating a new file and calling delete on that, with all the necessary
	// error handling overhead.
	//
	// * Accepts relative file path.
	DeleteFile(relFilePath string) error

	// URI returns the fully qualified absolute URI for the Location. IE, s3://bucket/some/path/
	//
	// URI's for locations must always end with a slash.
	URI() string
}
```

Location represents a file system path which serves as a start point for
directory-like functionality. A location may or may not actually exist on the
file system.

#### type Options

```go
type Options interface{}
```

Options are structs that contain various options specific to the file system

#### type Retry

```go
type Retry func(wrapped func() error) error
```

Retry is a function that can be used to wrap any operation into a definable
retry operation. The wrapped argument is called by the underlying VFS
implementation.

Ex:
```go
 var retrier Retry = func(wrapped func() error) error {
 var ret error
 for i := 0; i < 5; i++ {
 if err := wrapped(); err != nil { ret = err; continue }
 }
 return ret
 }
```
#### func DefaultRetryer

```go
func DefaultRetryer() Retry
```
DefaultRetryer returns a no-op retryer which simply calls the wrapped command
without looping.

The Best Go Libraries For File Handling (25)