ghorg

About ghorg

This large README is shown as a plain-text preview. Read the full README at the source

# ghorg

<a href="https://godoc.org/github.com/gabrie30/ghorg"><img src="https://godoc.org/github.com/gabrie30/ghorg?status.svg" alt="GoDoc"></a> [![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![WakeMeOps](https://docs.wakemeops.com/badges/ghorg.svg)](https://docs.wakemeops.com//packages/ghorg)

Pronounced [gore-guh]; similar to [gorge](https://www.dictionary.com/browse/gorge). You can use ghorg to gorge on orgs.

Use ghorg to quickly clone all of an orgs, or users repos into a single directory. This can be useful in many situations including

1. Searching an orgs/users codebase with ack, silver searcher, grep etc..
1. Bash scripting
1. Creating backups
1. Onboarding new team members (cloning all team repos)
1. Performing Audits

> With default configuration ghorg performs two actions.
> 1. Will clone a repo if its not inside the clone directory.
> 2. If repo does exists locally in the clone directory it will perform a git pull and git clean on the repo.

> So when running ghorg a second time on the same org/user, all local changes in the cloned directory by default will be overwritten by what's on GitHub. If you want to work out of this directory, make sure you either rename the directory or set the `--no-clean` flag on all future clones to prevent losing your changes locally.

<p align="center">
  <img width="648" alt="ghorg cli example" src="https://user-images.githubusercontent.com/1512282/63229247-5459f880-c1b3-11e9-9e5d-d20723046946.png">
</p>

## Supported Providers
- GitHub (Self Hosted & Cloud)
  - [Install](https://github.com/gabrie30/ghorg#installation) | [Setup](https://github.com/gabrie30/ghorg#github-setup) | [Examples](https://github.com/gabrie30/ghorg/blob/master/examples/github.md)
- GitLab (Self Hosted & Cloud)
  - [Install](https://github.com/gabrie30/ghorg#installation) | [Setup](https://github.com/gabrie30/ghorg#gitlab-setup)  | [Examples](https://github.com/gabrie30/ghorg/blob/master/examples/gitlab.md)
- Bitbucket (Cloud & Self-hosted Server)
  - [Install](https://github.com/gabrie30/ghorg#installation) | [Setup](https://github.com/gabrie30/ghorg#bitbucket-setup)  | [Examples](https://github.com/gabrie30/ghorg/blob/master/examples/bitbucket.md)
- Gitea (Self Hosted Only)
  - [Install](https://github.com/gabrie30/ghorg#installation) | [Setup](https://github.com/gabrie30/ghorg#gitea-setup)  | [Examples](https://github.com/gabrie30/ghorg/blob/master/examples/gitea.md)
- Codeberg (Cloud & Self Hosted Forgejo)
  - [Install](https://github.com/gabrie30/ghorg#installation) | [Setup](https://github.com/gabrie30/ghorg#codeberg-setup)  | [Examples](https://github.com/gabrie30/ghorg/blob/master/examples/codeberg.md)
- Sourcehut (Limited Features)
  - [Install](https://github.com/gabrie30/ghorg#installation) | [Setup](https://github.com/gabrie30/ghorg#sourcehut-setup)  | [Examples](https://github.com/gabrie30/ghorg/blob/master/examples/sourcehut.md)

> The terminology used in ghorg is that of GitHub, mainly orgs/repos. GitLab and BitBucket use different terminology. There is a handy chart thanks to GitLab that translates terminology [here](https://about.gitlab.com/images/blogimages/gitlab-terminology.png). Note, some features may be different for certain providers.

## High Level Features

- [Filter](#selective-repository-cloning) or select specific repositories for cloning
- Create [backups](#creating-backups) of repositories
- Simplify complex clone commands using [reclone](#reclone-command) shortcuts
- Update every repo in a directory with [pull](#pull-command), without discarding local changes
- Initiate clone operations via [HTTP server](https://github.com/gabrie30/ghorg/blob/master/examples/reclone-server.md)
- Schedule cloning tasks using [cron](https://github.com/gabrie30/ghorg/blob/master/examples/reclone-cron.md)
- Monitor and track clone [metrics](#tracking-clone-data-over-time) over time

## Installation

There are a installation methods available, please choose the one that suits your fancy:
- [Prebuilt Binaries](#prebuilt-binaries)
- [Homebrew](#homebrew)
- [Mise](#mise)
- [Golang](#golang)
- [Scoop](#scoop)
- [Docker](#docker)

For each installation method, optionally create a ghorg configuration file. See the [configuration](#configuration) section for more details.

```bash
mkdir -p $HOME/.config/ghorg
curl https://raw.githubusercontent.com/gabrie30/ghorg/master/sample-conf.yaml > $HOME/.config/ghorg/conf.yaml
vi $HOME/.config/ghorg/conf.yaml # To update your configuration
```

### Prebuilt Binaries

See [latest release](https://github.com/gabrie30/ghorg/releases/latest) to download directly for

- Mac (Darwin)
- Windows
- Linux

If you don't know which to choose its likely going to be the x86_64 version for your operating system.

### Homebrew

```bash
brew install ghorg
```

### Mise

If you use [Mise](https://github.com/jdx/mise), you can install the latest version of `ghorg` on Linux/MacOS/Windows:

```bash
mise use -g ghorg@latest
```

### Golang

Ensure `$HOME/go/bin` is in your path

```bash
go install github.com/gabrie30/ghorg@latest
```

### Scoop

Windows users can also install ghorg using [scoop](https://scoop.sh/#/)

  ```
  scoop bucket add main
  scoop install ghorg
  ```

## Configuration

Precedence for configuration is first given to the flags set on the command-line, then to what's set in your `$HOME/.config/ghorg/conf.yaml`. This file comes from the [sample-conf.yaml](https://github.com/gabrie30/ghorg/blob/master/sample-conf.yaml) and can be installed by performing the following.

```bash
mkdir -p $HOME/.config/ghorg
curl https://raw.githubusercontent.com/gabrie30/ghorg/master/sample-conf.yaml > $HOME/.config/ghorg/conf.yaml
vi $HOME/.config/ghorg/conf.yaml # To update your configuration
```

If no configuration file is found ghorg will use its defaults and try to clone a GitHub Org, however an api token is always required.

You can have multiple configuration files which is useful if you clone from multiple SCM providers with different tokens and settings. Alternative configuration files can only be referenced as a command-line flag `--config`.

If you have multiple different orgs/users/configurations to clone see the `ghorg reclone` command as a way to manage them.

Note: ghorg will respect the `XDG_CONFIG_HOME` [environment variable](https://wiki.archlinux.org/title/XDG_Base_Directory) if set.

## SCM Provider Setup

> Note: if you are running into issues, read the troubleshooting and known issues section below

### GitHub Setup
1. Create [Personal Access Token](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line) with all `repo` scopes. Update `GHORG_GITHUB_TOKEN` in your `ghorg/conf.yaml` or as a cli flag or place it in a file and add the path to `GHORG_GITHUB_TOKEN`. If your org has Saml SSO in front you will need to give your token those permissions as well, see [this doc](https://docs.github.com/en/github/authenticating-to-github/authenticating-with-saml-single-sign-on/authorizing-a-personal-access-token-for-use-with-saml-single-sign-on).
1. For cloning GitHub Enterprise (self hosted github instances) repos you must set `--base-url` e.g. `ghorg clone <github_org> --base-url=https://internal.github.com`
1. See [examples/github.md](https://github.com/gabrie30/ghorg/blob/master/examples/github.md) on how to run

#### GitHub App Authentication (Advanced)

1. [Create a GitHub App](https://docs.github.com/en/apps/creating-github-apps/setting-up-a-github-app/creating-a-github-app) in your Organization. You only need to fill out the required fields. Make sure to give Repository Permissions ->  contents -> read only permissions
1. Install the GitHub App into your Organization
1. Generate a a private key from the GitHub App, set the location of the key to `GHORG_GITHUB_APP_PEM_PATH`
1. Locate the GitHub App ID from the GitHub App, set the value to `GHORG_GITHUB_APP_ID`
1. Locate the GitHub Installation ID from the URL of the GitHub app, set the value to `GHORG_GITHUB_APP_INSTALLATION_ID`. NOTE: you will need to use the actual GitHub url to get this ID, go to your GitHub Organization Settings Page -> Third Party Access -> GitHub Apps -> Configure -> Get ID from URL

### GitLab Setup

1. Create [Personal Access Token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with the `read_api` scope (or `api` for self-managed GitLab older than 12.10). This token can be added to your `ghorg/conf.yaml` or as a cli flag.
1. Update the `GitLab Specific` config in your `ghorg/conf.yaml` or via cli flags or place it in a file and add the path to `GHORG_GITLAB_TOKEN`
1. Update `GHORG_SCM_TYPE` to `gitlab` in your `ghorg/conf.yaml` or via cli flags
1. See [examples/gitlab.md](https://github.com/gabrie30/ghorg/blob/master/examples/gitlab.md) on how to run

### Gitea Setup

1. Create [Access Token](https://docs.gitea.io/en-us/api-usage/) (Settings -> Applications -> Generate Token)
1. Update `GHORG_GITEA_TOKEN` in your `ghorg/conf.yaml` or use the (--token, -t) flag or place it in a file and add the path to `GHORG_GITEA_TOKEN`.
1. Update `GHORG_SCM_TYPE` to `gitea` in your `ghorg/conf.yaml` or via cli flags
1. See [examples/gitea.md](https://github.com/gabrie30/ghorg/blob/master/examples/gitea.md) on how to run

### Codeberg Setup

[Codeberg](https://codeberg.org) runs [Forgejo](https://forgejo.org), which is API-compatible with Gitea, so ghorg reuses its Gitea backend and defaults the base URL to `https://codeberg.org` (no `--base-url` required). Self-hosted Forgejo instances are also supported by setting `--base-url`; instances served over http additionally require `--insecure-codeberg-client`.

1. Create an [Access Token](https://codeberg.org/user/settings/applications) (Settings -> Applications -> Generate Token) with at least the `read:organization` and `read:repository` scopes
1. Update `GHORG_CODEBERG_TOKEN` in your `ghorg/conf.yaml` or use the (--token, -t) flag or place it in a file and add the path to `GHORG_CODEBERG_TOKEN`.
1. Update `GHORG_SCM_TYPE` to `codeberg` in your `ghorg/conf.yaml` or via cli flags
1. See [examples/codeberg.md](https://github.com/gabrie30/ghorg/blob/master/examples/codeberg.md) on how to run

### Sourcehut Setup

1. Create a [Personal Access Token](https://meta.sr.ht/oauth2). Click "Limit scope of access grant", check "Generate read-only access token", then ctrl-click the REPOSITORIES and OBJECTS permissions.
1. Ensure you have added an SSH key if you want to clone private repos (sourcehut does not accept PATs in https URLs)
1. Update `GHORG_SOURCEHUT_TOKEN` in your `ghorg/conf.yaml` or use the (--token, -t) flag or place it in a file and add the path to `GHORG_SOURCEHUT_TOKEN`.
1. Update `GHORG_SCM_TYPE` to `sourcehut` in your `ghorg/conf.yaml` or via cli flags

> **Note on usernames**: You can specify sourcehut usernames with or without the `~` prefix (e.g., both `ghorg clone username` and `ghorg clone ~username` work). Local folder paths will never include the `~` prefix to avoid shell expansion issues.

> **For detailed examples, API limitations, and sourcehut-specific features, see [examples/sourcehut.md](https://github.com/gabrie30/ghorg/blob/master/examples/sourcehut.md)**

### Bitbucket Setup

> Note: ghorg supports both Bitbucket Cloud and Bitbucket Server (self-hosted instances)

#### API Tokens (Recommended for Bitbucket Cloud)

Bitbucket has deprecated App Passwords in favor of API Tokens. This is the recommended authentication method for Bitbucket Cloud.

1. Create an [API token](https://support.atlassian.com/bitbucket-cloud/docs/create-an-api-token/) from your Atlassian account settings
1. **Important**: When creating the token, grant **all read scopes** (Account: Read, Workspace membership: Read, Projects: Read, Repositories: Read, etc.) to ensure ghorg can list and clone repositories
1. Set `GHORG_BITBUCKET_API_TOKEN` in your `$HOME/.config/ghorg/conf.yaml` or use the `--token` flag
1. Set `GHORG_BITBUCKET_API_EMAIL` to your Atlassian account email (or use `--bitbucket-api-email`)
1. Update SCM type to `bitbucket` in your `ghorg/conf.yaml` or via cli flags
1. See [examples/bitbucket.md](https://github.com/gabrie30/ghorg/blob/master/examples/bitbucket.md) on how to run

> Note: When using API tokens, ghorg automatically uses `x-bitbucket-api-token-auth` as the Git username for clone operations, as required by Bitbucket's API token authentication.

#### App Passwords (Legacy)

> Note: Bitbucket has deprecated App Passwords. Consider using API Tokens instead.

#### PAT/OAuth token

1. Create a [PAT](https://confluence.atlassian.com/bitbucketserver/personal-access-tokens-939515499.html)
1. Set the token with `GHORG_BITBUCKET_OAUTH_TOKEN` in your `$HOME/.config/ghorg/conf.yaml` or using the `--token` flag. Make sure you do not have `--bitbucket-username` set.
1. Update SCM TYPE to `bitbucket` in your `ghorg/conf.yaml` or via cli flags
1. See [examples/bitbucket.md](https://github.com/gabrie30/ghorg/blob/master/examples/bitbucket.md) on how to run

#### Bitbucket Server (Self-hosted)

1. To configure with Bitbucket Server you will need to provide your instance URL via `GHORG_SCM_BASE_URL` in your `$HOME/.config/ghorg/conf.yaml` or use the `--base-url` flag.
1. Create credentials (username/password or app password) and update your configuration or use the `--bitbucket-username` and `--token` flags.
1. For insecure connections (HTTP), set `GHORG_INSECURE_BITBUCKET_CLIENT=true`
1. Update [SCM type](https://github.com/gabrie30/ghorg/blob/master/sample-conf.yaml#L54-L57) to `bitbucket` in your `ghorg/conf.yaml` or via cli flags
1. See [examples/bitbucket.md](https://github.com/gabrie30/ghorg/blob/master/examples/bitbucket.md) on how to run

## How to Use

See [examples](https://github.com/gabrie30/ghorg/tree/master/examples) directory for more SCM specific docs or use the examples command e.g. `ghorg examples gitlab`

```bash
$ ghorg clone kubernetes --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
# Example how to use --token with a file path
$ ghorg clone kubernetes --token=~/.config/ghorg/gitlab-token.txt
$ ghorg clone davecheney --clone-type=user --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
$ ghorg clone gitlab-examples --scm=gitlab --preserve-dir --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
$ ghorg clone gitlab-examples/wayne-enterprises --scm=gitlab --token=bGVhdmUgYSBjb21tZW50IG9uIGlzc3VlIDY2
$ ghorg clone all-groups --scm=gitlab --base-url=https://gitlab.internal.yourcompany.com --preserve-dir
$ ghorg clone --help
# view cloned resources
$ ghorg ls
$ ghorg ls someorg
$ ghorg ls someorg | xargs -I %s mv %s bar/
```

## Changing Clone Directories

1. By default ghorg will clone the org or user repos into a directory like `$HOME/ghorg/org`. If you want to clone the org to a different directory use the `--path` flag or set `GHORG_ABSOLUTE_PATH_TO_CLONE_TO` in your ghorg conf. **This value must be an absolute path**. For example if you wanted to clone the kubernetes org to `/tmp/ghorg` you would run the following command.

    ```
    $ ghorg clone kubernetes --path=/tmp/ghorg
    ```
    which would create...

    ```
    /tmp/ghorg
    └── kubernetes
        ├── apimachinery
        ├── gengo
        ├── git-sync
        ├── kubeadm
        ├── kubernetes-template-project
        ├── ...
    ```

1. If you want to change the name of the directory the repos get cloned into, set the `GHORG_OUTPUT_DIR` in your ghorg conf or set the `--output-dir` flag. For example to clone only the repos starting with `sig-` from the kubernetes org into a direcotry called `kubernetes-sig-only`. You would run the following command.

    ```
    $ ghorg clone kubernetes --match-regex=^sig- --output-dir=kubernetes-sig-only
    ```
    which would create...

    ```
    $HOME/ghorg
    └── kubernetes-sig-only
        ├── sig-release
        ├── sig-security
        └── sig-testing
    ```

## Selective Repository Cloning

Ghorg provides several optional ways to narrow down which repositories get cloned. Filters are applied in this order: **flag-based filters → `--target-repos-path` → `ghorgonly` → `ghorgignore` → `--repo-filter-hook`**. They can work in combination for a fine-grained control.

### Flag-based filters

- **Match by regex**: use `--match-regex` to include, or `--exclude-match-regex` to exclude, repos whose names match a regex.
- **Match by prefix**: use `--match-prefix` to include, or `--exclude-match-prefix` to exclude, repos whose names start with one or more prefixes.
- **Skip archived repos**: use `--skip-archived` (not supported on Bitbucket).
- **Skip forked repos**: use `--skip-forks`.
- **Filter by topic**: use `--topics` (or `GHORG_TOPICS`) to clone only repos tagged with a matching [topic](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/classifying-your-repository-with-topics). GitHub, GitLab, and Gitea only.

#### `--target-repos-path` - explicit list of repo names

Maintain a file containing the exact **repository names** you want to clone (one per line) and point at it with `--target-repos-path` (or the `GHORG_TARGET_REPOS_PATH` env var). Only repos whose name appears in the file will be cloned. This is the right choice when you have a fixed, known set of repos e.g. a curated list driving a CI pipeline or a documented backup set.

- Matching is done against the **repo name only** (the basename of the clone URL with `.git` stripped) and is **case-insensitive and exact** — partial names will not match.
- There is no default location; you must always pass an explicit path via the flag or env var.
- Names listed in the file that don't exist on the org/user are reported in the clone summary so you can catch typos or repos that were renamed/removed.
- When `--clone-wiki` is enabled, each listed name will also match its corresponding `.wiki` repo. GitLab snippets are matched against their parent repo name.

#### `ghorgonly` - auto-detected substring allowlist file

`ghorgonly` is a plain-text file you create yourself. If a file exists at `$HOME/.config/ghorg/ghorgonly`, ghorg will pick it up automatically on every clone, no flag required, and only clone repos whose clone URL **contains** one of the listed substrings (one pattern per line). This is useful when you want a dynamic, pattern-based subset of a large organization e.g. "everything published under the `infra-` namespace" without having to enumerate every repo name by hand.

- Matching is a plain substring check against the **full clone URL**, so patterns like `infra-`, `internal/platform/`, or a full URL all work.
- If no file exists at the default path, this filter is skipped entirely.
- To use a different location, or to maintain **several `ghorgonly` files for different clone scenarios** (e.g. one per team or per environment), pass `--ghorgonly-path` (or set `GHORG_ONLY_PATH`) to select the right file for that run.
- `ghorgonly` is applied **after** `--target-repos-path` and **before** `ghorgignore`, so the three can be layered (e.g., target a known list, narrow further by substring, then drop a few specific repos).

**`ghorgonly` vs. `--target-repos-path`:** use `--target-repos-path` when you know the **exact set of repo names** you want and want missing-repo warnings; use `ghorgonly` when you want a **pattern-based subset** of the org's URLs and don't need an explicit list.

#### `ghorgignore` - auto-detected denylist file

Frequently Asked Questions