A versatile ETL library that parses text input (CSV/txt/JSON/XML/EDI/X12/EDIFACT/etc) in streaming fashion and transforms data into JSON output using data-driven schema.

# omniparser
![CI](https://github.com/jf-tech/omniparser/workflows/CI/badge.svg)
[![codecov](https://codecov.io/gh/jf-tech/omniparser/branch/master/graph/badge.svg)](https://codecov.io/gh/jf-tech/omniparser)
[![Go Report Card](https://goreportcard.com/badge/github.com/jf-tech/omniparser)](https://goreportcard.com/report/github.com/jf-tech/omniparser)
[![PkgGoDev](https://pkg.go.dev/badge/github.com/jf-tech/omniparser)](https://pkg.go.dev/github.com/jf-tech/omniparser)
[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)

Omniparser is a native Golang ETL parser that ingests input data of various formats (**CSV, txt, fixed length/width,
XML, EDI/X12/EDIFACT, JSON**, and custom formats) in streaming fashion and transforms data into desired JSON output
based on a schema written in JSON.

Min Golang Version: 1.16

## Licenses and Sponsorship
Omniparser is publicly available under [MIT License](./LICENSE).
[Individual and corporate sponsorships](https://github.com/sponsors/jf-tech/) are welcome and gratefully
appreciated, and will be listed in the [SPONSORS](./sponsors/SPONSORS.md) page.
[Company-level sponsors](https://github.com/sponsors/jf-tech/) get additional benefits and supports
granted in the [COMPANY LICENSE](./sponsors/COMPANY_LICENSE.md).

## Documentation

Docs:
- [Getting Started](./doc/gettingstarted.md): a tutorial for writing your first omniparser schema.
- [IDR](./doc/idr.md): in-memory data representation of ingested data for omniparser.
- [XPath Based Record Filtering and Data Extraction](./doc/xpath.md): xpath queries are essential to omniparser schema
writing. Learn the concept and tricks in depth.
- [All About Transforms](./doc/transforms.md): everything about `transform_declarations`.
- [Use of `custom_func`, Specially `javascript`](./doc/use_of_custom_funcs.md): An in depth look of how `custom_func`
is used, specially the all mighty `javascript` (and `javascript_with_context`).
- [CSV Schema in Depth](./doc/csv2_in_depth.md): everything about schemas for CSV input.
- [Fixed-Length Schema in Depth](./doc/fixedlength2_in_depth.md): everything about schemas for fixed-length (e.g. TXT)
input
- [JSON/XML Schema in Depth](./doc/json_xml_in_depth.md): everything about schemas for JSON or XML input.
- [EDI Schema in Depth](./doc/edi_in_depth.md): everything about schemas for EDI input.
- [Programmability](./doc/programmability.md): Advanced techniques for using omniparser (or some of its components) in
your code.

References:
- [Custom Functions](./doc/customfuncs.md): a complete reference of all built-in custom functions.

Examples:
- [CSV Examples](extensions/omniv21/samples/csv2)
- [Fixed-Length Examples](extensions/omniv21/samples/fixedlength2)
- [JSON Examples](extensions/omniv21/samples/json)
- [XML Examples](extensions/omniv21/samples/xml).
- [EDI Examples](extensions/omniv21/samples/edi).
- [Custom File Format](extensions/omniv21/samples/customfileformats/jsonlog)
- [Custom Funcs](extensions/omniv21/samples/customfuncs)

In the example folders above you will find pairs of input files and their schema files. Then in the
`.snapshots` sub directory, you'll find their corresponding output files.

## Online Playground (not functioning)

~~Use [The Playground](https://omniparser-prod-omniparser-qd0sj4.mo2.mogenius.io/)  (may need to wait for a few seconds for instance to wake up)
for trying out schemas and inputs, yours or existing samples, to see how ingestion and transform work.~~

As for now (2023/03/14), all of our previous free docker hosting solutions went away and we haven't found another one yet. For now please clone the repo and use `./cli.sh` as described in the [Getting Started](./doc/gettingstarted.md) page.

## Why
- No good ETL transform/parser library exists in Golang.
- Even looking into Java and other languages, choices aren't many and all have limitations:
    - [Smooks](https://www.smooks.org/) is dead, plus its EDI parsing/transform is too heavyweight, needing code-gen.
    - [BeanIO](http://beanio.org/) can't deal with EDI input.
    - [Jolt](https://github.com/bazaarvoice/jolt) can't deal with anything other than JSON input.
    - [JSONata](https://jsonata.org/) still only JSON -> JSON transform.
- Many of the parsers/transforms don't support streaming read, loading entire input into memory - not acceptable in some
situations.

## Requirements
- Golang 1.16 or later.

## Recent Major Feature Additions/Changes
- 2024/06: v1.0.5 released: **upgraded minimum go version to 1.16**; enabled full ES6 feature support in javascript custom function.
- 2022/09: v1.0.4 released: added `csv2` file format that supersedes the original `csv` format with support of hierarchical and nested records.
- 2022/09: v1.0.3 released: added `fixedlength2` file format that supersedes the original `fixed-length` format with support of hierarchical and nested envelopes.
- 1.0.0 Released!
- Added `Transform.RawRecord()` for caller of omniparser to access the raw ingested record.
- Deprecated `custom_parse` in favor of `custom_func` (`custom_parse` is still usable for
back-compatibility, it is just removed from all public docs and samples).
- Added `NonValidatingReader` EDI segment reader.
- Added fixed-length file format support in omniv21 handler.
- Added EDI file format support in omniv21 handler.
- Major restructure/refactoring
    - Upgrade omni schema version to `omni.2.1` due a number of incompatible schema changes:
        - `'result_type'` -> `'type'`
        - `'ignore_error_and_return_empty_str` -> `'ignore_error'`
        - `'keep_leading_trailing_space'` -> `'no_trim'`
    - Changed how we handle custom functions: previously we always use strings as in param type as well as result param
    type. Not anymore, all types are supported for custom function in and out params.
    - Changed the way we package custom functions for extensions: previously we collected custom functions from all
    extensions and then passed all of them to the extension that is used; this feels weird, now only the custom
    functions included in a particular extension are used in that extension.
    - Deprecated/removed most of the custom functions in favor of using 'javascript'.
    - A number of package renaming.
- Added CSV file format support in omniv2 handler.
- Introduced IDR node cache for allocation recycling.
- Introduced [IDR](./doc/idr.md) for in-memory data representation.
- Added trie based high performance `times.SmartParse`.
- Command line interface (one-off `transform` cmd or long-running http `server` mode).
- `javascript` engine integration as a custom_func.
- JSON stream parser.
- Extensibility:
    - Ability to provide custom functions.
    - Ability to provide custom schema handler.
    - Ability to customize the built-in omniv2 schema handler's parsing code.
    - Ability to provide a new file format support to built-in omniv2 schema handler.

## Footnotes
- omniparser is a collaboration effort of [jf-tech](https://github.com/jf-tech/),[Simon](https://github.com/liangxibing)
and [Steven](http://github.com/wangjia007bond).


omniparser

A message streaming bridge between a range of protocols.

Redpanda Connect
================

[![Build Status][actions-badge]][actions-url]

API for Apache V2 builds: [![godoc for redpanda-data/connect ASL][godoc-badge]][godoc-url-apache]

API for Enterprise builds: [![godoc for redpanda-data/connect RCL][godoc-badge]][godoc-url-enterprise]

Redpanda Connect is a high performance and resilient stream processor, able to connect various [sources][inputs] and [sinks][outputs] in a range of brokering patterns and perform [hydration, enrichments, transformations and filters][processors] on payloads.

It comes with a [powerful mapping language][bloblang-about], is easy to deploy and monitor, and ready to drop into your pipeline either as a static binary or docker image, making it cloud native as heck.

Redpanda Connect is declarative, with stream pipelines defined in as few as a single config file, allowing you to specify connectors and a list of processing stages:

```yaml
input:
  gcp_pubsub:
    project: foo
    subscription: bar

pipeline:
  processors:
    - mapping: |
        root.message = this
        root.meta.link_count = this.links.length()
        root.user.age = this.user.age.number()

output:
  redis_streams:
    url: tcp://TODO:6379
    stream: baz
    max_in_flight: 20
```

### Delivery Guarantees

Delivery guarantees [can be a dodgy subject](https://youtu.be/QmpBOCvY8mY). Redpanda Connect processes and acknowledges messages using an in-process transaction model with no need for any disk persisted state, so when connecting to at-least-once sources and sinks it's able to guarantee at-least-once delivery even in the event of crashes, disk corruption, or other unexpected server faults.

This behaviour is the default and free of caveats, which also makes deploying and scaling Redpanda Connect much simpler.

## Supported Sources & Sinks

AWS (DynamoDB, Kinesis, S3, SQS, SNS), Azure (Blob storage, Queue storage, Table storage), GCP (Pub/Sub, Cloud storage, Big query), Kafka, NATS (JetStream, Streaming), NSQ, MQTT, AMQP 0.91 (RabbitMQ), AMQP 1, Redis (streams, list, pubsub, hashes), Cassandra, Elasticsearch, HDFS, HTTP (server and client, including websockets), MongoDB, SQL (MySQL, PostgreSQL, Clickhouse, MSSQL), and [you know what just click here to see them all, they don't fit in a README][about-categories].

## Documentation

If you want to dive fully into Redpanda Connect then don't waste your time in this dump, check out the [documentation site][general-docs].

For guidance on building your own custom plugins in Go check out [the public APIs](https://pkg.go.dev/github.com/redpanda-data/benthos/v4/public/service).

## Install

Install on Linux:

```shell
curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
unzip rpk-linux-amd64.zip -d ~/.local/bin/
```

Or use Homebrew:

```shell
brew install redpanda-data/tap/redpanda
```

Or pull the docker image:

```shell
docker pull docker.redpanda.com/redpandadata/connect
```

For more information check out the [getting started guide][getting-started].

## Run

```shell
rpk connect run ./config.yaml
```

Or, with docker:

```shell
# Using a config file
docker run --rm -v /path/to/your/config.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run

# Using a series of -s flags
docker run --rm -p 4195:4195 docker.redpanda.com/redpandadata/connect run \
  -s "input.type=http_server" \
  -s "output.type=kafka" \
  -s "output.kafka.addresses=kafka-server:9092" \
  -s "output.kafka.topic=redpanda_topic"
```

## Monitoring

### Health Checks

Redpanda Connect serves two HTTP endpoints for health checks:
- `/ping` can be used as a liveness probe as it always returns a 200.
- `/ready` can be used as a readiness probe as it serves a 200 only when both the input and output are connected, otherwise a 503 is returned.

### Metrics

Redpanda Connect [exposes lots of metrics][metrics] either to Statsd, Prometheus, a JSON HTTP endpoint, [and more][metrics].

### Tracing

Redpanda Connect also [emits open telemetry tracing events][tracers], which can be used to visualise the processors within a pipeline.

## Configuration

Redpanda Connect provides lots of tools for making configuration discovery, debugging and organisation easy. You can [read about them here][config-doc].

## Build

Build with Go (any [currently supported version](https://go.dev/dl/)):

```shell
git clone git@github.com:redpanda-data/connect
cd connect
make
```

## Lint

Redpanda Connect uses [golangci-lint][golangci-lint] for linting, which you can install with:

```shell
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin
```

And then run it with `make lint`.

## Plugins

It's pretty easy to write your own custom plugins for Redpanda Connect in Go, for information check out [the API docs][godoc-url], and for inspiration there's an [example repo][plugin-repo] demonstrating a variety of plugin implementations.

## Extra Plugins

By default Redpanda Connect does not build with components that require linking to external libraries, such as the `zmq4` input and outputs. If you wish to build Redpanda Connect locally with these dependencies then set the build tag `x_benthos_extra`:

```shell
# With go
go install -tags "x_benthos_extra" github.com/redpanda-data/connect/v4/cmd/redpanda-connect@latest

# Using make
make TAGS=x_benthos_extra
```

Note that this tag may change or be broken out into granular tags for individual components outside of major version releases. If you attempt a build and these dependencies are not present you'll see error messages such as `ld: library not found for -lzmq`.

## Docker Builds

There's a multi-stage `Dockerfile` for creating a Redpanda Connect docker image which results in a minimal image from scratch. You can build it with:

```shell
make docker
```

Then use the image:

```shell
docker run --rm \
	-v /path/to/your/benthos.yaml:/config.yaml \
	-v /tmp/data:/data \
	-p 4195:4195 \
	docker.redpanda.com/redpandadata/connect run /config.yaml
```

## Contributing

Contributions are welcome, please [read the guidelines](CONTRIBUTING.md).

[inputs]: https://docs.redpanda.com/redpanda-connect/components/inputs/about
[about-categories]: https://docs.redpanda.com/redpanda-connect/about#components
[processors]: https://docs.redpanda.com/redpanda-connect/components/processors/about
[outputs]: https://docs.redpanda.com/redpanda-connect/components/outputs/about
[metrics]: https://docs.redpanda.com/redpanda-connect/components/metrics/about
[tracers]: https://docs.redpanda.com/redpanda-connect/components/tracers/about
[config-interp]: https://docs.redpanda.com/redpanda-connect/configuration/interpolation
[streams-api]: https://docs.redpanda.com/redpanda-connect/guides/streams_mode/streams_api
[streams-mode]: https://docs.redpanda.com/redpanda-connect/guides/streams_mode/about
[general-docs]: https://docs.redpanda.com/redpanda-connect/about
[bloblang-about]: https://docs.redpanda.com/redpanda-connect/guides/bloblang/about
[config-doc]: https://docs.redpanda.com/redpanda-connect/configuration/about
[releases]: https://github.com/redpanda-data/connect/releases
[plugin-repo]: https://github.com/benthosdev/benthos-plugin-example
[getting-started]: https://docs.redpanda.com/redpanda-connect/guides/getting_started
[benthos-studio]: https://studio.benthos.dev

[godoc-badge]: https://pkg.go.dev/badge/github.com/redpanda-data/benthos/v4/public
[godoc-url-apache]: https://pkg.go.dev/github.com/redpanda-data/connect/public/bundle/free/v4
[godoc-url-enterprise]: https://pkg.go.dev/github.com/redpanda-data/connect/public/bundle/enterprise/v4
[actions-badge]: https://github.com/redpanda-data/connect/actions/workflows/test.yml/badge.svg
[actions-url]: https://github.com/redpanda-data/connect/actions/workflows/test.yml

[golangci-lint]: https://golangci-lint.run/
[jaeger]: https://www.jaegertracing.io/


Benthos

A high-performance ELT data integration framework with pluggable architecture.

<h1 align="center"><img alt="cloudquery logo" width=75% src="https://github.com/cloudquery/cloudquery/raw/main/cli/docs/images/logo.png"/></h1>


[![License: MPL 2.0](https://img.shields.io/badge/License-MPL_2.0-brightgreen.svg)](https://opensource.org/licenses/MPL-2.0)  [![Go Report Card](https://goreportcard.com/badge/github.com/cloudquery/cloudquery)](https://goreportcard.com/report/github.com/cloudquery/cloudquery)  [![CLI Workflow](https://github.com/cloudquery/cloudquery/actions/workflows/cli.yml/badge.svg)](https://github.com/cloudquery/cloudquery/actions/workflows/cli.yml)  

CloudQuery is an [open-source](https://github.com/cloudquery/cloudquery),
high-performance data integration framework built for developers, with support for a wide range of plugins.

CloudQuery extracts, transforms, and loads configuration from cloud APIs, files or databases to
variety of supported destinations such as databases, data lakes, or streaming platforms
for further analysis.

### Installation

See the **[Quickstart guide](https://www.cloudquery.io/docs/quickstart)** for instructions how to start syncing data with CloudQuery.

## Why CloudQuery?

- **Blazing fast**: CloudQuery is optimized for performance, utilizing the excellent Go concurrency model with light-weight goroutines.
- **Deploy anywhere**: CloudQuery plugins are single-binary executables and can be deployed and run anywhere.
- **Open source framework**: Language-agnostic, extensible plugin architecture using [Apache Arrow](https://arrow.apache.org/): develop your own plugins in Go, Python, Java or JavaScript using the [CloudQuery SDK](https://docs.cloudquery.io/docs/developers/creating-new-plugin).
- **Pre-built queries**: CloudQuery maintains a number of out-of-the-box security and compliance policies for cloud infrastructure.
- **Unlimited scale**: CloudQuery plugins are stateless and can be scaled horizontally on any platform, such as EC2, Kubernetes, batch jobs or any other compute.

## Use Cases

- **Cloud Security Posture Management**: Use as a CSPM solution to monitor and enforce security policies across your cloud infrastructure for AWS, GCP, Azure and many more.
- **Cloud Asset Inventory**: First-class support for major cloud infrastructure providers such as AWS, GCP and Azure allow you to collect and unify configuration data.
- **Cloud FinOps**: Collect and unify billing data from cloud providers to drive financial accountability.
- **ELT Platform**: With hundreds of plugin combinations and extensible architecture, CloudQuery can be used for reliable, efficient export from any API to any database, or from one database to another.
- **Attack Surface Management**: [solution](https://www.cloudquery.io/how-to-guides/attack-surface-management-with-graph) for continuous discovery, analysis and monitoring of potential attack vectors that make up your organization's attack surface.
- **Eliminate data silos**: Eliminate data silos across your organization, unifying data between security, infrastructure, marketing and finance teams.

### Links

- Homepage: [https://www.cloudquery.io](https://www.cloudquery.io)
- Documentation: [https://www.cloudquery.io/docs](https://www.cloudquery.io/docs)
- Integrations: [https://hub.cloudquery.io](https://hub.cloudquery.io)
- Open Source Releases: [https://github.com/cloudquery/cloudquery/releases](https://github.com/cloudquery/cloudquery/releases)
- Plugin SDK: [https://github.com/cloudquery/plugin-sdk](https://github.com/cloudquery/plugin-sdk)

## License

By contributing to CloudQuery you agree that your contributions will be licensed as defined on the LICENSE file.

## Hiring

If you are into Go, Backend, Cloud, GCP, AWS - ping us at jobs [at] our domain

## Contribution

Feel free to open a pull request for small fixes and changes. For bigger changes and new plugins, please open an issue first to prevent duplicated work and to have the relevant discussions first.

## Open source

The CloudQuery framework, SDK, CLI and some plugins are open source - please [file an issue](https://github.com/cloudquery/cloudquery/issues/new/choose) before opening a PR.

The Best Go Libraries For Data Integration Frameworks - Frameworks for performing ELT / ETL (3)

omniparser

Benthos

CloudQuery