imaginary
Fast and simple HTTP microservice for image resizing
About imaginary
This large README is shown as a plain-text preview. Read the full README at the source
# imaginary [](https://hub.docker.com/r/h2non/imaginary/) [](https://hub.docker.com/r/h2non/imaginary/) [](https://fly.io/launch/github/h2non/imaginary) **[Fast](#benchmarks) HTTP [microservice](http://microservices.io/patterns/microservices.html)** written in Go **for high-level image processing** backed by [bimg](https://github.com/h2non/bimg) and [libvips](https://github.com/jcupitt/libvips). `imaginary` can be used as private or public HTTP service for massive image processing with first-class support for [Docker](#docker) & [Fly.io](#flyio). It's almost dependency-free and only uses [`net/http`](http://golang.org/pkg/net/http/) native package without additional abstractions for better [performance](#performance). Supports multiple [image operations](#supported-image-operations) exposed as a simple [HTTP API](#http-api), with additional optional features such as **API token authorization**, **URL signature protection**, **HTTP traffic throttle** strategy and **CORS support** for web clients. `imaginary` **can read** images **from HTTP POST payloads**, **server local path** or **remote HTTP servers**, supporting **JPEG**, **PNG**, **WEBP**, **HEIF**, and optionally **TIFF**, **PDF**, **GIF** and **SVG** formats if `[email protected]+` is compiled with proper library bindings. `imaginary` is able to output images as JPEG, PNG and WEBP formats, including transparent conversion across them. `imaginary` optionally **supports image placeholder fallback mechanism** in case of image processing error or server error of any nature, hence an image will be always returned by imaginary even in case of error, trying to match the requested image size and format type transparently. The error details will be provided in the response HTTP header `Error` field serialized as JSON. `imaginary` uses internally `libvips`, a powerful and efficient library written in C for fast image processing which requires a [low memory footprint](https://github.com/libvips/libvips/wiki/Benchmarks) and it's typically 4x faster than using the quickest ImageMagick and GraphicsMagick settings or Go native `image` package, and in some cases it's even 8x faster processing JPEG images. To get started, take a look the [installation](#installation) steps, [usage](#command-line-usage) cases and [API](#http-api) docs. ## Contents - [Supported image operations](#supported-image-operations) - [Prerequisites](#prerequisites) - [Installation](#installation) - [Docker](#docker) - [Fly.io](#flyio) - [Cloud Foundry](#cloudfoundry) - [Google Cloud Run](#google-cloud-run) - [Recommended resources](#recommended-resources) - [Production notes](#production-notes) - [Scalability](#scalability) - [Clients](#clients) - [Performance](#performance) - [Benchmark](#benchmark) - [Command-line usage](#command-line-usage) - [HTTP API](#http-api) - [Authorization](#authorization) - [URL signature](#url-signature) - [Errors](#errors) - [Form data](#form-data) - [Params](#params) - [Endpoints](#get-) - [Logging](#logging) - [Fluentd log ingestion](#fluentd-log-ingestion) - [Authors](#authors) - [License](#license) ## Supported image operations - Resize - Enlarge - Crop - SmartCrop (based on [libvips built-in algorithm](https://github.com/jcupitt/libvips/blob/master/libvips/conversion/smartcrop.c)) - Rotate (with auto-rotate based on EXIF orientation) - AutoRotate with further image transformations (based on EXIF metadata orientation) - Flip (with auto-flip based on EXIF metadata) - Flop - Zoom - Thumbnail - Fit - [Pipeline](#get--post-pipeline) of multiple independent image transformations in a single HTTP request. - Configurable image area extraction - Embed/Extend image, supporting multiple modes (white, black, mirror, copy or custom background color) - Watermark (customizable by text) - Watermark image - Custom output color space (RGB, black/white...) - Format conversion (with additional quality/compression settings) - Info (image size, format, orientation, alpha...) - Reply with default or custom placeholder image in case of error. - Blur ## Prerequisites - [libvips](https://github.com/jcupitt/libvips) 8.8+ (8.9+ recommended) - C compatible compiler such as gcc 4.6+ or clang 3.0+ - Go 1.12+ ## Installation ```bash go get -u github.com/h2non/imaginary ``` Also, be sure you have the latest version of `bimg`: ```bash go get -u github.com/h2non/bimg ``` ### libvips Run the following script as `sudo` (supports OSX, Debian/Ubuntu, Redhat, Fedora, Amazon Linux): ```bash curl -s https://raw.githubusercontent.com/h2non/bimg/master/preinstall.sh | sudo bash - ``` The [install script](https://github.com/h2non/bimg/blob/master/preinstall.sh) requires `curl` and `pkg-config` ### Docker See [Dockerfile](https://github.com/h2non/imaginary/blob/master/Dockerfile) for image details. Fetch the image (comes with latest stable Go and `libvips` versions) ``` docker pull h2non/imaginary ``` Start the container with optional flags (default listening on port 9000) ``` docker run -p 9000:9000 h2non/imaginary -cors ``` Start the container enabling remote URL source image processing via GET requests and `url` query param. ``` docker run -p 9000:9000 h2non/imaginary -p 9000 -enable-url-source ``` Start the container enabling local directory image process via GET requests and `file` query param. ``` docker run -p 9000:9000 h2non/imaginary -p 900 -mount /volume/images ``` Start the container in debug mode: ``` docker run -p 9000:9000 -e "DEBUG=*" h2non/imaginary ``` Enter to the interactive shell in a running container ``` sudo docker exec -it <containerIdOrName> bash ``` Stop the container ``` docker stop h2non/imaginary ``` For more usage examples, see the [command line usage](#command-line-usage). All Docker images tags are available [here](https://hub.docker.com/r/h2non/imaginary/tags/). #### Docker Compose You can add `imaginary` to your `docker-compose.yml` file: ```yaml version: "3" services: imaginary: image: h2non/imaginary:latest # optionally mount a volume as local image source volumes: - images:/mnt/data environment: PORT: 9000 command: -enable-url-source -mount /mnt/data ports: - "9000:9000" ``` ### Fly.io Deploy imaginary in seconds close to your users in [Fly.io](https://fly.io) cloud by clicking on the button below: <a href="https://fly.io/docs/app-guides/run-a-global-image-service/"> <img src="testdata/flyio-button.svg?raw=true" width="200"> </a> #### About Fly.io Fly is a platform for applications that need to run globally. It runs your code close to users and scales compute in cities where your app is busiest. Write your code, package it into a Docker image, deploy it to Fly's platform and let that do all the work to keep your app snappy. You can [learn more](https://fly.io/docs/) about how Fly.io can reduce latency and provide a better experience by serving traffic close to your users location. #### Global image service tutorial [Learn more](https://fly.io/docs/app-guides/run-a-global-image-service/) about how to run a custom deployment of imaginary on the Fly.io cloud. ### CloudFoundry Assuming you have cloudfoundry account, [bluemix](https://console.ng.bluemix.net/) or [pivotal](https://console.run.pivotal.io/) and [command line utility installed](https://github.com/cloudfoundry/cli). Clone this repository: ``` git clone https://github.com/h2non/imaginary.git ``` Push the application ``` cf push -b https://github.com/yacloud-io/go-buildpack-imaginary.git imaginary-inst01 --no-start ``` Define the library path ``` cf set-env imaginary-inst01 LD_LIBRARY_PATH /home/vcap/app/vendor/vips/lib ``` Start the application ``` cf start imaginary-inst01 ``` ### Google Cloud Run Click to deploy on Google Cloud Run: [](https://deploy.cloud.run) ### Recommended resources Given the multithreaded native nature of Go, in terms of CPUs, most cores means more concurrency and therefore, a better performance can be achieved. From the other hand, in terms of memory, 512MB of RAM is usually enough for small services with low concurrency (<5 requests/second). Up to 2GB for high-load HTTP service processing potentially large images or exposed to an eventual high concurrency. If you need to expose `imaginary` as public HTTP server, it's highly recommended to protect the service against DDoS-like attacks. `imaginary` has built-in support for HTTP concurrency throttle strategy to deal with this in a more convenient way and mitigate possible issues limiting the number of concurrent requests per second and caching the awaiting requests, if necessary. ### Production notes In production focused environments it's highly recommended to enable the HTTP concurrency throttle strategy in your `imaginary` servers. The recommended concurrency limit per server to guarantee a good performance is up to `20` requests per second. You can enable it simply passing a flag to the binary: ``` $ imaginary -concurrency 20 ``` ### Memory issues In case you are experiencing any persistent unreleased memory issues in your deployment, you can try passing this environment variables to `imaginary`: ``` MALLOC_ARENA_MAX=2 imaginary -p 9000 -enable-url-source ``` ### Graceful shutdown When you use a cluster, it is necessary to control how the deployment is executed, and it is very useful to finish the containers in a controlled manner. You can use the next command: ``` $ ps auxw | grep 'bin/imaginary' | awk 'NR>1{print buf}{buf = $2}' | xargs kill -TERM > /dev/null 2>&1 ``` ### Scalability If you're looking for a large scale solution for massive image processing, you should scale `imaginary` horizontally, distributing the HTTP load across a pool of imaginary servers. Assuming that you want to provide a high availability to deal efficiently with, let's say, 100 concurrent req/sec, a good approach would be using a front end balancer (e.g: HAProxy) to delegate the traffic control flow, ensure the quality of service and distribution the HTTP across a pool of servers: ``` |==============| | Dark World | |==============| |||| |==============| | Balancer | |==============| | | / \ / \ / \ /-----------\ /-----------\ | imaginary | | imaginary | (*n) \-----------/ \-----------/ ``` ## Clients - [node.js](https://github.com/h2non/node-imaginary) Feel free to send a PR if you created a client for other language. ## Performance libvips is probably the faster open source solution for image processing. Here you can see some performance test comparisons for multiple scenarios: - [libvips speed and memory usage](https://github.com/libvips/libvips/wiki/Benchmarks) - [bimg](https://github.com/h2non/bimg#Performance) (Go library with C bindings to libvips) ## Benchmark See [benchmark.sh](https://github.com/h2non/imaginary/blob/master/benchmark.sh) for more details Environment: Go 1.4.2. libvips-7.42.3. OSX i7 2.7Ghz ``` Requests [total] 200 Duration [total, attack, wait] 10.030639787s, 9.949499515s, 81.140272ms Latencies [mean, 50, 95, 99, max] 83.124471ms, 82.899435ms, 88.948008ms, 95.547765ms, 104.384977ms Bytes In [total, mean] 23443800, 117219.00 Bytes Out [total, mean] 175517000, 877585.00 Success [ratio] 100.00% Status Codes [code:count] 200:200 ``` ### Conclusions `imaginary` can deal efficiently with up to 20 request per second running in a multicore machine, where it crops a JPEG image of 5MB and spending per each request less than 100 ms The most expensive image operation under high concurrency scenarios (> 20 req/sec) is the image enlargement, which requires a considerable amount of math operations to scale the original image. In this kind of operation the required processing time usually grows over the time if you're stressing the server continuously. The advice here is as simple as taking care about the number of concurrent enlarge operations to avoid server performance bottlenecks. ## Command-line usage ``` Usage: imaginary -p 80 imaginary -cors imaginary -concurrency 10 imaginary -path-prefix /api/v1 imaginary -enable-url-source imaginary -disable-endpoints form,health,crop,rotate imaginary -enable-url-source -allowed-origins http://localhost,http://server.com,http://*.example.org imaginary -enable-url-source -enable-auth-forwarding imaginary -enable-url-source -authorization "Basic AwDJdL2DbwrD==" imaginary -enable-placeholder imaginary -enable-url-source -placeholder ./placeholder.jpg imaginary -enable-url-signature -url-signature-key 4f46feebafc4b5e988f131c4ff8b5997 imaginary -enable-url-source -forward-headers X-Custom,X-Token imaginary -h | -help imaginary -v | -version Options: -a <addr> Bind address [default: *] -p <port> Bind port [default: 8088] -h, -help Show help -v, -version Show version -path-prefix <value> Url path prefix to listen to [default: "/"] -cors Enable CORS support [default: false] -gzip Enable gzip compression (deprecated) [default: false] -disable-endpoints Comma separated endpoints to disable. E.g: form,crop,rotate,health [default: ""] -key <key> Define API key for authorization -mount <path> Mount server local directory -http-cache-ttl <num> The TTL in seconds. Adds caching headers to locally served files. -http-read-timeout <num> HTTP read timeout in seconds [default: 60] -http-write-timeout <num> HTTP write timeout in seconds [default: 60] -enable-url-source Enable remote HTTP URL image source processing (?url=http://..) -enable-placeholder Enable image response placeholder to be used in case of error [default: false] -enable-auth-forwarding Forwards X-Forward-Authorization or Authorization header to the image source server. -enable-url-source flag must be defined. Tip: secure your server from public access to prevent attack vectors -forward-headers Forwards custom headers to the image source server. -enable-url-source flag must be defined. -enable-url-signature Enable URL signature (URL-safe Base64-encoded HMAC digest) [default: false] -url-signature-key The URL signature key (32 characters minimum) -allowed-origins <urls> Restrict remote image source processing to certain origins (separated by commas). Note: Origins are validated against host *AND* path. -max-allowed-size <bytes> Restrict maximum size of http image source (in bytes) -max-allowed-resolution <megapixels> Restrict maximum resolution of the image [default: 18.0] -certfile <path> TLS certificate file path -keyfile <path> TLS private key file path -authorization <value> Defines a constant Authorization header value passed to all the image source servers. -enable-url-source flag must be defined. This overwrites authorization headers forwarding behavior via X-Forward-Authorization -placeholder <path> Image path to image custom placeholder to be used in case of error. Recommended minimum image size is: 1200x1200 -concurrency <num> Throttle concurrency limit per second [default: disabled] -burst <num> Throttle burst max cache size [default: 100] -mrelease <num> OS memory release interval in seconds [default: 30] -cpus <num> Number of used cpu cores. (default for current machine is 8 cores) -log-level Set log level for http-server. E.g: info,warning,error [default: info]. Or can use the environment variable GOLANG_LOG=info. ``` Start the server in a custom port: ```bash imaginary -p 8080 ``` Also, you can pass the port as environment variable: ```bash PORT=8080 imaginary ``` Enable HTTP server throttle strategy (max 10 requests/second): ``` imaginary -p 8080 -concurrency 10 ``` Enable remote URL image fetching (then you can do GET request passing the `url=http://server.com/image.jpg` query param): ``` imaginary -p 8080 -enable-url-source ``` Mount local directory (then you can do GET request passing the `file=image.jpg` query param): ``` imaginary -p 8080 -mount ~/images ``` Enable authorization header forwarding to image origin server. `X-Forward-Authorization` or `Authorization` (by priority) header value will be forwarded as `Authorization` header to the target origin server, if one of those headers are present in the incoming HTTP request. Security tip: secure your server from public access to prevent attack vectors when enabling this option: ``` imaginary -p 8080 -enable-url-source -enable-auth-forwarding ``` Or alternatively you can manually define an constant Authorization header value that will be always sent when fetching images from remote image origins. If defined, `X-Forward-Authorization` or `Authorization` headers won't be forwarded, and therefore ignored, if present. **Note**: ``` imaginary -p 8080 -enable-url-source -authorization "Bearer s3cr3t" ``` Send fixed caching headers in the response. The headers can be set in either "cache nothing" or "cache for N seconds". By specifying `0` imaginary will send the "don't cache" headers, otherwise it sends headers with a TTL. The following example informs the client to cache the result for 1 year: ``` imaginary -p 8080 -enable-url-source -http-cache-ttl 31556926 ``` Enable placeholder image HTTP responses in case of server error/bad request. The placeholder image will be dynamically and transparently resized matching the expected image `width`x`height` define in the HTTP request params. Also, the placeholder image will be also transparently converted to the desired image type defined in the HTTP request params, so the API contract should be maintained as much better as possible. This feature is particularly useful when using `imaginary` as public HTTP service consumed by Web clients. In case of error, the appropriate HTTP status code will be used to reflect the error, and the error details will be exposed serialized as JSON in the `Error` response HTTP header, for further inspection and convenience for API clients. ``` imaginary -p 8080 -enable-placeholder -enable-url-source ``` You can optionally use a custom placeholder image. Since the placeholder image should fit a variety of different sizes, it's recommended to use a large image, such as `1200`x`1200`. Supported custom placeholder image types are: `JPEG`, `PNG` and `WEBP`. ``` imaginary -p 8080 -placeholder=placeholder.jpg -enable-url-source ``` Enable URL signature (URL-safe Base64-encoded HMAC digest). This feature is particularly useful to protect against multiple image operations attacks and to verify the requester identity. ``` imaginary -p 8080 -enable-url-signature -url-signature-key 4f46feebafc4b5e988f131c4ff8b5997 ``` It is recommended to pass key as environment variables: ``` URL_SIGNATURE_KEY=4f46feebafc4b5e988f131c4ff8b5997 imaginary -p 8080 -enable-url-signature ``` Increase libvips threads concurrency (experimental): ``` VIPS_CONCURRENCY=10 imaginary -p 8080 -concurrency 10 ``` Enable debug mode: ``` DEBUG=* imaginary -p 8080 ``` Or filter debug output by package: ``` DEBUG=imaginary imaginary -p 8080 ``` Disable info logs: ``` GOLANG_LOG=error imaginary -p 8080 ``` #### Examples Reading a local image (you must pass the `-mount=<directory>` flag): ``` curl -O "http://localhost:8088/crop?width=500&height=400&file=foo/bar/image.jpg" ```
Frequently Asked Questions
What is imaginary?
imaginary is a Images library for the Go programming language. Fast and simple HTTP microservice for image resizing
How many GitHub stars does imaginary have?
imaginary has 6,084 GitHub stars in the directory's latest synchronization.
How do I install imaginary?
Install imaginary with the Go module system using `go get h2non/imaginary`. Check the repository for the current installation instructions.
What category does imaginary belong to?
imaginary is listed under Images, specifically Images.