# HyperFlow: a scientific workflow management system


![HyperFLow](https://img.shields.io/docker/v/hyperflowwms/hyperflow)
## Description

HyperFlow is a Workflow Management System (WMS) dedicated for scientific workflows. 

Browse the [wiki pages](https://github.com/balis/hyperflow/wiki) to learn more about the HyperFlow workflow model. 

## Getting started

### Installation
* Install Node.js (http://nodejs.org)
* Install Redis (http://redis.io) 
* Install HyperFlow: <br>`npm install https://github.com/hyperflow-wms/hyperflow/archive/{version}.tar.gz` (where `{version}` is for example `v1.3.38`) 
* For latest features, install from the master branch: <br>`npm install https://github.com/hyperflow-wms/hyperflow/archive/master.tar.gz`
* Add `<install_root>/node_modules/.bin` to your path

### Running locally
* Start the redis server: `redis-server`
* Run example workflows using command `hflow run <wf_directory>`, for example:<br>```hflow run ./examples/Sqrsum```

### Running locally using Docker images
* Use the latest Docker image for the HyperFlow engine, published in Docker Hub as `hyperflowwms/hyperflow` 
* You can build the image yourself: `make container`
* Start redis container:<br> 
```docker run -d --name redis redis --bind 127.0.0.1```
* Run workflow via HyperFlow container, for example:
```
docker run -a stdout -a stderr --rm --network container:redis \
       -e HF_VAR_WORKER_CONTAINER="hyperflowwms/soykb-workflow-worker" \ 
       -e HF_VAR_WORK_DIR="$PWD/input" \ 
       -e HF_VAR_HFLOW_IN_CONTAINER="true" \
       -e HF_VAR_function="redisCommand" \
       -e REDIS_URL="redis://127.0.0.1:6379" \
       --name hyperflow \
       -v /var/run/docker.sock:/var/run/docker.sock \
       -v $PWD:/wfdir \
       --entrypoint "/bin/sh" hyperflowwms/hyperflow -c "apk add docker && hflow run /wfdir"
```
Where
* `hyperflowwms/soykb-workflow-worker` is the name of the workflow worker container ([Soykb](https://github.com/hyperflow-wms/soykb-workflow) in this case)
* current directory contains `workflow.json`
* subdirectory `inputs` contains workflow input data 

Outputs:
* Directory `inputs` will contain files generated by the workflow run
* Directory `inputs/logs-hf` will contain logs of all workflow jobs

### Running in a Kubernetes cluster
See [HyperFlow Kubernetes deployment](https://github.com/hyperflow-wms/hyperflow-k8s-deployment) project for more information. 

### Running in a distributed infrastructure using the RabbitMQ executor
* Start the RabbitMQ container: `docker run -d --name rabbitmq rabbitmq:3`
* Add option `-e AMQP_URL=amqp://rabbitmq`
* More information in the [hyperflow-amqp-executor](https://github.com/hyperflow-wms/hyperflow-amqp-executor) project (warning: currently not maintained and not tested with latest HyperFlow versions)

### Local configuration files
You can provide workflow configuration through local configuration files:
* `workflow.config.json` -- main configuration file 
* `workflow.config.{name}.json` -- any number of secondary configuration files

The content from all configuration files will be merged and passed to workflow functions via `context.appConfig`. For example for files:
```
workflow.config.json:
{
  "main": "mainValue"
}

workflow.config.foo.json:
{
   "secondary": "secondaryValue"
}
```
The following will be passed in `context.appConfig`:
```
{
  "main": "mainValue",
  "foo": {
     "secondary": "secondaryValue"
  }
}
```

### HyperFlow server
The HyperFlow engine can be started in a server mode using command: ```hflow start-server```

If succesfully started, the server prints its URL:
```
HyperFlow server started at http://localhost:38775
```

Workflows can be run through the HyperFlow server as follows:

```
hflow run --submit=<hyperflow_server_url> <workflow_dir>
```

Currently `<workflow_dir>` must be a local directory accessbile by the server. This allows running multiple workflows (concurrently) using the same instance of the HyperFlow engine.
