Developer Guide

We love pull requests from everyone. By participating in this project, you agree to abide by the Microsoft Open Source Code of Conduct

Setting up the Agogosml Library for Development

Requirements to Run Locally

Build with Docker and Test Your App

Build the base image using:

docker build -t agogosml/agogosml -f agogosml/Dockerfile.agogosml agogosml

Then the input reader image:

docker build -t agogosml/input_reader -f input_reader/Dockerfile.input_reader .

The app:

docker build -t agogosml/app -f sample_app/Dockerfile.app .

And finally the output writer:

docker build -t agogosml/output_writer -f output_writer/Dockerfile.output_writer .

Run with Docker Locally

Set required environment variables (You can see an example env.example.sh)

export INPUT_READER_NAME=input-reader
export APP_NAME=app
export OUTPUT_WRITER_NAME=output-writer
export NETWORK_NAME=testnetwork       # Docker network name
export MESSAGING_TYPE=eventhub        # eventhub/kafka
export AZURE_STORAGE_ACCOUNT=         # storage account name for EH processor
export AZURE_STORAGE_ACCESS_KEY=      # storage account key for EH processor
export LEASE_CONTAINER_NAME=          # storage account container for EH processor
export EVENT_HUB_NAMESPACE=           # EH namespace
export INPUT_EVENT_HUB_NAME=          # input EH
export INPUT_EVENT_HUB_SAS_POLICY=    # input EH policy name
export INPUT_EVENT_HUB_SAS_KEY=       # input EH policy key
export OUTPUT_EVENT_HUB_NAME=         # output EH
export OUTPUT_EVENT_HUB_SAS_POLICY=   # output EH policy name
export OUTPUT_EVENT_HUB_SAS_KEY=      # output EH policy key
export APP_PORT=5000                  # app port
export OUTPUT_WRITER_PORT=8080        # output writer app port

A Docker network must then be created with:

docker network create $NETWORK_NAME

The four Docker images must then be run, prepending the parameter -e to any environment variables. An example of how to run one of these Docker images is:

# Run Input reader
docker run --rm --network $NETWORK_NAME --name $INPUT_READER_NAME -d \
    -e MESSAGING_TYPE=$MESSAGING_TYPE \
    -e AZURE_STORAGE_ACCOUNT=$AZURE_STORAGE_ACCOUNT \
    -e AZURE_STORAGE_ACCESS_KEY=$AZURE_STORAGE_ACCESS_KEY \
    -e LEASE_CONTAINER_NAME=$LEASE_CONTAINER_NAME \
    -e EVENT_HUB_NAMESPACE=$EVENT_HUB_NAMESPACE \
    -e EVENT_HUB_NAME=$INPUT_EVENT_HUB_NAME \
    -e EVENT_HUB_SAS_POLICY=$INPUT_EVENT_HUB_SAS_POLICY \
    -e EVENT_HUB_SAS_KEY=$INPUT_EVENT_HUB_SAS_KEY \
    -e APP_HOST=$APP_NAME \
    -e APP_PORT=$APP_PORT \
    agogosml/input_reader:latest

# Run app
docker run --rm --name $APP_NAME -d --network $NETWORK_NAME \
    -e HOST=$APP_NAME \
    -e PORT=$APP_PORT \
    -e OUTPUT_URL=http://$OUTPUT_WRITER_NAME:$OUTPUT_WRITER_PORT \
    -e SCHEMA_FILEPATH=schema_example.json \
    agogosml/app

# Run Output writer
docker run --rm --name $OUTPUT_WRITER_NAME -d --network $NETWORK_NAME \
    -e MESSAGING_TYPE=$MESSAGING_TYPE \
    -e EVENT_HUB_NAMESPACE=$EVENT_HUB_NAMESPACE \
    -e EVENT_HUB_NAME=$OUTPUT_EVENT_HUB_NAME \
    -e EVENT_HUB_SAS_POLICY=$OUTPUT_EVENT_HUB_SAS_POLICY \
    -e EVENT_HUB_SAS_KEY=$OUTPUT_EVENT_HUB_SAS_KEY \
    -e OUTPUT_WRITER_HOST=$OUTPUT_WRITER_NAME \
    -e OUTPUT_WRITER_PORT=$OUTPUT_WRITER_PORT \
    agogosml/output_writer:latest

Now you can send a message to Event Hub with the following sample payload and check the output Event Hub for the transformed result:

{
    "key": "SAMPLE_KEY",
    "intValue": 40
}

Setting up the CLI for Development

agogosml_cli is a cli tool developed with Python using the Click_ in combination with cookiecutter.

Requirements to Run Locally

Local Installation

Installing Dependencies (and Dev Dependencies):

$ cd agogosml_cli/
$ pipenv install --dev

Running Tests:

$ pipenv run make test

Running Linter:

$ pipenv run make lint

Deployment and Provisionning to Azure

You can follow the same steps in the User Guide to deploy the build to Azure.

Before Submitting a PR to the Project

Make sure the tests pass

Make your change. Add tests for your change. Make the tests pass

Push to your fork and submit a pull request.

At this point you’re waiting on us. We like to at least comment on pull requests within three business days (and, typically, one business day). We may suggest some changes or improvements or alternatives.

Some things that will increase the chance that your pull request is accepted: