Flite Careers

Apache Buildr and Annotation Processing for AWS Simple Workflow

Amazon’s Simple Workflow Service allows application developers to take advantage of a scalable coordination and orchestration framework to develop asynchronous and distributed applications by providing a reliable central hub through which data can be exchanged.

Here at Flite we’ve been developing new applications (as well as moving some existing services) to use the Simple Workflow Service for various asynchronous tasks and processes that require durable state across distributed execution. An example would be an Ad or Component approval system that upon Publish moves the Ad or Component through a workflow (either creative or functional) incorporating out-of-band follow-up and feedback coordinating the logic of success or failure.

First a bit about the Amazon Simple Workflow (SWF) Service

At a high-level SWF consists of a few key concepts: the workflow itself, various activities to be invoked by the workflow to accomplish some objective, and deciders which determine how activities flow through the system.

Figure 1: Amazon Simple Workflow Basics

Activity Workers

Activity workers are typically long-running processes that continually poll the SWF for activity tasks to perform. Activity tasks can be scoped to specific task lists (and thus to specific workers that are scoped to the same task list) and represent a unit of work to perform in the workflow. Workers perform an activity task to completion and report the result back to the Simple Workflow Service before resuming to poll again for a new task.

Deciders

The decider is an implementation of the workflow’s coordination logic. During workflow execution deciders respond to changes and schedule new activities, if necessary, using the current execution state of the workflow.

Tasks

There are two types of tasks in the SWF Service:

  1. Activity task - a unit of work to be performed by an Activity Worker. The task contains all of information necessary for the worker to perform its function.

  2. Decision task - captures the current state of the workflow execution so that deciders can determine the next activity that should be performed.

Domains

Workflows are themselves scoped to a given domain within an AWS account. Workflows in different domains cannot interact with each other but Amazon allows for more than one workflow in a given domain.


Generating sources via Java annotation processing

Amazon provides the AWS Flow Framework for Java that works with the SWF service to aid in developing Java applications on the Simple Workflow Service. This is part of the Amazon AWS SDK for Java which includes example code for interacting with most if not all of the services provided by Amazon.

For Java development we primarily use Maven as our build tool but over the last year we’ve been slowly migrating to Apache Buildr such that new projects are typically written using Buildr as their build tool. One of the prerequisites for developing with the AWS Flow Framework is that client code to access activities and even the workflow must generated via Java annotation processing to create client classes which are used in your calling code. In prototyping a new service that works using the AWS Flow Framework one of the first challenges was to figure out how to do Java annotation processing in Buildr in a manner that fit our development process.

One gotcha that a lot of people run into is the need to have the flow build tools jar:

1
aws-java-sdk-flow-build-tools-#{VERSION}.jar

It’s tied to the same version of the AWS Java SDK and is generally available in most public Maven repositories.

Luckily, Buildr has support for the Java annotation processor via the Java compiler functionality. Unfortunately the built-in apt functionality does not correctly identify the Amazon annotation processors included in the aws-java-sdk-flow-build-tools jar that are needed to process annotations added to your source. Additionally, we’d like more flexibility as to not only the output but also the input directory. Here is where the flexibility of having a build tool backed by Ruby comes in handy. We bypass the current APT functionality in the Buildr Java compiler class and instead directly call ant as Buildr eventually does. It’s pretty easy to write the functionality we need and include it into a project as a task. Additionally this task could then be packaged into a Buildr add-on and used across multiple projects.

The task does a couple of things all of which go into outputting a single Java apt command that is called via Buildr’s ant integration against the appropriate source files with the generated Java source files placed in the specified location.

The task by default is set to verbose to show what the apt compiler is doing. Unfortunately the Java annotation processor currently works only if the source is specified to be “1.5”. This should not a problem as this only means that the generated client code is Java 1.5 compatible and we’ve seen no issues then compiling this source inside of our projects with the Buildr command: compile.using :source=>'1.6', :target=>'1.6'.

Enjoy and happy workflow.

AWS