Akka, DDD, CQRS, Event Sourcing and Me

| Comments

Introduction

I’ve been involved in some really interesting discussions on the Akka User mailing list of late and thought it would be good to translate something I wrote, in one particular thread, into a blog post.

I recently became rather obsessed with Domain Driven Design (DDD), CQRS and Event Sourcing. Given an existing passion for Akka (actor based programming), it dawned on me that the actor model might be an extremely good fit for the aforementioned “Holy Trio” of paradigms/patterns. Whilst trying to build a prototype, I brought up the topic on the Akka User mailing list and became aware that the Akka Team were actively working on Akka Persistence, which is well ahead of my dirty little prototype! For one reason or another, within a post on the mailing list, I ended up brain dumping my understanding of how Akka Persistence and the “Holy Trio” fits together, so here follows a blog friendly version!

Akka and the Holy Trio

So, it seems like the actor model fits very well with DDD, and I’m not the first person to think this – Vaughn Vernon is well ahead of the game!

It feels quite natural to me to consider an actor analogous to an aggregate root (in DDD speak). Rather than just seeing an actor as a mediator sitting in front of a database, you see the actor conceptually as an entity. By incorporating event sourcing, the actor works on the basis of processing a command (representing some action) and outputting one or many events that represent the result of processing that command. At all times the actor encapsulates its current state, which can simply be seen as the result of a series of ordered events applied to it.

Initially, it’s actually quite helpful to ignore persistence requirements altogether and just rely on the state within the actor – in this way you don’t allow persistence concerns to influence the way you design your domain – I often find that domain models end up messy because, despite attempts to avoid doing so, persistence concerns end up, in one form or another, leaking into the business logic. Any approach to building a domain layer, that can genuinely remove the need to consider persistence technologies, is an attractive proposition in my mind!

By following an event sourcing approach, persistence becomes no more complex than just persisting the events – that the actor produces – in an event store (journal). Given that events represent an immutable history of state changes, the journal can be append only, and that’s pretty exciting from a performance perspective. This is something that can quite clearly be bolted on once you’ve already put together a working in-memory only version of your domain – the actors themselves can be completely agnostic to the existence of the persistence mechanism. This is where Akka Persistence comes in – it works on the following basis:

  1. Actor receives command
  2. Actor processes command (applies business logic) and produces events that represent results of processing
  3. Events are persisted to journal
  4. Events are applied to actor so it can update it’s internal state

Akka Persistence pretty much deals with everything other than the command handling step, which represents your custom business logic – Akka is not quite good enough (yet) to do that bit ;–)

Step 4 is quite important – it’s key that internal state is only applied once you’re sure events have been persisted to the journal. Also, by separating this step, it serves the additional purpose of allowing events in the journal to be replayed onto a “green” instance of the actor such to bring its internal state back up to current. Akka Persistence also has support for “snapshots” which allow you to take periodic snapshots of current state to be used as a performance optimisation when replaying events.

So, to sum up, the actor model just fits with the “Holy Trio”. It seems to deal with a majority of the pain points experienced when building a domain layer using more traditional CRUD techniques. It’s pretty exciting to me how natural this design feels without any need to consider using an ORM ;–)

The Future

And that’s the end of the blog friendly version of my brain dump! I truly believe the combination of the actor model with DDD, CQRS and Event Sourcing is going to become a very prevalent domain model design solution in the future.

I have to add that the Akka Team (including Patrik, Martin, Roland etc.) continue to be such an inspiration. Keep up the good work, guys!

Access Logs in Play! Using RequestHeader Tags

| Comments

Log files as event streams

Having long been a fan of treating log files as streams of events (JSON formatted), I thought it would be useful to share a little bit of code to demonstrate how to achieve access style logging in a Play application. There’s certainly more than one solution to this problem (there is, for example, a nice little plugin here: https://github.com/briannesbitt/play-accesslog), but I wanted to share a little snippet of code – I’ve used plenty of times in my own applications – utilising the RequestHeader tags feature in Play!.

Just after starting this post, I noticed another great article from @matthiasnehlsen about using Kibana to store log events generated in Play applications. Kibana is a great tool – backed by the awesome Elasticsearch – and I also highly recommend it. I won’t go into any detail in this article about sending logs to Kibana as Matthias has done a great job of doing that :–) Suffice to say, though, the information in my article here would certainly play nicely with Kibana – I simply want to make you aware of how you can extract some useful information from Play to enhance your log events no matter where you might wish to send them (log files included!).

RequestHeader tags

So…back to this feature – RequestHeader tags! It turns out that Play associates some very useful ‘tags’ along with the RequestHeader generated for each request. You can get hold of this Map of tags by calling RequestHeader.tags.

Given this learning, we can build a Play EssentialFilter – that will wrap every request – and log the JSON formatted request information to the Play Logger.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
object AccessLogFilter extends EssentialFilter {

  val dateTimeFormat = ISODateTimeFormat.ordinalDateTimeNoMillis()

  def apply(next: EssentialAction) = new EssentialAction {
    def apply(rh: RequestHeader) = {
      val startTime = System.currentTimeMillis()

      def logTime(result: PlainResult): Result = {
        val event = Json.obj(
          "uri" -> rh.uri,
          "timestamp" -> dateTimeFormat.print(new DateTime),
          "execution_time" -> (System.currentTimeMillis() - startTime),
          "status" -> result.header.status,
          "tags" -> Json.toJson(rh.tags.map(entry => entry._1.toLowerCase -> entry._2))
        )
        Logger.info(Json.stringify(event))
        result
      }

      next(rh).map {
        case plain: PlainResult => logTime(plain)
        case async: AsyncResult => async.transform(logTime)
      }
    }
  }
}

On top of adding the tags (and some other available request info) to the JSON object, you’ll see the execution time is also captured alongside. To enable this filter, we simply create a custom Global.scala, in the default package:

1
2
object Global extends WithFilters(AccessLogFilter) {
}

Here is an example log line generated by this filter (prettified for readability):

1
2
3
4
5
6
7
8
9
10
11
12
13
{
   "uri":"/search/*",
   "timestamp":"2013-194T18:14:27+01:00",
   "execution_time":20,
   "status":200,
   "tags":{
      "route_verb":"GET",
      "route_action_method":"search",
      "route_pattern":"/search/$searchString",
      "route_controller":"controllers.Application",
      "route_comments":"Root action"
   }
}

One of the most useful things about capturing the tags is that you get easy access to ‘normalized’ actions/endpoints. So, it’s immediately easy (e.g. in a tool like Kibana) to pick all log events related to a specific action. The URI alone is not good enough because parameterized path variables, or query string parameters, lead to non-unique URI representations for the same logical action bindings.

As the filter wraps every request, you will see log lines generated for static assets too – something that is most likely too verbose for most use cases. It’s not too hard to apply filtering to the URI extension to ignore assets, but I left it out of the example code for the sake of cleanliness.

Keep on logging!

Reactive, Real-time Log Search With Play, Akka, AngularJS and Elasticsearch

| Comments

Introduction

So, I’ve decided to contribute an Activator Template to TypeSafe (will submit soon, promise!). Having recently become more and more involved in Elasticsearch, I saw a great opportunity to put together a neat “reactive” application combining Play & Akka with the “bonsai cool” percolation feature of Elasticsearch. Then, to put a cherry on top, use AngularJS on the client-side to create a dynamically updating UI.

What I came up with is slightly contrived – a very basic real-time log entry search tool – but I think it provides a really nice base for apps that want to integrate this bunch of technologies.

All the code for the application is available on Github here. In this post, I’ve attempted to dissect the Activator Template tutorial I’ve written and regurgitate it in blog form.

Play and Akka

Play and Akka are used to implement the reactive server-side application. The application favours SSE (Server Sent Events) to push updates to the client. The template introduces a number of interesting topics, including Play Iteratees/Enumerators and Akka Actors.

AngularJS

AngularJS has been chosen on the client-side to demonstrate how simple it can be to build a dynamic, single page user experience with very little code.

Elasticsearch

The “bonsai cool” percolation feature of Elasticsearch achieves the real-time search aspect of the application. The application starts up an embedded Elasticsearch node, so no need to run your own external instance. Take a look at EmbeddedESServer for the embedded server code. There is a custom Global where the embedded server is started and shutdown as part of the application lifecycle.

The Actors

The application has three actors:

MainSearchActor

This actor’s job is to coordinate the reactive parts of the application and supervise the other actors. It is the main dependency of the application’s single Play controller.

Starting/stopping a search

The actor responds to a StartSearch message by ‘replying’ with an Enumerator to the sender. The Enumerator wraps a unicast channel to which log entries are pushed that match the query string sent within the message. Let’s take a look at some code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
private def startSearching(startSearch: StartSearch) =
    Concurrent.unicast[JsValue](
        onStart = (c) => {
            channels += (startSearch.id -> c)
            elasticSearchActor ! startSearch
        },
        onComplete = {
            self ! StopSearch(startSearch.id)
        },
        onError = (str, in) => {
            self ! StopSearch(startSearch.id)
        }
    ).onDoneEnumerating(
        callback = {
            self ! StopSearch(startSearch.id)
        }
    )

The Play Iteratees library has the very handy Concurrent utilities. In this case, Concurrent.unicast is called to create an Enumerator that encloses a Concurrent.Channel. When the channel starts (onStart), it is stored in a map local to the actor (using UUID as key) and the StartSearch message is forwarded onto the ElasticSearchActor where the query will be percolated in Elasticsearch. It’s worth noting that this code is not production ready – it ought to be a transactional operation, i.e. we should only store the channel once we know Elasticsearch has successfully percolated the query. You will notice that a StopSearch message is sent to self such that the channel is removed from the local map, and the percolated query is deleted, when the channel is no longer useful (i.e. is closed by the client, or an error occurs).

Broadcasting matching results

The actor will receive a SearchMatch message when a log entry has matched a percolated query.

1
2
3
4
5
6
7
private def broadcastMatch(searchMatch: SearchMatch) {
    searchMatch.matchingChannelIds.foreach {
        channels.get(_).map {
            _ push searchMatch.logEntry.data
        }
    }
}

On receipt of the message, each matching id is iterated over and the corresponding channel is retrieved from the local map. The log entry is then pushed to the channel, and thus onto the client.

Scheduling log entry creation

The actor uses the Akka scheduler to send a Tick to the LogEntryProducerActor every second – in the real world, this would obviously be unnecessary, as genuine log entries would be fed into the application in some other way. The Tick is sent to self before being forwarded on to the LogEntryProducerActor.

ElasticsearchActor

This actor has responsibility for both registering queries in Elasticsearch and percolating log entry documents against those queries. Rather than utilise the Elasticsearch Java Client, the code, instead, crafts the Elasticsearch API calls manually, demonstrating the use of the asynchronous Play WS API to execute them. For simplicity, the calls are hard coded to talk to Elasticsearch on localhost:9200 (where the embedded server will be listening).

The code is fairly self explanatory within this actor. Do note that there is a lack of error handling on the API calls thus making this actor unsuitable for production use in its current form. It is recommended you read the Elasticsearch documentation on percolation to learn more about this powerful feature.

There’s one little important gotcha this code has avoided – closing over the sender ref in an asynchronous callback block. The sender ref is part of the shared mutable state of the actor and so, if the actor were to reply to the sender in the percolate callback, a race condition would be encountered if another thread had modified the actor’s state before the percolation call to Elasticsearch had completed. This race condition has been avoided by ensuring to ‘freeze’ the sender ref, by sending it to a private function:

1
private def percolate(logJson: JsValue, requestor: ActorRef)

and close over the parameter instead.

LogEntryProducerActor

I won’t go into the detail of this actor. Suffice to say, its job is to generate a random, JSON formatted log event whenever it receives a Tick message. In reality, a genuine source of log events would replace this actor.

The Play Controller

As most of the server-side logic exists within the actors, the single Play controller is very simple. The most interesting aspect of the controller is the action that opens an event stream connected with the client:

1
2
3
4
5
6
7
def search(searchString: String) = Action {
    Async {
        (searchActor ? StartSearch(searchString = searchString)).map {
            case SearchFeed(out) => Ok.stream(out &> EventSource()).as("text/event-stream")
        }
    }
}

The most important thing to note is the use of the Akka ‘ask’ pattern of message exchange (notice the use of ‘?’ instead of ‘!’). This differs from the more typical fire-and-forget approach in that we’re able to asynchronously pick up a reply from the recipient actor. In this scenario, a StartSearch message is sent to the MainSearchActor which replies with an Enumerator used to stream search results to the client. Given the use of the ‘ask’ pattern, we wrap the action logic in an Async block – so not to hold up other requests – rather than blocking until the Future yields a result.

The User Interface

The key parts of the application UI are:

  1. A Play template with AngularJS specific markup
  2. A single AngularJS controller

The application makes use of the WebJars project to simplify the introduction of its JS and CSS dependencies (e.g. AngularJS and Twitter Bootstrap).

UI Template

Firstly, the opening <div> is linked to the controller SearchCtrl that subsequently enables the automagical databinding power of AngularJS. A simple search form captures an Apache Lucene formatted query string. A search can be started by clicking on the ‘Search’ button which invokes the startSearching() function defined in the controller. Finally, you can see the use of AngularJS two-way databinding to render matching search results contained within the view model (only displays the latest 10 matches):

1
<tr ng-repeat="searchResult in searchResults | limitTo:10">

The AngularJS Controller

The AngularJS controller is fairly straightforward. The key part is handling a new search:

1
2
3
4
5
6
$scope.startSearching = function () {
    $scope.stopSearching()
    $scope.searchResults = [];
    $scope.searchFeed = new EventSource("/search/" + $scope.searchString);
    $scope.searchFeed.addEventListener("message", $scope.addSearchResult, false);
};

Firstly, an existing search is stopped (if running) and the results model cleared (will automagically clear any existing results from the HTML markup). Secondly, an event stream connection is made to the server and an event listener is added that pushes matching search results into the model as they arrive from the server.

As we are updating the model (in the addSearchResult function) outside of it’s knowledge, we’ve had to explicitly tell Angular that we’ve pushed a new result to the model by wrapping the code in a function passed to $scope.apply. See the Angular docs for the $scope object for more information.

Using the application

Fire up the application using play run and point your browser (not IE – it doesn’t support SSE. Doh!) to http://localhost:9000. Using the application is as simple as entering an Apache Lucene formatted query. Please read the Lucene query parser syntax documentation for more information on how to construct Lucene query strings. A simple example would be to enter “GET” as your query string, thus matching all log entries for GET requests.

Enjoy!

Taking a Walk on the Wild Side

| Comments

Having spent nearly 10 years doing my best to avoid wrestling with the client-side web, the necessity to better prepare myself for a new future (think tech startup) means I’ve begun some serious dabbling on “the wild side”.

At first it was just a case of some casual flirting, but that has now extended as far as a proper first date.

I’ve long been coming round to the idea that, in order to build a rich application experience in the browser, Java does not have the answer – for me, the Java component based framework experiments have failed (ThoughtWorks agreed with me in October 2012). I feel sad to say this because, whilst I despised JSF right from the start, I kinda liked Wicket and Tapestry. Without wanting to get too heretical, the Java Swing programming model is grounded in some pretty solid coding patterns and, as close as Wicket comes to realising a similar approach on the web, you just can’t get away from the fact that the web just ain’t really built for that stuff.

So, that date I was referring to was with a cute little beast called AngularJS. After initially being slightly sick in my mouth at the thought of a journey beyond statically typed languages, I’ve begun to come round to the idea that JavaScript needn’t be so bad as long as you play by the rules, and you couple your exploration with a neat framework like AngularJS. With AngularJS, I’m once again playing with a component-ish framework that is not trying to exist on top of an unsuitable protocol – the rich interactions live in the browser where they feel most at home.

One of the coolest things about AngularJS is that I can even apply some of my most loyal of server-side patterns (I’m thinking dependency injection) when hacking client code. And, even more importantly, I still get my server-side fix through building predictable, testable, stateless APIs for the stateful client to consume.

I definitely see this going to a second date, and beyond.

On a final note, I do feel that Twitter Bootstrap deserves a special mention here as well. There’s no doubt that I suck at CSS, and I truly mean that. But, with Bootstrap, no longer must server-side developers walk naked and exposed in the presentational side of the web. I don’t care for CSS – I don’t think I ever will – but with Bootstrap I’ve now got half a chance of making web apps that don’t only work good, but look good at the same time.

Play2 & Scala – a New Beginning

| Comments

Today marks the beginning of a new chapter in my software engineering life. I’ve been an enterprise Java dev for so many years that it’s boring to even talk about it. Don’ t get me wrong, I love my job (if you can even call it a job) but sometimes one needs a spark, a chance to re-experience a honeymoon period that extends beyond love and into a kind of uncontrollable, rapturous madness.

The murmurings for this new chapter began a few weeks ago when I enrolled on Martin Odersky’s “Functional Programming Principles in Scala” course – a fantastic FREE online course that runs for seven weeks, supported by EPFL in Switzerland. For sure, I’ve been well aware of the rise of Scala within the industry but had never gotten round to really indulging in this exciting language (except, maybe, for getting through a few chapters of Programming in Scala on my Kindle). The same goes for the Play! Framework, something I’ve kept an eye on at arm’s length for quite a few years now.

I’m now into the fourth week of Odersky’s course and, following completion of the first three challenging, but almost perversely absorbing, assignments, today I decided it was time to fire up my first Play2 application. Fortunately, I picked up a little freelance client project last week and it was a bit of a no brainer, given my newly mobilised appetite for all things Scala, to implement it using a technology (Play!) I really knew was overdue an evaluation.

Just a few hours after downloading Play! 2.0.4, I have fallen hopelessly under it’s spell – I’ve been wandering around the house in what resembles a drunken stupor, close to screaming with excitement at what I’ve learned and been able to achieve in just one afternoon. My face aches, my jaw having been locked into a permanent smile. I know this is the future, though I feel a certain degree of sadness as I acknowledge my love affair with JEE, and it’s wonderful layer upon layer of abstraction, to be very nearly over (we need to stay friends, mind – my day job will see to that).

The anticipation. The butterflies. So the journey begins.