Experiments with Event Sourcing

As I was continuing with my little “to-do application”, I thought I add an other interesting technology to the set. While checking the persistence options for the todo-application, I read about the event sourcing pattern. The basic idea is to have your persistent store like a log file, append only.

This means that data in the database is un-modifiable. It can neither be changed, nor be deleted. This fits nicely with the idea of immutable data structures of Clojure and has other advantages, such as far easier synchronization and distribution of data, as it is append-only.

The idea is rather simple. All “data changes” get expressed as events. In my case the events where three:

  1. Todo item added
  2. Todo item deleted
  3. Todo item changed

The database consists then in single table, for all “to-do events”. As every event type has a different payload, the best way to persist this in a classical SQL database is to serialize / deserialize the data into a string and store that.  I used Clojure EDN for this, which makes the code really easy.  Given that the simple table structure would be the same, even for a very complex domain model, using an SQL database is overkill. Some NoSQL data store is for sure a better option.

The table would hase four columns:

  • id
  • uuid
  • event type
  • event payload (as String)

 

To rebuild the state of the todos, the code needs to apply all events one after the other.

This can look like this:


(defn apply-event-to-world [event-string world]
    (let [event (read-string (:EDN event-string))]
    (condp = (:TYPE event-string)
      "todo-created" (conj world event)
      "todo-deleted" (remove #(= (:uuid %) event) world)
      "todo-changed" (replace-todo event world))))

(defn aggregate-todos [todo-events world]</span>
    (if (empty? todo-events)
      world
      (recur (rest todo-events) (apply-event-to-world (first todo-events) world))))

If we have more events, using a command pattern makes the code much simpler. So there would be one command per event.

The source code of the application so far is available here: https://github.com/behrica/todo-clojure/commit/a35f0da685a5c421dbc2272e4f40c7a2b63fa14f

Leave a comment

Clojure macros – I finally got it

On my path of learning Clojure, I struggled a bit with the macros.

But actually it is not that hard. I found a very good example here: http://java.dzone.com/articles/clojure-macros-beginners

It helps a lot to think that a Clojure macro:

just returns Clojure code, while it does some magic with all those special characters, ~ ' ` @ ... 
The expression returned is like a template, and the special characters trigger various types of modification 
of the template related to the evaluation of the expressions in the template.

Following the step-by-step development of the blog post I mentioned before shows very clearly what happens.

Leave a comment

Full automatic installation of Oracle 11g database with Chef

Setting up test databases is always a pain for any developer.

A lot of times we need to make compromises between easy setup and “production like”. As easier it is to setup, as more far away it is from the production environment. I very recently found a complete Chef recipe to install the Oracle 11g Enterprise Edition database full-automatic via Chef. Nothing to configure by hand, just running 

knife bootstrap ..

on an empty machine (can be as well cloud or other virtual machines such as Amazon EC2 or DigitalOcean) will do the trick.

It downloads the installer files from a configurable location and installs Oracle 11. After it has finished, you can just connect to the running database server on port 1521.

The recipe is here: https://github.com/echaeu/echa-oracle

The creator has some info on his blog as well: http://oraarir.blogspot.fi/

 

 

 

 

 

I

1 Comment

Best development environment for Clojure ?

As I am starting to learn Clojure, the questions of the best development environment was there from day one.

Build tool

Regarding the “build environment”, which is related, the answer was pretty clear. Nearly everyboddy seems to use Leiningen, so I did.

And it is a very good tool for the jobs. It misses maybe some plugins, compared to maven for Java, but that it just a question of time.

I started to contribute 2 Leiningen  plugins, to execute Karma and Protractor from Leiningen, and so allow a “one command build” for Clojure/Javascript projects, by running


lein build

You can find them here:

IDE

Coming from  a Java background, I am used to have very strong language specific support in my IDE to allow very quick code navigation, refactoring support and so on. Working with a IDE like this, there is very little actual typing.

To expect the same support in a dynamic language as Clojure, is probably impossible.

In any case I am still doubting, which is the best IDE to use. I tried the following 3:

  1. La Clojure in Intellij
  2. lighttable
  3. Emacs

I am not 100% convinced on any of the three. All of them have strengths and weaknesses, so there is no clear winner.

La Clojure has a  UI  very close to the Java IDE. But the standard feature, such as auto-completion and synchronization between code in editors and the repl do not wok that well, compared to Java or compared to Emacs.

On this Emacs shines. It is more tricky to setup, as it is very flexible and there are packages for everything, which need to be combined in the right way. But if it works ones, it is very reliable and allows a very efficient workflow.

Lighttable is the newest of the three. Thanks to it’s instant evaluation of code,as you type, the feedback loops should become even shorter. But as it is new, it still lacks a lot of features compared to the others.

All of three support a generic way of doing a TDD driven style with Clojure, which is to have three windows open to see at the same time:

  1. Current namespace to work on
  2. Tests for current name space
  3. a autotest enabled buffer, in which all tests get automatically run on each code changes (and pressing save)

The difference between the three is in which situations something gets out-of-sync and you need to restart the Repl with the autotests. They differ as well in the level of support, to copy pieces of code between the Repl buffers and the code windows. This is an important factor, as development with Clojure is typically Repl based.

,

3 Comments

“To-do” web app done in TDD style with Clojure and AngularJS (part 3)

One command to run all tests

The subject of today’s blog is the build of the application with Leiningen. It is a good practice to start a project with the build infrastructure, even before writing the first line of code.

Even though historically dynamic languages such as JavaScript focus less on the build, I believe it is very essential to have a single command build right from the start as well for JavaScript / Clojure based projects.

My goal was to have a single build command, which compiles the Clojure code and runs the 3 different types of tests I need:

  1. Unit tests for all Clojure code
  2. JavaScript Unit tests for AngularJS
  3. End-to-end tests for AngularJS

Using these three types of tests allow all code and parts of the html to be developed in a pure test first style. This means to not write any line of code, without having a failing test first.

Clojure unit tests with Midje

To test the Clojure code, I decided to use the Midje framework.

AngularJS unit tests

Specifically for AngularJS the 2 types of tests drive different types of code. The unit tests drive the creation of AngularJS controllers, directives and services. Thanks to dependency injection in AngularJS, the test of these is very easy and does not do any type of DOM manipulation. Nevertheless this tests should run in actual browsers (and not in a standalone JavaScript interpreter) to be as close to the production environment as possible.

As a tool to run the Angular unit tests, I have chosen Karma

AbgularJS end-to-end tests

The end-to-end tests drive the html files, more specifically the usage of the AngularJS specific code in html such as attributes and directives.

In our client/architecture architecture with a JavaScript client and a Clojure server which communicate via Json the e2e tests are the only way to assure that the Json data produced by the Clojure code fits with the Json expected by the client code.

As a tool to run the Angular unit tests, I have chosen protractor 

Integrate all tests into Leiningen

So in order to have a single-command-build, Leiningen need to be configured to execute:

  1. The Clojure/Java compiler
  2. The Midje test runner
  3. The Karma test runner
  4. The Protractor test runner, which itself needs to have started:
    1. The web server which exposes the Json REST API of the application
    2. Selenium Server, as Protractor is based on Selenium

Karma plugin for Leiningen

For integrating the JavaScript/Node.js based tools Karma and Protractor I did not find any Leiningen plugin. So I decided to develop a plugin for Karma my self, which is hosted here: https://github.com/behrica/leiningen-karma-plugin . It was trivial to do, because it just needs to execute the command line tool “karma” and pass a configuration file.

Run Protractor from Leiningen

I wanted to do something similar for Protractor, but it was not obvious to develop a standalone Leiningen plugin for this, as the start and stop of the web server and the selenium server need to be manged as well. So the easiest solution was to run some custom Clojure code during the build which does the following steps:

  1. Initialises the database to a known state
  2. Start Selenium Server
  3. Start Clojure REST API server (based on Jetty)
  4. Execute command line “protractor” and report result
  5. Stop Clojure REST API server
  6. Stop Selenium Server

This runs therefore an Integration Test including browser, REST server and a real database, in my case an embedded H2 database. (but using a database server such as MySql would work similar)

The Clojure code for this Integration test looks like this:

todo.it:

(defn -main [& [port]]
 (init-database)
 (System/setProperty "webdriver.chrome.driver" "/home/carsten/bin/chromedriver")
 (let [seleniumServer (org.openqa.selenium.server.SeleniumServer.)]
   (.start seleniumServer)
   (let [appServer (run-jetty #'app {:port 8080 :join? false :daemon? true})]
     (println (:out (sh "protractor" "resources/protractor_conf.js")))
     (.stop appServer)
   (.stop seleniumServer)
   (System/exit 0))))

I am not sure, if it would be difficult to develop a standalone Leiningen plugin which does this all together. It could be based on the code or be an extension on the Leiningen-ring plugin, as it needs to start/stop Selenium before/after and run a command  to start protractor.

Having all this set up I can run the full build with all test like this:

lein do clean, install, midje, karma, run -m todo.it

The full code is available here: https://github.com/behrica/todo-clojure

Leave a comment

“To-do” web app done in TDD style with Clojure and AngularJS (part 2)

First I want to explain the motivations for the technological choices and why I believe there will play a big role in future application development.

Why a web application ?

The future for application development is the Internet. More and more applications need to be always available from all devices. The only valid technical option for this right now is a “web application”.

Done it right it can produce almost desktop-like look and feel; that what “web 2.0” mainly means.

Why Clojure ?

I have a very strong Java background and like a lot OO modelling. So Clojure is a radical change and a lot to learn. But thanks to the Java platform, Clojure allows reuse of all existing Java libraries and this means that an adoption of Clojure can be done with less risks. I always wanted to do more work in Smalltalk, an other fascinating platform, but the lack of libraries compared to Java is a strong argument against using it.

For a non-statically-typed language such as Clojure it is in general much more difficult to come up with intelligent IDE support for auto-completion and refactoring. This  has an impact on productivity.

BUT Clojure focuses on REPL based programming. So code gets always tried very quickly in the REPL and pasted into the source code files.

Combining this with advanced auto-loading / auto-test-running mechanisms which gives immediate feedback might compensate the productivity loss due to its non-static nature. Finally using TDD can help to have a very high code coverage, so manual refactorings have their safety net.

So I believe that the combination of all the following can yield in very high productivity in Clojure:

  • Simple syntax of Clojure itself
  • Using TDD in order to detect regression due to refactorings
  • Super fast feedback by using REPL and auto-load/auto-test
  • reuse of all Java libraries
  • The proponents of Clojure claim higher productivity by using “a functional language”. I cannot judge on this yet.
  • Using a smart IDE with strong Clojure support just as Intellij, Eclipse , Emacs, or LightTable

Why AngularJS

To start the argument,  some history on Java web application development.

I did in the past classic Java server side web applications. This means something runs on the server and generates html on the fly for each request. There are various “generations” of this type of frameworks, and they got  easier to use in each generation and productivity increased as well:

  • Servlet API
  • JSP
  • Struts-like frameworks ( “Action” based)
  • Wicket or JSF as component oriented frameworks

The common theme on them was to more-and-more abstract from the fundamental stateless request/response architecture on which http is build.

The component-oriented ones allow a nearly full OO style of coding, by creating your own components easily and reuse them.

Then came Ajax and the need for partial page refreshes…. The component oriented frameworks support it now, but it still feels clunky sometimes, as you need to deal with producing JavaScript code by Java code ….

So in the end we still write Java application which produce html code and JavaScript code….

In the last years a new trend got hugely popular due to the universal presence of JavaScript in each browser.  Thanks to standardization organisations, browser vendors and projects such as jQuery it is now feasible to write a web 2.0 desktop-like web application, which is supported by nearly all browsers without need to develop browser specific code.

I still didn’t find this approach very interesting, due to the language JavaScript and the rather poor existing tool support for JavaScript. It seemed not be ready year for “large scale” projects and ease of maintenance which is required for long running projects.

From my personal opinion, AngularJS brings a change here due to it’s features such as:

  1. Pure html as template language
  2. Removing a lot of boiler-plate code due to it’s two-way-databinding
  3. The concrete MVC model used by AngularJS allows very clean separation of UI and controller
  4. “Directives” allow extending html with custom tags and bring reusable components
  5. Created with testability in mind. All parts are easily testable thanks to depedency injection framework
  6. Contains module system

Why a client/server architecture ?

The client/server architecture style I mean here, is to have back end JSON services, which expose CRUD operations for all data and other “command methods” to operate on the data. It should be completely UI agnostic. This API is used by the client, so the Javascript running in the browser

This architecture has several advantages, which become even more important with the idea of cloud computing.

The frequently mentioned advantages are:

  • Supports inherently multiple device, so the same API can be used for desktop browser and phone/tablet (browser / app)
  • fits well to the HTTP protocol
  • can be scale up easily

In my view, a requirement for a today successful cloud application, is the existence of a full featured API. With this people can integrated the cloud service with their services and applications, and are less afraid to depend their business on something external. It can be used as well for potential backup and migrations scenario. In a lot of situations it helps a lot to use a SaaS, if you know that you just “could get all data out”, if you want. This helps IT manager to sleep better.

By using the upper mentioned client/server architecture this API does not need to be developed extra. It exists anyway, because the same API is used by the JavaScript client in the browser. So this API comes basically for free.

Why TDD ?

No further words are needed on TDD (Test Driven Development). I will never code again without it.

Unit test avoid the most boring task of a developers life, which  is repeated manual retesting of all code to avoid regression bugs. To be very sure, you need 100 % test coverage, and the easiest way to have it is TDD. And it results in good software design as well.

I would be very happy to get some feedback on my arguments.

The next posts will go in more technical details.

Leave a comment

“To-do” web app done in TDD style with Clojure and AngularJS (part 1)

I recently discovered various rather new  tools, which in my view will have a very strong influence on the way we will do web applications in the future. So I always wanted to combine them in a single demo web application.

I finally started to do so, and will blog in various parts about it. The application will be the well known “to-do list” with a web front end and a database back end.

The tools I want to use are

  • TDD as a general way to write all code
  • Clojure as the language of the back end
  • JavaScript  and AngluarJS for the front end
  • JSON REST services to glue back end and front end together
  • Leiningen to run a “build all” with a single command
  • Bower to manage client-side javascript libraries
  • Three tools for testing everything:
    • Karma for testong all JavaScript
    • Protractor for end-to-end tests
    • Midje for testing Clojure code

Stay with me for further posts.

 

Leave a comment

Let’s try blogging

I thought it’s about the time as well for me to talk  about great software development tools, such as TDD, Clojure, AngularJS, Chef

Leave a comment