Spring, Reactor & Event driven programming on the JVM

Following my previous outing playing with the Spring-Boot microservices stuff, I once again found myself looking through some of the Spring libraries and came across the Reactor integration stuff.  It looked interesting, and thought I would have a quick look at the asynchronous event-driven model.

As always with Spring Boot, it was super simple to get an app up and running - it's worth noting that the app doesn't really get that much into the async processing side (or really expose scenarios with potential benefits/pitfalls of the approach) - but it gets an app up and running pretty easily.

Event streaming

Obviously, to get started I needed some kind of event source (I could have just stubbed out some code to just randomly create events in the system, but I wanted something more real). Obviously, with the modern web & big data, we have a tonne of events that we could use. I went for the obvious choice of the Twitter streaming API - as it exposes a streaming API I could just connect to that and on receipt of any tweet then just push an event on to my EventBus for any interested parties to process.

The basics of connecting to the Twitter streaming API are pretty simple using Spring-Boot and Spring-Social - I just created a simple Spring-Boot webapp that just exposed a simple page to connect to Twitter (OAuth via Spring-Social) and then on connection just connected to the streaming API and started listening.

First I just connected to the sample "firehose" stream, which is supposed to be 1% of the total twitter stream (I saw reported that there are 500million tweets a day, so you'd be looking at about 5million random tweets sent a day) - But I decided to consume the tweet events to pull out data about the current ongoing Rugby World Cup (England 2015), so I switched to the search stream limiting by references to the world cup.

The searching stream provided a reasonable number of tweets, and I think whilst I was running during the England vs Australia game I processed ~500,000 rugby related tweets.

Configuring reactor

Getting reactor up and running is really easy with Spring:

The EventBus configuration is interesting, as it has options to use different patterns including the LMAX pattern - in this case I just went with a standard thread pool approach.

The tweet-eater

To be honest, the use of reactor was overkill for this experiment, as the Spring-Social API allows you to define listeners for the streaming APIs which have a handle tweet method, much like an event consuming interface. But as it was just an experiment I continued and just used that listener to push the events on to the event bus.

As you can see above, its pretty simple - the listener just creates the event object and pushes it onto the EventBus.  So, with that done, and the basic Spring-Social connection setup to listen to the stream we will have a nice flow of events being pushed (at quite a high rate!) onto the EventBus. Now we just need something to consume those events.

Event consumers

The first consumer I created was just a very basic logging consumer - all it did was count events and then log numbers

Pretty simple

Next up,  I created a basic consumer to inspect the tweets, identify rugby teams mentioned and persist the data to Redis - this was also pretty easy, as Redis integration works pretty simply and Twitter created a set of standard team hashtags for the competition - So I just mapped those to countries, and setup my consumer to check for those

I quickly stuck some lipstick on the interface to display number of tweets per country and we were off!

Like I said, it's only barely scratching the surface of async event based programming, but it shows how easy it is to get the EventBus up and running with Spring.

As always, the code for the full app is over on GitHub - if you just clone the project and add in your own Twitter API keys in the config then you should just be able to build the JAR and run it directly.

Spring-Boot & Netflix OSS - An adventure into microservices

Honestly, I still need convincing on microservices.

I can see that they are a compelling argument compared to a monolithic application, but I think I need to get my head around some of the challenges they face - the first one that comes to mind being how to effectively define the microservice boundaries - as it seems to me a lot of the applications I have ever worked with are monolithic because these boundaries are so blurred.

Anyway, I wanted to do some tech stuff, so decided to start building out an application using the microservice architectural pattern and Spring Boot seemed like a good place to get started.

This is very much a work in progress, and I am continuing to progress through different aspects of the application and at the moment there is very little actual code written (in part that is due to the simplicity that Spring-Boot provides).  All code is being kept up to date in GitHub so feel free to have a look at that.

There are lots of great blogs covering this stuff already, so won't re-cover their work, the following article gives a great write up of the Netflix OSS and the Spring integration which is worth reading:


(Image from: Building Microservices with Spring-Cloud and Netflix OSS)

Getting started: A service registry - Eureka

One of the first things that is needed is a central Service Registry to allow service-discovery - this is not a new concept to microservices and is an approach used by SOA.  Straight out of the box, Spring-Boot provides integration with Netflix's OSS application Eureka, that provides this.  I opted to have a dedicated application for my registry (code can be seen here) and it really is as simple as adding the relevant dependencies to the build.gradle file, adding an @EnableEurekaServer annotation to our application config then a simple config file defining the server port/name etc and its done!  You can just run gradle assemble in that project to build the JAR file, then run java -jar [the new JAR file] and the application will spin up - you should then be able to go to http://localhost:1111 and you will see the Eureka dashboard (with no microservices registered of course).

My first microservice

So, I had Eureka up and running, but it was looking pretty lonely with no services registered.

A microservice in Spring is also very simple - as really, all it is is a simple web application that runs in its own process with a limited domain - so spinning up a Spring Boot MVC RESTful webapp with a single controller/endpoint is enough to get me a microservice (even just a tweet would do it..)
So we can create our new microservice to do anything we like, in my case I created a QuoteService (the application is slowly evolving into an insurance engine).  Just having the standalone app isn't helping much, so we need to add some configuration to tell the service to register with our Eureka server - this will make our new microservice discoverable by other services wanting to use it. 

Again this is quite simple: we need to tell our application it should try and register with Eureka, and we should add the config to do so:

You can see that we simply annotate our application config in java, and then add some properties that define where the Eureka server is hosted and that's basically it.

Now if we build the project JAR and start up again (and we still have our Eureka service registry running) then after 30seconds or so you should see the Quote-Service registered and ready to use.

On to the next one.. 

Now, we have a microservice, and we have a registry that makes it discoverable, but still - just one microservice is pretty lonely. So next I created another dummy RESTful microservice, this time called ProductService which just followed the same pattern as the first.

Once that was started up then the Eureka dashboard started looking a bit happier with the two services registered - the obvious next challenge is seamless interaction between the two: splitting the services into their own processes is all good an well, but meaningless if you can't easily integrate them.  The way I look at it is when reading the application code of a service (or application using microservices) then it should just look like a normal application with sevice classes - there shouldn't be any fanfare around the fact that my service class actually gets the data from a dedicated microservice over HTTP/AMQP rather than just getting directly from the DB in the traditional way.

So, still just stubbing out the endpoints, I updated my QuoteService endpoint to make a call to my ProductService, and then just jammed that response into the JSON response I was returning anyway:

As you can see, it could be a standard controller in a normal monolithic application from this point, we are just calling a method on our autowired ProductService class and returning that.

So the really interesting part is in the ProductService class - at the moment this isn't a really elegant, abstracted class yes, so there is still some boiler plate, but that will have the advantage of making it clear what is going on:

As you can see, it's just making a REST call to the Product microservice and returning the response cast as a Map - but the really nice part of this is that the service url is just the service name (in this case "PRODUCT-SERVICE", that is injected to the class)  and with the RestTemplate annotated with Spring-Cloud's @LoadBalanced that microservice will be looked up in Eureka (and load balanced if there is more than one PRODUCT-SERVICE running).

So our setup is starting to take shape now - we have two microservices, both registered with Eureka and able to interact with each other in a fairly clean, loosely coupled way.

Don't push me, 'cos I'm close to the edge..

As your microservices start to proliferate, you will get different levels of service granularity, and undoubtedly you won't just want to expose all your microservices as a public API.  One option would be to create a RESTful application and define nicely named endpoints you want to expose and then use the standard integration described above to integrate it.

Fortunately, there is an easier way - Netflix provides a library called Zuul that can be simply configured to map URL patterns to given defined service names (and again looks up in the Eureka service registry).  Much like Eureka, this is super easy to setup and just needs an annotation and the config again:

And the config is pretty easy to understand:

As you can see, we just define service names against URL patterns.

Now once all are apps are up and running, and the microservices are registered on Eureka then you have a single API interface to start interacting with the services (rather than having to access each service on its designated port etc).


 So that's as far as I have got - I wired up the QuoteService to MongoDB so the data all gets persisted there (and have added a get quote endpoint which gets the same data from mongo) and starting to wire up the product service with JPA.  So far it's been enjoyable, and things are making more sense than when I started - but there are a few questions still:

  • It seems like there is still duplication of service names throughout the different projects - for example the ProductService name ("product-service" - case insensitive) is proliferated throughtout - the service itself defines it, the QuoteService needs to know the name of the service, the Zuul edge server needs to know the name etc.  I guess this is unavoidable as these are intrinsic dependencies but still seems a bit flaky.
  • It feels like the Service classes could be factored out - our ProductService class that allows HTTP REST interactions with the Product microservice would likely need to be re-used across all applications/microservices that need to use the Product microservice

NerdAbility - A presentation

I recently had to give a presentation about NerdAbility, so for fun, here are the slides I put together. It was just an intro to the product, where the idea came from and what it was trying to solve, followed by a brief jump into one or two interesting elements of the project from a tech point of view.

Innovating towards the traditional - Facebook & Amazon

Just a short post now - and really just two observations/trends I have commented on before that are continuing their trajectory.

Amazon bring grocery delivery to the UK - Announced today, Amazon will be bringing their "Fresh" service to the UK, starting with London and Leeds - continuing with my previous observations on innovating towards the traditional.

Facebook messenger no longer needs a Facebook account - this is older news, and I have been meaning to comment on this for a while. Basically, customers can now sign up for and start using Messenger without an account, just using a phone number, which is an un-surprising move towards phone as identity and recognising that the phone has the potential to be a node in any social connected network on the platform (much like I have preached about before..).  It is also another step towards Facebook moving further down the stack and more directly competing with native messaging apps as a system level.

Hello Facebook..

I have been away on a mini-break for the last few days, and whilst relaxing I was mentally planning out a blog post about mobile phones as identity and the social platform they provide (again!), then, I woke up this morning to Facebook's announcement of their Hello app.  In short the app will attach FB user details to phone numbers, so if you get a call from a mobile number you don't know, but have the app installed on your phone, it will attempt display FB info to identify the caller.

For my money, this is a very smart move.

Moving down the stack

One of the questions for the future mobile landscape is whether players like Facebook and Amazon can move "further down the stack" - that is, to start being more core to devices, or making their own. Amazon have tried with their Kindle Fire phone, and Facebook had Home - both bombed out, however, and there hasn't been much to suggest that there is more promise for them.

However, Hello seems to me like it could be a small step to bridging that gap.  The app is simple in itself, and done well I could see it being very popular - after installing, it doesn't need any active engagement from the user, but (hopefully - without having tried it myself) just sits there and enhances the users phone experience (as long as they don't try and do anything stupid, crazy annoying notifications, battery hogging etc) - As a simple enhancement to the incoming call screen, I can imagine pretty high retention/install rate - because why would you go back to unknown numbers?

One reason this could be a huge win for Facebook is that this step could make Facebook more core to the mobile device (and seemingly appear more and more like just part of the OS).  If you have the app installed, and miss a call, I would expect the missed call notification screen to be Facebook's Hello - with the normal stuff you would expect - so and so called you, return call, send message etc - which will inevitably give FB the hook to drive users to FB messenger, WhatsApp etc - This is a chance for FB to hijack the core functionality of the phone and drive traffic to its other properties, quickly FB messenger could become the default messaging app.

Your phone as your identity & network

This is something that I have ranted about before - there are two powerful and fairly unique traits of mobile phones that have the power to really disrupt the social/technology/online space:

Your phone is your identity - Whilst people continue to struggle to fix online identification/authentication issues, with a host of different providers offering Login-as-a-Service (Twitter, Facebook etc) and lots of people complaining about the proliferation of user logins/passwords that need to be remembered, the phone offers an interesting solution to that - Granted, phone numbers are not for life, but they offer simple, unique identification of individuals, plus, with two-factor authentication, it can be simple to verify that the person attempting to login is in physical possession of the device.  And with more moves to ApplePay and NFC payment with your device, this identity/device becomes even more tightly coupled.  

By Facebook tying your phone number with your profile it is an impressive move to help future proof their platform: so your phone is your identity? Well now your FB profile is intrinsically linked to that too.

Your phone as a social network - For a long time, Facebook have been the dominant force in social networking. Despite Google's effort with Google+, they haven't been able to displace the social network king. Do you think they struggled because they couldn't match Facebook's superior product? You think people stay at Facebook because they just love the features so much? Of course not. They stay because of user inertia - the feature that keeps users at Facebook is the network effect, no one would move to Google+ without the rest of the network, and how to you move your network?  As if to prove this, there are fairly regular outcries from the Facebook masses when FB changes their layout, or people find out about some terrible clause in the privacy policy - but do people leave? Nope. They just get on with it and keep using Facebook.  After all, the users are the real product, not the software.

Now, mobile phones change that a bit.  Traditional online social-networking basically just models the real world network of friends/colleagues/etc - In your real life social network, you are the node in the graph, and your mobile phone basically represents your node - If you think of Facebook as a centralised social network, mobile phones are a distributed social network (in other words, Facebook knows about the entire graph, and keeps that information centralised, but with your phone you know about your immediate neighbours in the graph - e.g. the people you have contact details for, but not the rest of the graph).  This is a pretty powerful idea - and this is what has made it possible for apps like SnapChat and WhatsApp to enjoy sudden and un-precedented user growth where non-mobile only apps (Google+) haven't.

If we add to that the fact that modern smart phones aren't just an address book - they have your photos, they have your videos, they have your messaging, they have your friends - is there much on your Facebook profile that you don't have (the ability) to have locally on your mobile phone?

All of this makes the smart phone and the real world mobile connections a very powerful platform for anyone to come and disrupt Facebook's crown - So this move by Facebook again just helps establish them as leading the mobile-social platform, not just an old internet company trying to compete.

An aside - Google needs to catch up

My original thinking for the blog was going to be about how Google should, in my opinion, be capitalising on the Android platform as a social network, and rather than trying to convince users to sign up to Google+ (or Wave, or Buzz etc) focus their efforts on making Android a more social OS and make more of the Android as your identity.  They would have to do it carefully with regards to stuff like "private by default" so not to cause uproar, but if they could effectively make all their Android users a node in the social network, then all of a sudden they could find themselves with a pretty good social network, and can start thinking about how to add value to the product.  Furthermore, with Android being the OS (as low down the stack as possible) then they would have a lot more options with regards bringing together all the aspects above - Your smart phone knows instantly when you take a photo or record a video (because you are doing it with the device!), it knows your location, it knows when/who you are messaging, it knows who your friends are - some nice usability around that and it starts to sound like a very strong value proposition!

Nerdability: A retrospective

In 2011, I came up with the idea to build a web based platform that lets developers and technologists build better CVs.  The idea was that as a technologist, we have a prety big online footprint that represents our skills/interests/experiences much more than a traditional CV that just rattles off academic achievements and work experience.

As a technologist applying for jobs, I wanted to make my CV stand out, and figured I have a StackOverflow profile that could demonstrate my knowledge & communication skills, I had Google-Code/GitHub/BitBucket projects which showed examples of my code, OSS contributions, technologies I have used plus then LinkedIn, blogs, Coursera courses, Geeklist.. the list goes on - and more importantly, these were things that were all changing much faster than my traditional CV was.

On the flipside of that, as an employer trying to recruit technologists is really hard - CV screening doesn't rule out many candidates, and beyond that you are left to navigate technical tests & interviews which are fraught with difficulties in trying to really assess whether someone is actually fir for a job.

Nerdability started life as an idea that was entered in a cloud competition to build a webapp (some details here originally called NerdStar) - which it won.  I then recruited two co-founders and we re-wrote a bunch of it (originally all data persistence was mongo, but switched that to relational) and launched it to the public!

In January this year (2015) we retired the application, we had somewhere in the region of 800 users signed up (with no active/paid marketing).  We have currently parked the page and a basic user profile tour on nerdability.zz.vc and are in the process of moving this holding page back to nerdability.com.

Things that went well

There was a lot of great things we learnt/that came out of the project, here are some highlights:

We contributed to OSS - During development I created a Spring-Social implementation for Khan Academy API and for the GeekList API - both of which are now listed on the Spring site as community projects.

The content driven marketing - As mentioned, we didn't do any marketing, just a few tweets, etc - but the big win was the blog. We setup a basic blog over at blog.nerdability.com (still up and running!) using blogger for hosting and just posted tech articles - tutorial type stuff plus more link-bait-y type stuff like new years resolutions for developers, or getting hackathon projects into production - and the blog drove lots of our traffic.  The beauty was the people who visited/discovered our blog were a perfect match for our target demographic, so we just had plenty of placements for nerdability and the users just trickled in!  The link-bait stuff would give us spikes of traffic, but the tutorial/how-to stuff still drives a few thousand unique views a week.

User response - The user response was good, we had a fair few early adopters who liked what we were doing and blogged/tweeted positively about the project.

Things that were hard

Competition - The weekend we were due to launch, I got an early beta-invite to StackCareers - with an almost identical value proposition, but they already have money and a massive developer community.

Two-sided marketplace problem - so we started getting users signed up, but there was little engagement as users couldn't really do much with their profiles - we had not companies/jobs listed, so beyond just sharing their profile links there was not much they could do.  We were working on the company integration when we finished, but didn't get into it.


In the end, the three of us weren't at the right place/point in time to go into the project full time (family stuff, wrong career stage etc). However, I still believe there is a good business model in the idea, and still think that even if it was setup today it could be successful.  Despite the community and backing of StackCareers, I don't think they have really nailed it, and seem to have stagnated at a slightly improved jobs board - but not really using the intelligence/information to really improve the recruitment process and take any burden of employers/interviewers (which is where I think one of the big wins is for the model)