Spring-Boot & Netflix OSS - An adventure into microservices

Honestly, I still need convincing on microservices.

I can see that they are a compelling argument compared to a monolithic application, but I think I need to get my head around some of the challenges they face - the first one that comes to mind being how to effectively define the microservice boundaries - as it seems to me a lot of the applications I have ever worked with are monolithic because these boundaries are so blurred.

Anyway, I wanted to do some tech stuff, so decided to start building out an application using the microservice architectural pattern and Spring Boot seemed like a good place to get started.

This is very much a work in progress, and I am continuing to progress through different aspects of the application and at the moment there is very little actual code written (in part that is due to the simplicity that Spring-Boot provides).  All code is being kept up to date in GitHub so feel free to have a look at that.

There are lots of great blogs covering this stuff already, so won't re-cover their work, the following article gives a great write up of the Netflix OSS and the Spring integration which is worth reading:


(Image from: Building Microservices with Spring-Cloud and Netflix OSS)

Getting started: A service registry - Eureka

One of the first things that is needed is a central Service Registry to allow service-discovery - this is not a new concept to microservices and is an approach used by SOA.  Straight out of the box, Spring-Boot provides integration with Netflix's OSS application Eureka, that provides this.  I opted to have a dedicated application for my registry (code can be seen here) and it really is as simple as adding the relevant dependencies to the build.gradle file, adding an @EnableEurekaServer annotation to our application config then a simple config file defining the server port/name etc and its done!  You can just run gradle assemble in that project to build the JAR file, then run java -jar [the new JAR file] and the application will spin up - you should then be able to go to http://localhost:1111 and you will see the Eureka dashboard (with no microservices registered of course).

My first microservice

So, I had Eureka up and running, but it was looking pretty lonely with no services registered.

A microservice in Spring is also very simple - as really, all it is is a simple web application that runs in its own process with a limited domain - so spinning up a Spring Boot MVC RESTful webapp with a single controller/endpoint is enough to get me a microservice (even just a tweet would do it..)
So we can create our new microservice to do anything we like, in my case I created a QuoteService (the application is slowly evolving into an insurance engine).  Just having the standalone app isn't helping much, so we need to add some configuration to tell the service to register with our Eureka server - this will make our new microservice discoverable by other services wanting to use it. 

Again this is quite simple: we need to tell our application it should try and register with Eureka, and we should add the config to do so:

You can see that we simply annotate our application config in java, and then add some properties that define where the Eureka server is hosted and that's basically it.

Now if we build the project JAR and start up again (and we still have our Eureka service registry running) then after 30seconds or so you should see the Quote-Service registered and ready to use.

On to the next one.. 

Now, we have a microservice, and we have a registry that makes it discoverable, but still - just one microservice is pretty lonely. So next I created another dummy RESTful microservice, this time called ProductService which just followed the same pattern as the first.

Once that was started up then the Eureka dashboard started looking a bit happier with the two services registered - the obvious next challenge is seamless interaction between the two: splitting the services into their own processes is all good an well, but meaningless if you can't easily integrate them.  The way I look at it is when reading the application code of a service (or application using microservices) then it should just look like a normal application with sevice classes - there shouldn't be any fanfare around the fact that my service class actually gets the data from a dedicated microservice over HTTP/AMQP rather than just getting directly from the DB in the traditional way.

So, still just stubbing out the endpoints, I updated my QuoteService endpoint to make a call to my ProductService, and then just jammed that response into the JSON response I was returning anyway:

As you can see, it could be a standard controller in a normal monolithic application from this point, we are just calling a method on our autowired ProductService class and returning that.

So the really interesting part is in the ProductService class - at the moment this isn't a really elegant, abstracted class yes, so there is still some boiler plate, but that will have the advantage of making it clear what is going on:

As you can see, it's just making a REST call to the Product microservice and returning the response cast as a Map - but the really nice part of this is that the service url is just the service name (in this case "PRODUCT-SERVICE", that is injected to the class)  and with the RestTemplate annotated with Spring-Cloud's @LoadBalanced that microservice will be looked up in Eureka (and load balanced if there is more than one PRODUCT-SERVICE running).

So our setup is starting to take shape now - we have two microservices, both registered with Eureka and able to interact with each other in a fairly clean, loosely coupled way.

Don't push me, 'cos I'm close to the edge..

As your microservices start to proliferate, you will get different levels of service granularity, and undoubtedly you won't just want to expose all your microservices as a public API.  One option would be to create a RESTful application and define nicely named endpoints you want to expose and then use the standard integration described above to integrate it.

Fortunately, there is an easier way - Netflix provides a library called Zuul that can be simply configured to map URL patterns to given defined service names (and again looks up in the Eureka service registry).  Much like Eureka, this is super easy to setup and just needs an annotation and the config again:

And the config is pretty easy to understand:

As you can see, we just define service names against URL patterns.

Now once all are apps are up and running, and the microservices are registered on Eureka then you have a single API interface to start interacting with the services (rather than having to access each service on its designated port etc).


 So that's as far as I have got - I wired up the QuoteService to MongoDB so the data all gets persisted there (and have added a get quote endpoint which gets the same data from mongo) and starting to wire up the product service with JPA.  So far it's been enjoyable, and things are making more sense than when I started - but there are a few questions still:

  • It seems like there is still duplication of service names throughout the different projects - for example the ProductService name ("product-service" - case insensitive) is proliferated throughtout - the service itself defines it, the QuoteService needs to know the name of the service, the Zuul edge server needs to know the name etc.  I guess this is unavoidable as these are intrinsic dependencies but still seems a bit flaky.
  • It feels like the Service classes could be factored out - our ProductService class that allows HTTP REST interactions with the Product microservice would likely need to be re-used across all applications/microservices that need to use the Product microservice

NerdAbility - A presentation

I recently had to give a presentation about NerdAbility, so for fun, here are the slides I put together. It was just an intro to the product, where the idea came from and what it was trying to solve, followed by a brief jump into one or two interesting elements of the project from a tech point of view.

Innovating towards the traditional - Facebook & Amazon

Just a short post now - and really just two observations/trends I have commented on before that are continuing their trajectory.

Amazon bring grocery delivery to the UK - Announced today, Amazon will be bringing their "Fresh" service to the UK, starting with London and Leeds - continuing with my previous observations on innovating towards the traditional.

Facebook messenger no longer needs a Facebook account - this is older news, and I have been meaning to comment on this for a while. Basically, customers can now sign up for and start using Messenger without an account, just using a phone number, which is an un-surprising move towards phone as identity and recognising that the phone has the potential to be a node in any social connected network on the platform (much like I have preached about before..).  It is also another step towards Facebook moving further down the stack and more directly competing with native messaging apps as a system level.

Hello Facebook..

I have been away on a mini-break for the last few days, and whilst relaxing I was mentally planning out a blog post about mobile phones as identity and the social platform they provide (again!), then, I woke up this morning to Facebook's announcement of their Hello app.  In short the app will attach FB user details to phone numbers, so if you get a call from a mobile number you don't know, but have the app installed on your phone, it will attempt display FB info to identify the caller.

For my money, this is a very smart move.

Moving down the stack

One of the questions for the future mobile landscape is whether players like Facebook and Amazon can move "further down the stack" - that is, to start being more core to devices, or making their own. Amazon have tried with their Kindle Fire phone, and Facebook had Home - both bombed out, however, and there hasn't been much to suggest that there is more promise for them.

However, Hello seems to me like it could be a small step to bridging that gap.  The app is simple in itself, and done well I could see it being very popular - after installing, it doesn't need any active engagement from the user, but (hopefully - without having tried it myself) just sits there and enhances the users phone experience (as long as they don't try and do anything stupid, crazy annoying notifications, battery hogging etc) - As a simple enhancement to the incoming call screen, I can imagine pretty high retention/install rate - because why would you go back to unknown numbers?

One reason this could be a huge win for Facebook is that this step could make Facebook more core to the mobile device (and seemingly appear more and more like just part of the OS).  If you have the app installed, and miss a call, I would expect the missed call notification screen to be Facebook's Hello - with the normal stuff you would expect - so and so called you, return call, send message etc - which will inevitably give FB the hook to drive users to FB messenger, WhatsApp etc - This is a chance for FB to hijack the core functionality of the phone and drive traffic to its other properties, quickly FB messenger could become the default messaging app.

Your phone as your identity & network

This is something that I have ranted about before - there are two powerful and fairly unique traits of mobile phones that have the power to really disrupt the social/technology/online space:

Your phone is your identity - Whilst people continue to struggle to fix online identification/authentication issues, with a host of different providers offering Login-as-a-Service (Twitter, Facebook etc) and lots of people complaining about the proliferation of user logins/passwords that need to be remembered, the phone offers an interesting solution to that - Granted, phone numbers are not for life, but they offer simple, unique identification of individuals, plus, with two-factor authentication, it can be simple to verify that the person attempting to login is in physical possession of the device.  And with more moves to ApplePay and NFC payment with your device, this identity/device becomes even more tightly coupled.  

By Facebook tying your phone number with your profile it is an impressive move to help future proof their platform: so your phone is your identity? Well now your FB profile is intrinsically linked to that too.

Your phone as a social network - For a long time, Facebook have been the dominant force in social networking. Despite Google's effort with Google+, they haven't been able to displace the social network king. Do you think they struggled because they couldn't match Facebook's superior product? You think people stay at Facebook because they just love the features so much? Of course not. They stay because of user inertia - the feature that keeps users at Facebook is the network effect, no one would move to Google+ without the rest of the network, and how to you move your network?  As if to prove this, there are fairly regular outcries from the Facebook masses when FB changes their layout, or people find out about some terrible clause in the privacy policy - but do people leave? Nope. They just get on with it and keep using Facebook.  After all, the users are the real product, not the software.

Now, mobile phones change that a bit.  Traditional online social-networking basically just models the real world network of friends/colleagues/etc - In your real life social network, you are the node in the graph, and your mobile phone basically represents your node - If you think of Facebook as a centralised social network, mobile phones are a distributed social network (in other words, Facebook knows about the entire graph, and keeps that information centralised, but with your phone you know about your immediate neighbours in the graph - e.g. the people you have contact details for, but not the rest of the graph).  This is a pretty powerful idea - and this is what has made it possible for apps like SnapChat and WhatsApp to enjoy sudden and un-precedented user growth where non-mobile only apps (Google+) haven't.

If we add to that the fact that modern smart phones aren't just an address book - they have your photos, they have your videos, they have your messaging, they have your friends - is there much on your Facebook profile that you don't have (the ability) to have locally on your mobile phone?

All of this makes the smart phone and the real world mobile connections a very powerful platform for anyone to come and disrupt Facebook's crown - So this move by Facebook again just helps establish them as leading the mobile-social platform, not just an old internet company trying to compete.

An aside - Google needs to catch up

My original thinking for the blog was going to be about how Google should, in my opinion, be capitalising on the Android platform as a social network, and rather than trying to convince users to sign up to Google+ (or Wave, or Buzz etc) focus their efforts on making Android a more social OS and make more of the Android as your identity.  They would have to do it carefully with regards to stuff like "private by default" so not to cause uproar, but if they could effectively make all their Android users a node in the social network, then all of a sudden they could find themselves with a pretty good social network, and can start thinking about how to add value to the product.  Furthermore, with Android being the OS (as low down the stack as possible) then they would have a lot more options with regards bringing together all the aspects above - Your smart phone knows instantly when you take a photo or record a video (because you are doing it with the device!), it knows your location, it knows when/who you are messaging, it knows who your friends are - some nice usability around that and it starts to sound like a very strong value proposition!

Nerdability: A retrospective

In 2011, I came up with the idea to build a web based platform that lets developers and technologists build better CVs.  The idea was that as a technologist, we have a prety big online footprint that represents our skills/interests/experiences much more than a traditional CV that just rattles off academic achievements and work experience.

As a technologist applying for jobs, I wanted to make my CV stand out, and figured I have a StackOverflow profile that could demonstrate my knowledge & communication skills, I had Google-Code/GitHub/BitBucket projects which showed examples of my code, OSS contributions, technologies I have used plus then LinkedIn, blogs, Coursera courses, Geeklist.. the list goes on - and more importantly, these were things that were all changing much faster than my traditional CV was.

On the flipside of that, as an employer trying to recruit technologists is really hard - CV screening doesn't rule out many candidates, and beyond that you are left to navigate technical tests & interviews which are fraught with difficulties in trying to really assess whether someone is actually fir for a job.

Nerdability started life as an idea that was entered in a cloud competition to build a webapp (some details here originally called NerdStar) - which it won.  I then recruited two co-founders and we re-wrote a bunch of it (originally all data persistence was mongo, but switched that to relational) and launched it to the public!

In January this year (2015) we retired the application, we had somewhere in the region of 800 users signed up (with no active/paid marketing).  We have currently parked the page and a basic user profile tour on nerdability.zz.vc and are in the process of moving this holding page back to nerdability.com.

Things that went well

There was a lot of great things we learnt/that came out of the project, here are some highlights:

We contributed to OSS - During development I created a Spring-Social implementation for Khan Academy API and for the GeekList API - both of which are now listed on the Spring site as community projects.

The content driven marketing - As mentioned, we didn't do any marketing, just a few tweets, etc - but the big win was the blog. We setup a basic blog over at blog.nerdability.com (still up and running!) using blogger for hosting and just posted tech articles - tutorial type stuff plus more link-bait-y type stuff like new years resolutions for developers, or getting hackathon projects into production - and the blog drove lots of our traffic.  The beauty was the people who visited/discovered our blog were a perfect match for our target demographic, so we just had plenty of placements for nerdability and the users just trickled in!  The link-bait stuff would give us spikes of traffic, but the tutorial/how-to stuff still drives a few thousand unique views a week.

User response - The user response was good, we had a fair few early adopters who liked what we were doing and blogged/tweeted positively about the project.

Things that were hard

Competition - The weekend we were due to launch, I got an early beta-invite to StackCareers - with an almost identical value proposition, but they already have money and a massive developer community.

Two-sided marketplace problem - so we started getting users signed up, but there was little engagement as users couldn't really do much with their profiles - we had not companies/jobs listed, so beyond just sharing their profile links there was not much they could do.  We were working on the company integration when we finished, but didn't get into it.


In the end, the three of us weren't at the right place/point in time to go into the project full time (family stuff, wrong career stage etc). However, I still believe there is a good business model in the idea, and still think that even if it was setup today it could be successful.  Despite the community and backing of StackCareers, I don't think they have really nailed it, and seem to have stagnated at a slightly improved jobs board - but not really using the intelligence/information to really improve the recruitment process and take any burden of employers/interviewers (which is where I think one of the big wins is for the model)

The joyless world of a data-less startup

I recently read "The Joyless World of Data-Driven Startups" over on medium.

It's a nice article, and there are plenty of good points made - but my concern with the article is that it champions data-less decisions without covering the times when data-less or gut-instinct decisions are not appropriate.  In many ways it mirrors some conversations I have had over the past year or two regarding the benefits of data and using the Build-Measure-Learn approach from the Lean methodology.

I actually think the article hits the nail on the head, but really this post is to address the fact that some people may read what they want from it (and use it as an excuse not to measure).  I think really, the main take away point from the article is this:

Data-driven innovation sucks.

That is really what is at the crux of the article, using data to drive innovation is as close to paint-by-numbers as you can get.  Think about it, innovation is by its very nature a creative process - its the act of creating something new, or something that tackles a particular domain or problem in a new and original way - obviously not something that can be done formulaic-ally by numbers.  Creating a new and exciting product has to be creative by nature. Creating a clone of an existing product/service is not (that's not to say that it wouldn't still be a viable business though!).

The problem is, in the past I have ended up in lots of conversations as a proponent of Lean and the B-M-L iterative approach to attempting to better understand the solution and problem domain and have faced this very argument - that its OK that we aren't measuring result correctly(if at all!), and its OK that we are releasing multiple changes, and pulling lots of levers, all at once (thereby making data we do measure difficult to tie back to single changes), because just look at Steve Jobs! He didn't bother about asking what users wanted! he just gave them his vision!

And that is the main distinction I would like to highlight - there is a big difference between using data to drive your creative/innovative decisions and using data to learn from your innovations.  Yes its OK to be creative and out-of-the-box with innovation and new product features - that's how we got so many amazing products that we have today - but you need to be ruthless in your learning and understanding of the data.

To the Steve Jobs example - yes, he was a visionary and he ignored data/user research when designing products - Jobs/Apple are famous for their approach of just giving customers what they decide they want, and with great success - but they still use the data.  You think if the iPhone was a dismal failure they would have ignored that data and not changed it? You think Apple haven't killed off products that haven't performed well?  These are fairly extreme examples, as in both cases the data is very visible (e.g. sales figures) that people might not think of as post-product learning, but really its the same thing.

Essentially, the bottom line is whilst data-driven innovation sucks, data-driven learning is essential.

That's my opinion, anyway.