Spring, Reactor & Event driven programming on the JVM

9:24 PM , , , , , , , , , 0 Comments

Following my previous outing playing with the Spring-Boot microservices stuff, I once again found myself looking through some of the Spring libraries and came across the Reactor integration stuff.  It looked interesting, and thought I would have a quick look at the asynchronous event-driven model.

As always with Spring Boot, it was super simple to get an app up and running - it's worth noting that the app doesn't really get that much into the async processing side (or really expose scenarios with potential benefits/pitfalls of the approach) - but it gets an app up and running pretty easily.


Event streaming

Obviously, to get started I needed some kind of event source (I could have just stubbed out some code to just randomly create events in the system, but I wanted something more real). Obviously, with the modern web & big data, we have a tonne of events that we could use. I went for the obvious choice of the Twitter streaming API - as it exposes a streaming API I could just connect to that and on receipt of any tweet then just push an event on to my EventBus for any interested parties to process.

The basics of connecting to the Twitter streaming API are pretty simple using Spring-Boot and Spring-Social - I just created a simple Spring-Boot webapp that just exposed a simple page to connect to Twitter (OAuth via Spring-Social) and then on connection just connected to the streaming API and started listening.




First I just connected to the sample "firehose" stream, which is supposed to be 1% of the total twitter stream (I saw reported that there are 500million tweets a day, so you'd be looking at about 5million random tweets sent a day) - But I decided to consume the tweet events to pull out data about the current ongoing Rugby World Cup (England 2015), so I switched to the search stream limiting by references to the world cup.

The searching stream provided a reasonable number of tweets, and I think whilst I was running during the England vs Australia game I processed ~500,000 rugby related tweets.


Configuring reactor

Getting reactor up and running is really easy with Spring:

The EventBus configuration is interesting, as it has options to use different patterns including the LMAX pattern - in this case I just went with a standard thread pool approach.

The tweet-eater

To be honest, the use of reactor was overkill for this experiment, as the Spring-Social API allows you to define listeners for the streaming APIs which have a handle tweet method, much like an event consuming interface. But as it was just an experiment I continued and just used that listener to push the events on to the event bus.

As you can see above, its pretty simple - the listener just creates the event object and pushes it onto the EventBus.  So, with that done, and the basic Spring-Social connection setup to listen to the stream we will have a nice flow of events being pushed (at quite a high rate!) onto the EventBus. Now we just need something to consume those events.


Event consumers

The first consumer I created was just a very basic logging consumer - all it did was count events and then log numbers

Pretty simple


Next up,  I created a basic consumer to inspect the tweets, identify rugby teams mentioned and persist the data to Redis - this was also pretty easy, as Redis integration works pretty simply and Twitter created a set of standard team hashtags for the competition - So I just mapped those to countries, and setup my consumer to check for those

I quickly stuck some lipstick on the interface to display number of tweets per country and we were off!



Like I said, it's only barely scratching the surface of async event based programming, but it shows how easy it is to get the EventBus up and running with Spring.

As always, the code for the full app is over on GitHub - if you just clone the project and add in your own Twitter API keys in the config then you should just be able to build the JAR and run it directly.

0 comments:

Spring-Boot & Netflix OSS - An adventure into microservices

10:12 PM , , , , , , , , , , 0 Comments

Honestly, I still need convincing on microservices.

I can see that they are a compelling argument compared to a monolithic application, but I think I need to get my head around some of the challenges they face - the first one that comes to mind being how to effectively define the microservice boundaries - as it seems to me a lot of the applications I have ever worked with are monolithic because these boundaries are so blurred.


Anyway, I wanted to do some tech stuff, so decided to start building out an application using the microservice architectural pattern and Spring Boot seemed like a good place to get started.

This is very much a work in progress, and I am continuing to progress through different aspects of the application and at the moment there is very little actual code written (in part that is due to the simplicity that Spring-Boot provides).  All code is being kept up to date in GitHub so feel free to have a look at that.


There are lots of great blogs covering this stuff already, so won't re-cover their work, the following article gives a great write up of the Netflix OSS and the Spring integration which is worth reading:

http://callistaenterprise.se/blogg/teknik/2015/04/10/building-microservices-with-spring-cloud-and-netflix-oss-part-1/

(Image from: Building Microservices with Spring-Cloud and Netflix OSS)


Getting started: A service registry - Eureka

One of the first things that is needed is a central Service Registry to allow service-discovery - this is not a new concept to microservices and is an approach used by SOA.  Straight out of the box, Spring-Boot provides integration with Netflix's OSS application Eureka, that provides this.  I opted to have a dedicated application for my registry (code can be seen here) and it really is as simple as adding the relevant dependencies to the build.gradle file, adding an @EnableEurekaServer annotation to our application config then a simple config file defining the server port/name etc and its done!  You can just run gradle assemble in that project to build the JAR file, then run java -jar [the new JAR file] and the application will spin up - you should then be able to go to http://localhost:1111 and you will see the Eureka dashboard (with no microservices registered of course).

My first microservice

So, I had Eureka up and running, but it was looking pretty lonely with no services registered.

A microservice in Spring is also very simple - as really, all it is is a simple web application that runs in its own process with a limited domain - so spinning up a Spring Boot MVC RESTful webapp with a single controller/endpoint is enough to get me a microservice (even just a tweet would do it..)
So we can create our new microservice to do anything we like, in my case I created a QuoteService (the application is slowly evolving into an insurance engine).  Just having the standalone app isn't helping much, so we need to add some configuration to tell the service to register with our Eureka server - this will make our new microservice discoverable by other services wanting to use it. 

Again this is quite simple: we need to tell our application it should try and register with Eureka, and we should add the config to do so:

You can see that we simply annotate our application config in java, and then add some properties that define where the Eureka server is hosted and that's basically it.

Now if we build the project JAR and start up again (and we still have our Eureka service registry running) then after 30seconds or so you should see the Quote-Service registered and ready to use.


On to the next one.. 

Now, we have a microservice, and we have a registry that makes it discoverable, but still - just one microservice is pretty lonely. So next I created another dummy RESTful microservice, this time called ProductService which just followed the same pattern as the first.

Once that was started up then the Eureka dashboard started looking a bit happier with the two services registered - the obvious next challenge is seamless interaction between the two: splitting the services into their own processes is all good an well, but meaningless if you can't easily integrate them.  The way I look at it is when reading the application code of a service (or application using microservices) then it should just look like a normal application with sevice classes - there shouldn't be any fanfare around the fact that my service class actually gets the data from a dedicated microservice over HTTP/AMQP rather than just getting directly from the DB in the traditional way.


So, still just stubbing out the endpoints, I updated my QuoteService endpoint to make a call to my ProductService, and then just jammed that response into the JSON response I was returning anyway:

As you can see, it could be a standard controller in a normal monolithic application from this point, we are just calling a method on our autowired ProductService class and returning that.


So the really interesting part is in the ProductService class - at the moment this isn't a really elegant, abstracted class yes, so there is still some boiler plate, but that will have the advantage of making it clear what is going on:

As you can see, it's just making a REST call to the Product microservice and returning the response cast as a Map - but the really nice part of this is that the service url is just the service name (in this case "PRODUCT-SERVICE", that is injected to the class)  and with the RestTemplate annotated with Spring-Cloud's @LoadBalanced that microservice will be looked up in Eureka (and load balanced if there is more than one PRODUCT-SERVICE running).

So our setup is starting to take shape now - we have two microservices, both registered with Eureka and able to interact with each other in a fairly clean, loosely coupled way.


Don't push me, 'cos I'm close to the edge..

As your microservices start to proliferate, you will get different levels of service granularity, and undoubtedly you won't just want to expose all your microservices as a public API.  One option would be to create a RESTful application and define nicely named endpoints you want to expose and then use the standard integration described above to integrate it.

Fortunately, there is an easier way - Netflix provides a library called Zuul that can be simply configured to map URL patterns to given defined service names (and again looks up in the Eureka service registry).  Much like Eureka, this is super easy to setup and just needs an annotation and the config again:

And the config is pretty easy to understand:

As you can see, we just define service names against URL patterns.

Now once all are apps are up and running, and the microservices are registered on Eureka then you have a single API interface to start interacting with the services (rather than having to access each service on its designated port etc).


Conclusion

 So that's as far as I have got - I wired up the QuoteService to MongoDB so the data all gets persisted there (and have added a get quote endpoint which gets the same data from mongo) and starting to wire up the product service with JPA.  So far it's been enjoyable, and things are making more sense than when I started - but there are a few questions still:

  • It seems like there is still duplication of service names throughout the different projects - for example the ProductService name ("product-service" - case insensitive) is proliferated throughtout - the service itself defines it, the QuoteService needs to know the name of the service, the Zuul edge server needs to know the name etc.  I guess this is unavoidable as these are intrinsic dependencies but still seems a bit flaky.
  • It feels like the Service classes could be factored out - our ProductService class that allows HTTP REST interactions with the Product microservice would likely need to be re-used across all applications/microservices that need to use the Product microservice

0 comments:

NerdAbility - A presentation

9:08 PM , , , 0 Comments

I recently had to give a presentation about NerdAbility, so for fun, here are the slides I put together. It was just an intro to the product, where the idea came from and what it was trying to solve, followed by a brief jump into one or two interesting elements of the project from a tech point of view.

0 comments:

Innovating towards the traditional - Facebook & Amazon

1:33 PM , , , , , 0 Comments

Just a short post now - and really just two observations/trends I have commented on before that are continuing their trajectory.

 Amazon bring grocery delivery to the UK - Announced today, Amazon will be bringing their "Fresh" service to the UK, starting with London and Leeds - continuing with my previous observations on innovating towards the traditional.

Facebook messenger no longer needs a Facebook account - this is older news, and I have been meaning to comment on this for a while. Basically, customers can now sign up for and start using Messenger without an account, just using a phone number, which is an un-surprising move towards phone as identity and recognising that the phone has the potential to be a node in any social connected network on the platform (much like I have preached about before..).  It is also another step towards Facebook moving further down the stack and more directly competing with native messaging apps as a system level.

0 comments:

Hello Facebook..

10:36 PM , , , , , , , , , , 0 Comments

I have been away on a mini-break for the last few days, and whilst relaxing I was mentally planning out a blog post about mobile phones as identity and the social platform they provide (again!), then, I woke up this morning to Facebook's announcement of their Hello app.  In short the app will attach FB user details to phone numbers, so if you get a call from a mobile number you don't know, but have the app installed on your phone, it will attempt display FB info to identify the caller.




For my money, this is a very smart move.


Moving down the stack

One of the questions for the future mobile landscape is whether players like Facebook and Amazon can move "further down the stack" - that is, to start being more core to devices, or making their own. Amazon have tried with their Kindle Fire phone, and Facebook had Home - both bombed out, however, and there hasn't been much to suggest that there is more promise for them.

However, Hello seems to me like it could be a small step to bridging that gap.  The app is simple in itself, and done well I could see it being very popular - after installing, it doesn't need any active engagement from the user, but (hopefully - without having tried it myself) just sits there and enhances the users phone experience (as long as they don't try and do anything stupid, crazy annoying notifications, battery hogging etc) - As a simple enhancement to the incoming call screen, I can imagine pretty high retention/install rate - because why would you go back to unknown numbers?

One reason this could be a huge win for Facebook is that this step could make Facebook more core to the mobile device (and seemingly appear more and more like just part of the OS).  If you have the app installed, and miss a call, I would expect the missed call notification screen to be Facebook's Hello - with the normal stuff you would expect - so and so called you, return call, send message etc - which will inevitably give FB the hook to drive users to FB messenger, WhatsApp etc - This is a chance for FB to hijack the core functionality of the phone and drive traffic to its other properties, quickly FB messenger could become the default messaging app.



Your phone as your identity & network

This is something that I have ranted about before - there are two powerful and fairly unique traits of mobile phones that have the power to really disrupt the social/technology/online space:

Your phone is your identity - Whilst people continue to struggle to fix online identification/authentication issues, with a host of different providers offering Login-as-a-Service (Twitter, Facebook etc) and lots of people complaining about the proliferation of user logins/passwords that need to be remembered, the phone offers an interesting solution to that - Granted, phone numbers are not for life, but they offer simple, unique identification of individuals, plus, with two-factor authentication, it can be simple to verify that the person attempting to login is in physical possession of the device.  And with more moves to ApplePay and NFC payment with your device, this identity/device becomes even more tightly coupled.  

By Facebook tying your phone number with your profile it is an impressive move to help future proof their platform: so your phone is your identity? Well now your FB profile is intrinsically linked to that too.


Your phone as a social network - For a long time, Facebook have been the dominant force in social networking. Despite Google's effort with Google+, they haven't been able to displace the social network king. Do you think they struggled because they couldn't match Facebook's superior product? You think people stay at Facebook because they just love the features so much? Of course not. They stay because of user inertia - the feature that keeps users at Facebook is the network effect, no one would move to Google+ without the rest of the network, and how to you move your network?  As if to prove this, there are fairly regular outcries from the Facebook masses when FB changes their layout, or people find out about some terrible clause in the privacy policy - but do people leave? Nope. They just get on with it and keep using Facebook.  After all, the users are the real product, not the software.

Now, mobile phones change that a bit.  Traditional online social-networking basically just models the real world network of friends/colleagues/etc - In your real life social network, you are the node in the graph, and your mobile phone basically represents your node - If you think of Facebook as a centralised social network, mobile phones are a distributed social network (in other words, Facebook knows about the entire graph, and keeps that information centralised, but with your phone you know about your immediate neighbours in the graph - e.g. the people you have contact details for, but not the rest of the graph).  This is a pretty powerful idea - and this is what has made it possible for apps like SnapChat and WhatsApp to enjoy sudden and un-precedented user growth where non-mobile only apps (Google+) haven't.

If we add to that the fact that modern smart phones aren't just an address book - they have your photos, they have your videos, they have your messaging, they have your friends - is there much on your Facebook profile that you don't have (the ability) to have locally on your mobile phone?

All of this makes the smart phone and the real world mobile connections a very powerful platform for anyone to come and disrupt Facebook's crown - So this move by Facebook again just helps establish them as leading the mobile-social platform, not just an old internet company trying to compete.



An aside - Google needs to catch up

My original thinking for the blog was going to be about how Google should, in my opinion, be capitalising on the Android platform as a social network, and rather than trying to convince users to sign up to Google+ (or Wave, or Buzz etc) focus their efforts on making Android a more social OS and make more of the Android as your identity.  They would have to do it carefully with regards to stuff like "private by default" so not to cause uproar, but if they could effectively make all their Android users a node in the social network, then all of a sudden they could find themselves with a pretty good social network, and can start thinking about how to add value to the product.  Furthermore, with Android being the OS (as low down the stack as possible) then they would have a lot more options with regards bringing together all the aspects above - Your smart phone knows instantly when you take a photo or record a video (because you are doing it with the device!), it knows your location, it knows when/who you are messaging, it knows who your friends are - some nice usability around that and it starts to sound like a very strong value proposition!

0 comments:

Nerdability: A retrospective

9:47 PM , , , , 0 Comments

In 2011, I came up with the idea to build a web based platform that lets developers and technologists build better CVs.  The idea was that as a technologist, we have a prety big online footprint that represents our skills/interests/experiences much more than a traditional CV that just rattles off academic achievements and work experience.



As a technologist applying for jobs, I wanted to make my CV stand out, and figured I have a StackOverflow profile that could demonstrate my knowledge & communication skills, I had Google-Code/GitHub/BitBucket projects which showed examples of my code, OSS contributions, technologies I have used plus then LinkedIn, blogs, Coursera courses, Geeklist.. the list goes on - and more importantly, these were things that were all changing much faster than my traditional CV was.



On the flipside of that, as an employer trying to recruit technologists is really hard - CV screening doesn't rule out many candidates, and beyond that you are left to navigate technical tests & interviews which are fraught with difficulties in trying to really assess whether someone is actually fir for a job.


Nerdability started life as an idea that was entered in a cloud competition to build a webapp (some details here originally called NerdStar) - which it won.  I then recruited two co-founders and we re-wrote a bunch of it (originally all data persistence was mongo, but switched that to relational) and launched it to the public!


In January this year (2015) we retired the application, we had somewhere in the region of 800 users signed up (with no active/paid marketing).  We have currently parked the page and a basic user profile tour on nerdability.zz.vc and are in the process of moving this holding page back to nerdability.com.



Things that went well

There was a lot of great things we learnt/that came out of the project, here are some highlights:

We contributed to OSS - During development I created a Spring-Social implementation for Khan Academy API and for the GeekList API - both of which are now listed on the Spring site as community projects.

The content driven marketing - As mentioned, we didn't do any marketing, just a few tweets, etc - but the big win was the blog. We setup a basic blog over at blog.nerdability.com (still up and running!) using blogger for hosting and just posted tech articles - tutorial type stuff plus more link-bait-y type stuff like new years resolutions for developers, or getting hackathon projects into production - and the blog drove lots of our traffic.  The beauty was the people who visited/discovered our blog were a perfect match for our target demographic, so we just had plenty of placements for nerdability and the users just trickled in!  The link-bait stuff would give us spikes of traffic, but the tutorial/how-to stuff still drives a few thousand unique views a week.

User response - The user response was good, we had a fair few early adopters who liked what we were doing and blogged/tweeted positively about the project.



Things that were hard

Competition - The weekend we were due to launch, I got an early beta-invite to StackCareers - with an almost identical value proposition, but they already have money and a massive developer community.

 Two-sided marketplace problem - so we started getting users signed up, but there was little engagement as users couldn't really do much with their profiles - we had not companies/jobs listed, so beyond just sharing their profile links there was not much they could do.  We were working on the company integration when we finished, but didn't get into it.



Conclusion

In the end, the three of us weren't at the right place/point in time to go into the project full time (family stuff, wrong career stage etc). However, I still believe there is a good business model in the idea, and still think that even if it was setup today it could be successful.  Despite the community and backing of StackCareers, I don't think they have really nailed it, and seem to have stagnated at a slightly improved jobs board - but not really using the intelligence/information to really improve the recruitment process and take any burden of employers/interviewers (which is where I think one of the big wins is for the model)

0 comments:

The joyless world of a data-less startup

9:34 PM , , , 0 Comments

I recently read "The Joyless World of Data-Driven Startups" over on medium.

It's a nice article, and there are plenty of good points made - but my concern with the article is that it champions data-less decisions without covering the times when data-less or gut-instinct decisions are not appropriate.  In many ways it mirrors some conversations I have had over the past year or two regarding the benefits of data and using the Build-Measure-Learn approach from the Lean methodology.


I actually think the article hits the nail on the head, but really this post is to address the fact that some people may read what they want from it (and use it as an excuse not to measure).  I think really, the main take away point from the article is this:

Data-driven innovation sucks.

That is really what is at the crux of the article, using data to drive innovation is as close to paint-by-numbers as you can get.  Think about it, innovation is by its very nature a creative process - its the act of creating something new, or something that tackles a particular domain or problem in a new and original way - obviously not something that can be done formulaic-ally by numbers.  Creating a new and exciting product has to be creative by nature. Creating a clone of an existing product/service is not (that's not to say that it wouldn't still be a viable business though!).


The problem is, in the past I have ended up in lots of conversations as a proponent of Lean and the B-M-L iterative approach to attempting to better understand the solution and problem domain and have faced this very argument - that its OK that we aren't measuring result correctly(if at all!), and its OK that we are releasing multiple changes, and pulling lots of levers, all at once (thereby making data we do measure difficult to tie back to single changes), because just look at Steve Jobs! He didn't bother about asking what users wanted! he just gave them his vision!


And that is the main distinction I would like to highlight - there is a big difference between using data to drive your creative/innovative decisions and using data to learn from your innovations.  Yes its OK to be creative and out-of-the-box with innovation and new product features - that's how we got so many amazing products that we have today - but you need to be ruthless in your learning and understanding of the data.


To the Steve Jobs example - yes, he was a visionary and he ignored data/user research when designing products - Jobs/Apple are famous for their approach of just giving customers what they decide they want, and with great success - but they still use the data.  You think if the iPhone was a dismal failure they would have ignored that data and not changed it? You think Apple haven't killed off products that haven't performed well?  These are fairly extreme examples, as in both cases the data is very visible (e.g. sales figures) that people might not think of as post-product learning, but really its the same thing.


Essentially, the bottom line is whilst data-driven innovation sucks, data-driven learning is essential.

That's my opinion, anyway.

0 comments:

Spring MVC - Page caching and If-Modified-Since

9:09 PM , , , , , , , , , , , , 1 Comments

Spring (and in general Java/groovy etc) support a range of caching layers for your web application - with options for Hibernate first/second level cache, Spring's @Cacheable and simple integration with lots of providers (EHCache, Redis, etc), but aside from server side caching, you can also implement page level caching using the standard If-Modified-Since cache headers.

If your web application is sitting behind an apache web server, then using this mechanism, you can set it up such that Apache will do a lot of the work, and a lot of requests never even trouble our web application.  This will of course only work if you have fairly static pages that don't update to frequently, and aren't user specific (e.g. so its actually possible to cache at a web-page level - user account pages obviously are harder to cache at this level as they are very user specific).  Even without Apache in the mix, most modern browsers are built to handle If-Modified-Since headers and 304 response codes, so using the approach can mean that browsers are less eager to even request the page from your server whilst a user is browsing the site if we have said its cache-able.


Thankfully, the machinery for this stuff is all baked into Spring MVC.


Updating the RequestMappingHandlerAdapter

This walkthrough is assuming that your app is using standard @Controller annotations and a standard RequestMappingHandlerAdapter to route the requests - although there are still relatively easy mechanisms to do this if you are using other Controller/HandlerAdapter patterns.

The RequestMappingHandlerAdapter class has a method that can be overridden called getLastModifiedInternal() - This method simply returns a long value (the epoch time) that the requested resource was last modified. All we need to do is extend the RequestMappingHandlerAdapter and implement this method to return a timestamp.  For example:



The above assumes we have initialised a timestamp at startup (easy to do using Java config) and assumes that if you have visited the site since last app startup, then there is no change (in reality, we will likely need something more complicated to calculate this)

And that's really all we need, as Spring will handle the rest for us - from this, if a brand new request comes in with no If-Modified-Since headers, then Spring will take this date/time stamp and return it with our response as the Last-Modified date (HTTP standard header - this will inform the browsers/web servers action next time).  If however, a user has already visited your site and the browser has received a valid Last-Modified timestamp, then on the next request it will include this value in the request If-Modified-Since header, when Spring recieves this request, it compares the timestamp against the value that is returned from our getLastModifiedInternal() method, and if there has been no change then Spring will automatically return the response to the client with a 304 response - so none of the Controller code will be executed.

As you can probably guess, this can provide huge efficiencies and improvement on latency, server overhead etc.



More complex Last Modified Dates

As mentioned, in all likelyhood you will need a more sophisticated mechanism than just checking the application start time - So we have plenty of options here: we could query the DB for changes, we could have other flags/properties set for when particular resources are set (bare in mind that if static resources like CSS are changed, these need to be considered)

If you need to consider a fairly unique LastModified date for every endpoint, then this solution isn't a good fit, as this is the more generic approach - you can alternatively implement a similar last modified method on your (every) controllers.


Another option that I have considered is a halfway compromise - where I need endpoints to have specific considerations, but I only have a set of 4-5 (or relatively manageable) specific queries/checks that need to happen, and every endpoint will just need to check some subset of these conditions - this solution involves custom annotations and marking up Controller methods with this annotation to signal to our HandlerAdapter what checks should be considered for the Last Modified timestamp.  This has the advantage of relatively little intrusion in all my controller endpoints, but granular enough to provide enough control for effective page caching.


A custom annotation

First we define a simple annotation that can be used to mark up Controller methods to indicate which conditions are important to the endpoint:



The above code shows the annotation code, a sample enum (just to provide some type safety whilst using the annotation - this could be a string or anything else) and an example of the annotation on a controller endpoint.


Updating our Last Modified method

Now, in our last modified method, we have access to the Handler (e.g. the controller method being invoked) so we can quite simply check the controller method for our annotation, and if found we can just check the values said and perform the appropriate checks to work out what the last modified date needs to consider:




As mentioned, this won't work if there are lots of pages with lots of different requirements, but if you have a manageable of changing entities in your application this can strike a nice balance between clean code and flexibility.



1 comments:

Spring Boot - Tomcat error handling

9:09 PM , , , , , 0 Comments

The default (maybe even recommended?) behaviour for Spring Boot is to package your web application as a JAR with a bundled Tomcat - following the increasingly popular pattern of "Application Servers are Dead" - a more extreme version of one-app-per-app-server pattern I guess.  However, by and large I am still using Tomcat instances (although usually still one-app-per-server) so I am building WAR files for deployment.


So error handling.  Normally, in our web.xml we would define errorPages for response codes, exception types etc, that Tomcat would be able to forward any response code/exception that comes from your application back into an end-point within your application so you can display a custom error page and handle any additional logging etc.


Now for Spring Boot (and Spring 4 generally) there is a shift away from XML and a lot of effort has been put in to make it possible to create web apps with out a single XML file in sight.  This includes the error page mappings - so if you want to define custom mappings you can simply create an error configuration file as so:




Simple right? Now that all makes sense if we are running tomcat bundled with our app - we can use this configuration to configure Tomcat and everything would work as expected - but what about if we are deploying a built WAR file to a Tomcat instance? How can Spring be configuring Tomcat's error page mapping?  To me it just seemed too magical, but as you might expect, there is a simple answer:  ErrorPageFilter - This is just a default Filter that is added that intercepts all requests with error codes/exceptions and forwards it to the correct endpoint.  From the comment at the top of the Filter:



 * A special {@link AbstractConfigurableEmbeddedServletContainer} for non-embedded

 * applications (i.e. deployed WAR files). It registers error pages and handles

 * application errors by filtering requests and forwarding to the error pages instead of

 * letting the container handle them. Error pages are a feature of the servlet spec but

 * there is no Java API for registering them in the spec. This filter works around that by

 * accepting error page registrations from Spring Boot's

 * {@link EmbeddedServletContainerCustomizer} (any beans of that type in the context will

 * be applied to this container).


A simple solution that will work for standalone JARs or deployed WARs.

0 comments:

Thoughts on Spring Boot

8:40 PM , , , , 0 Comments

 Having long been a fan of the Spring framework (Spring MVC, Spring Social etc), I have recently started using Spring Boot for two different projects I am currently working on.
In both cases, they are web applications - both with some key differences in technology (in terms of what I am using for client side libraries and data persistence).

Spring Boot is an opinionated implementation of Spring - and it works really nicely (most of the time).  You simply add the Spring Boot dependencies to your build file (Gradle or Maven both well supported) and you can have an application up and running with a very small amount of code (you may have seen the Spring boot application in a Tweet a while ago)



Now of course, in reality you do need other configuration stuff, but the take away point is that if you are happy with Spring opinions, you can get an application up and running in pretty quick time.  For me, this is really a turning point for Java/Spring productivity - especially as Groovy has become more mature and sits so easily in Spring Boot, I don't think there is much to the argument that Java/Spring is too slow for early stage companies/prototypes/rapid development process (Aside: I know grails has been around for a while, but I think Spring and Java have stepped up here, plus I honestly have my suspicions that Spring Boot and Grails may clash at some point - they have certainly been on a collision course since Spring Boot was announced at Spring One).


The way it works is quite simple really, you add a relevant dependency to your buildfile, for example:

compile("org.springframework.boot:spring-boot-starter-web")  
compile("org.springframework.boot:spring-boot-starter-thymeleaf")


The first one is just your standard web application Spring Boot dependency, and the second is the opinionated version of Thymeleaf configuration.  Then, at startup Spring Boot has lots of AutoConfiguration files that check for the existence of key classes in the classpath, and if present it executes the auto configuration.

See here for the Thymeleaf autoconfiguration - you will see that there is heavy use of the @ConditionalOnClass annotation - which checks for relevant Thymeleaf classes, and if they are on the classpath it configures them.



Undoubtedly, there are a few times where you find yourself scratching your head at the magic, and I have on more than one occasion had to go through the Spring source code.  And sometimes, you want most of the auto configuration, but just want to tweak one or two properties, and you are left having to turn off the autoconfig and do it yourself, but for me the biggest take away point is the speed at which it is now possible to get up and running with Spring/Java/Groovy and have a decent web platform for building your product or company.

0 comments:

Personalised education & tech

8:05 PM , , , 0 Comments

As I have written about before, I am pretty interested in education.  There seem to be some pretty fundamental flaws in the modern education system, but it still seems unclear to me what exactly a good solution looks like.

The problem with modern education seems to be the whole standardised testing thing, and forcing all children down the standard curriculum path rather than recognising that intelligence and ability takes different forms etc.


In Seth Godin's TED talk, he explains these problems and describes the idea of having personalized education experiences, making a comparison to car manufacture:

Precise, focused education instead of mass batch stuff. That's the way we make almost everything we buy now, right?   It used to be you could have any colour of car you wanted as long as it was black - so we could keep the assembly line going. But now, we make ten thousand kinds of cars, because they can! So we should make ten thousand kind of education.


And that's absolutely right. In both car manufacturing and education, the tools and technology have moved on - we have the technology to cost effectively make loads of car, just like we have the technology, with things like Coursera, Khan Academy, iTunesU etc, to deliver lessons and lectures on pretty much any topic, covered by some of the experts in the field.


He also repeats some of the ideas that Sal Khan talked about, such as "homework at day, lessons at night" - the idea that watching lectures/reading etc should be done in the evenings independently, and then the daytime, classroom based stuff, where there are teachers and real human lead interaction focuses on solving problems, exploring ideas etc.  This idea is great for two reasons: 1) teachers engaging with students on problems, peers discussing ideas or concepts, interactive learning - this seems much more likely to get students enjoying learning new things (which is really a much better bi-product of the education system than just "knowing things")  2) Being able to study independently, reading and watching lectures is much closer to the real life world of work - one of the benefits that is sometimes preached from the home-school camp - having to learn independently makes it much easier to fit into the modern workforce.


Further more, standardised testing creates an environment of graded achievement - parents want to know what their children have achieved, and how they compare to other students or what they have achieved this week - which only serves to put more pressure on teachers to try and teach-to-test, and makes it hard to spend longer period on in-depth study and exploration over an extended period.

0 comments:

2015. Plans & resolutions

9:07 PM , 0 Comments

As it is a new year again, I have been toying with the idea of some sort of resolutions, but it's hard to come up with anything really meaningful and achievable.

I thought about (and am still thinking about) creating something every month - which has the nice fallback that even during a hectic month, I should have a chance to at least cook something, so even if it ends up being 12 months of just creating some meals, at least its something to aim for.  There are lots of projects that I have either started or keep meaning to get underway which would be good contenders for the monthly fun.

One of the stretch-goals, kind of in hand with the monthly objectives is towards the end of the year it would be nice to setup another company - doing what is still undecided!  Hopefully I suppose the plan is that this will be the result of one of the monthly projects, but let's see.


Possible things on the list..
  • Made of these - a family/private photo app to capture, annotate and privately share photos of your family
  • My lego Pi robot - a year later, I still haven't had a chance to sit down and concentrate on getting even my lego dc motor working with my Pi
  • CREAM - personal finance management app - previously mentioned before
  • a whole bunch of food ideas
  • unknown ideas - still interested in the enterprise, education or public services space


More generally, the aim is to be more pro-active and productive.. I have signed up for the Stanford Machine Learning course on Coursera, that has just started last week - but I'm not sure I will have the time to complete that on schedule (a good start to the year, right?!) so will maybe complete that over a longer time period - I completed the Stanford Algorithms course last year in my own time, and that was pretty good.


Anyway, it's approaching the end of January 2015, so what have I done:

Cooked a bunch, and started a food blog



So yeah. I cooked a bit, made up some recipes, and finally setup a food blog. I am really enjoying that so far, as the blog is helping motivate me to be creative in cooking.


Created/Updated a friends site


An aspiring singer friend of mine recently appeared on the BBC's The Voice (TV show in the UK) and was keen for his site to be updated.  It's just a basic Twitter Bootstrap template, but it was fun playing with js settings and general layout to get the stuff online. Check it out here.

0 comments:

Android: Building a cloud based quiz application

10:13 PM , , , , , , , , , , , , , , , , 4 Comments

A long time ago, when Android was still in its infancy (1.5 I think..) I built and open sourced a basic quiz app.  The app was just a basic multiple choice question-answer app, driven from some questions in the database, but it did ok - it has had over 10k downloads on the app store, and the blog post tutorial here is still one of the most popular articles.

But here we are, Android 5.0 is released and the state of Android development is very different now. But I didn't really want to just re-skin/tweak the old app and push it out again - and I also wanted to write up some notes on using parse.com as a backend service - so this seemed like a good opportunity.

The source code for the app is all on GitHub.


The goal

So the aim is to create a an android app quiz game, but rather than using local storage, using the cloud to power the questions. This avoids the need for lots of boiler plate DB code and also makes it easier for us to update questions.  The tutorial will be broken into two parts - the first part will be basic quiz stuff with cloud questions, the second part will enhance the app to support user login and to track scores to allow users to compete against each other.


Parse

Before you start the tutorial, you need to get an account setup at parse.com - its a cloud DB/backend as a service that was recently bought by Facebook. They allow real easy setup of DBs and provide a load of nice libraries/APIs to really easily interact with their endpoints across lots of platforms (its also incredibly well priced -the free tiew is really good and if you find your mobile app is going beyond that then you can probably think about monetising the app!).  All you need to do is head over there, sign-up and then create a new app - all you need to do is give it a name and hey presto!  You can either make a note of the keys then, or come back and grab them later. There are no other changes you need to make now, as that will get handled from our source code. The only other thing to do is to download the parse android library to make use of their sdk in android (if you have grabbed the source code from GitHub then you will not have to worry about these)



OK, lets get started!

Again, I am not going to spend a lot of time covering the real basics of Android and only really mention the things of note or that are different from standard application development - hopefully the code & general explanation will be clear enough to get an understanding of what is going on.


Android manifest
First, lets get our AndroidManifest.xml file configured.  The only things to note here are the permissions we are setting - we will request internet access and network state permissions. Also worth noting that I have set the min sdk for my sample app at version 16.


Our Application class
We will have to create a custom implementation of the Android Application class. This class is instantiated on application startup, and hopefully if you are looking at Android development you should be familiar with this class.  We will do a couple of things in this class:

  1. Register our parse.com application with out secret keys
  2. Initialise the Parse library and our domain objects
  3. Try to fetch all the questions for the quiz and store them for offline usage 
  4. Create a GamePlay object, that will keep track of the state of the current game in progress
First lets look at the Parse setup - this is standard parse boilerplate and is covered in parse docs and sample apps - you just need to add your ID/Key here (also note, we have just registered the Parse object class Question - this is our domain object - like a JPA entity etc - if we add more domain objects they need to be added here too)

Next we will make a call to parse.com to fetch the questions from our cloud API - we will save this in the background (make an asynchronous call) and "pin it" to make it available for offline usage. Also note that we do not un-pin existing questions until we have successfully found new ones - that way users should always have questions once they have successfully loaded them the first time.
Hopefully the above is quite clear - the parse libraries are quite straight forward to understand - we create a query (typed Question) and then call findInBackground and implement an on success handler.


Domain objects: Question
Parse library provides a nice interface to create POJOs to model your domain model, if you are familiar with JPA/Hibernate/etc and the approach of POJOs representing a domain model its much like this. Using these classes you can easily query/load/save data from the cloud by just using these objects. You will have spotted that in the query we use in the application class to load all our questions we just run a plain query with the Question class - this, as you should expect, will just retrieve all Question objects from your cloud table (parse). The domain models are just an annotated POJO, and you just define appropriate getter/setters for the fields you want to include.


Welcome activity
Now we are into Android basic stuff really - we have setup parse and fetched the questions for local usage, now we just need a basic menu to start the quiz and some activities to display the questions and the end results.

We will just apply the layout and then implement a method to handle the button clicks. For this tutorial we are skipping the high score stuff and just playing the quiz.
All we need to do is reset the current GamePlay object and grab the questions from the local store (by this point they should be updated from the cloud so no problems, then kick off the quiz!


Question activity
There is nothing really interesting to see here - it's all on github if you want to take a look in detail (or have already downloaded and working along) - This is just a standard Android activity that pulls out the question and possible answers and presents them.


This just progresses along fairly simply, until it gets to the end of the quiz and then it presents a simple screen saying the score - all this stuff can be tweaked/styled etc - but there are the basics for a cloud powered multiple choice quiz app!


Creating the questions

All the questions will be stored in the cloud in parse.com - once you have a look at the interface, it should be pretty clear - you can easily create the data for questions either manually or by importing a csv/json file.



You will need to login to your parse account and the quiz app and create a Question class. This will just match the domain POJO we have created. Just login, go to "CORE" then "DATA" Then select "+ Add Class", adda a custom class called "Question" (this must be exactly the same as the name provided in the POJO annotation for the Question class).. Then select "Add Col" and add the fields to match the POJO (question[string], option1[string], etc).  Once you have the class/table added on parse, you can simply add data by selecting "Add Row" and just manually entering the data, or using the import function.



Source code

4 comments:

Tech: Functional programming in Groovy

6:33 PM , , , , , , , 0 Comments

I have recently started the Coursera Functional Programming with Scala course (taught by Martin Odersky - the creator of Scala) - which is actually serving as an introduction to both FP and Scala at the same time having done neither before. The course itself is great, however, trying to watch the videos and take in both the new Scala syntax and the FP concepts at the same time can take a bit of effort.

I wanted to work through some of the core FP concepts in a more familiar context, so am going to apply some of the lessons/principles/exercises in Groovy.


Functions:

If you have done any Groovy programming then you will have come across Groovy functions/closures. As Groovy is dynamically typed (compared to Scala's static typing), you can play it fairly fast and loose.

For example, if we take the square root function that is demonstrated in the Scala course, it is defined as follows:



As you can see, Scala expects the values to be typed (aside, you don't actually always need to provide a return type in Scala). But in the Groovy function it is:



A groovy function can be defined and assigned to any variable, thereby allowing it to be passed around as a first class object. If we look at the complete solution for calculating the square root of a number (using Newton's method - To compute the square root of "x" we start with an estimate of the square root, "y" and continue to improve the the guess by taking the mean of x and x/y )


So that's in Scala, if we try Groovy we will see we can achieve pretty much the same thing easily:


Recursion (and tail-recursion):

As FP avoids having mutable state, the most common approach to solve problems is to break the problem down in to simple functions and call them recursively - This avoids having to maintain state whilst iterating through loops, and each function call is given its input and produces an output.

If we again consider an example from the Scala course, with the example of a simple function that calculates the factorial for a given number.

This simple function recursively calculates the factorial, continuing to call itself until all numbers to zero have been considered. As you can see, there is no mutable state - every call to the factorial function simply takes the input and returns an output (the value of n is never changed or re-assigned, n is simply used to calculate output values)

There is a problem here, and that is as soon as you attempt to calculate the factorial of a significantly large enough number you will encounter a StackOverflow exception - this is because in the JVM every time a function is called, a frame is added to the stack, so working recursively its pretty easy to hit upon the limit of the stack and encounter this problem.  The common way to solve this is by using Tail-Call recursion. This trick is simply to have the last code that is evaluated in the function to be the recursive call - normally in FP languages the compiler/interpreter will recognise this pattern and under the hood, it will really just run the code as a loop (e.g. if we know the very last piece of code in the block of code is calling itself, its really not that different to just having the block of code/function inside a loop construct)

In the previous factorial example, it might look like the last code to be executed is the recursive call factorial(n-1) - however, the value of that call is actually returned to the function and THEN multiplied by n - so actually the last piece of code to be evaluated in the function call is actually n * return value of factorial(n-1).
Let's have a look at re-writing the function so it is tail-recursive.


Now, using an accumulator, the last code to be evaluated in the function is our recursive function call. In most FP languages, including Scala, this is enough - however, the JVM doesn't automatically support tail-call recursion, so you actually need to use a rather clunkier approach in Groovy:

The use of the trampoline() method means that the function will now be called using tail-call recursion, so there should never be a StackOverflow exception. It's not as nice as in Scala or other languages, but the support is there so we can continue.


Currying:

This is like function composition - the idea being you take a generic function, and then you curry it with some value to make a more specific application of the function. For example, if we look at a function that given values x and y, it returns z which is the value x percent of y (e.g. given x=10, y=100, it returns the 10 percent of 100,  z=10)

The above simple function is a generic mechanism to get a percentage value of another, but if we consider that we wanted a common application of this function was to always calculate 10% of a given value - rather than write a slightly modified version of the function we can simply curry the function as follows:

Now, if the function tenPercent(x) is called, it uses the original percentage() function, but curries the value 10 as the first argument. (If you need to curry other argument positions you can also use the rcurry() function to curry the right most argument, or ncurry() which also takes an argument position - check the Groovy docs on currying for more info)


Immutability:

Immutability is partially supported in Java normally with use of the final keyword (meaning variables can't be changed after being initially set on object instantiation). Groovy also provides a quick and easy @Immutable annotation that can be added to a class to easily make it immutable.  But really, there is more to avoiding immutable state than just having classes as immutable - As we have functions as first class objects, we can easily assign variables and mutate them within a function - so this is more of a mindset or philosophy that you have to get used to. For example:

The first example is probably more like the Groovy/Java code we are used to writing, but that is mutating the state of the list - where as the second approach leaves the original list unchanged.


Map Reduce:

As a final note, there are some functions in FP that are pretty common techniques - the most famous of which these days (in part thanks to Google) is Map-Reduce, but the trio of functions are actually Map, Reduce(also known as Fold) & Filter - you can read more about the functions here (or just google them!), but these functions actually correlate pretty nicely to core Groovy functions that you probably use a lot of (assuming you are groovy programmers).

Map

map is the easiest to understand of the three. It takes in two inputs - a function, and a list. It then applies this function to every element in the list. You can basically do the same thing with a list comprehension however. 
Sound familiar? This is basically the .collect{} function in Groovy

Reduce/Fold

This one is a bit more complicated to descibe, but is the same as the .inject{} function in groovy

Filter

And another simple one - filtering out a list for desired elements, this is Groovy's .findAll{} function



As I said at the start, I am new to FP and coming from an OO background, but hopefully the above isn't too far from the truth!  As I get further through the Coursera course I will try to post again, maybe with some of the assignments attempted in Groovy to see how it really stands up.


Some useful references:

0 comments:

Tech: Building an RSS reader for Android (RIP Google Reader)

6:31 PM , , , , , , , , , , 0 Comments

This tutorial will walk through building an RSS reader on the Android platform (focusing on 3.0 + as it will be using Fragments). All the code is available as a complete, working Android app that you can fork/download and fire up straight away on a compatible Android device/emulator. So feel free to go grab that from GitHub before continuing.

It is not an unusual requirement for mobile apps these days to be able to consume an RSS feed from another site (or many) to aggregate the content -  Or maybe you just want to build your own Android app now that Google has announced it will be retiring Reader.

Those of you who have worked with RSS in other JVM languages will know that there are plenty of libraries available that can handle parsing RSS - however, because the android platform doesn't actually contain all the core java classes, almost all of the RSS libraries have not been supported.

Fear not though, as Java's SAX parser is available, so with a bit of code we can get a custom RSS parser up and running in no time!

This walk through will cover off the basics of getting an RSS reader app up and running quickly and will also cover some details of Android's fragment system for tablet optimization as well as some things to watch out for (such as changes in the platform that mean network operations cannot be run in the main thread, which requires some tweaking if you have worked on earlier versions).

All the code for the app is also available on our GitHub so feel free to fork that and try loading it up on your Android device/emulator.

Parsing an RSS Feed:


So to get started we will look at parsing the feed - if you have any experience parsing XML using SAX in Java then you will know how easy this is. All we need to do is to tell the parser which XML nodes we care about, and what to do with them.

If  you have never implemented a SAX parser before, there are three primary methods that we will override: 
  • startElement() - This is called by the parser every time a new XML node is found
  • endElement() - This is called by the parser every time an XML node is closed (e.g. </.. )
  • chars() - this is called when characters are found between nodes



Because we only really care about capturing data from the leaf nodes, our startElement() method is left empty. The chars() element has to be watched, as there is no guarantee when it will be called (e.g. in a node like hello world  this method might be called several times between the start and end) so every time we will just append the contents to a StringBuffer - that way we can be sure that we will have captured all the data in the node.  By the time the endElement() method is called, we know that we have the contents of the node itself, and we just have to store the data.  At this point, we just quickly knocked up a POJO with the attributes that we wanted to capture - the Strings that we match on are the node names from the ATOM RSS feed (that Blogger uses) - if you are using another feed, just have a quick look at the feed and update the node names appropriately.

Using our Feed in an Android App

So, that was easy right? Once that parser has run through (and you could use that code standalone in any java app really) then you will have a list of Java objects that have the core details about the latest blog posts on the feed (title, author, datecreated, content, etc) - So now lets look at using it in an Android app.

We will assume a basic understanding of the Android SDK and the concept of Fragments, so won't go back to basics with that stuff.

What we will do, is create a basic ListFragment and an RSSService class that we will use to populate the list. In our ListFragment we will simply tell our RSS service to populate the list:



Simple right?

Let's take a look at what our helpful RSS service is doing for us.



The first thing to note is that this class is extending Android's AsyncTask- The reason for this is that since Android 3.0, you are no longer able to perform network operations in the main application thread, so being as our class is going to have to fetch some RSS feeds we are going to have to spin off a new thread.

As you can see, the constructor just sets some context info that we will use later on, and then builds a progress dialogue - this is then displayed in the onPreExecute() method - This lets us show a "loading" spinning disk whilst we fetch the data.

Android's AsyncTask's primary method that handles the actual work that you want to do asynchronously is called "doInBackground()" - In our case, this is simple - we just invoke our SAX RSS parser and fetch our feed data:



Finally, we will override the "onPostExecute()" method of the async class to use our newly fetched list to populate our ListFragment.  You note how when we overrode the doInBackground() method earlier we set the return to List of Articles (where Article is my simple POJO containing my RSS blog post info) - well this must correspond to the argument of the "onPostExecute()" method, which looks like this:



Actually, all we really needed to do in this method would be pass our new List or articles to the ListFragment and notify it of the change as below:



However, in our application we have added a bit more sugar on the app - and we have actually backed the app with a simple DB that records the unique IDs of the posts and tracks whether or not they have been read to provide a nicer highlighting of listed blog posts.

So that's it - there's plenty more you can add to your RSS reader app, such as storing posts for reading later, and supporting multiple feeds/feed types - but feel free to fork the code on GitHub, or just download on to your android device to enjoy all our NerdAbility updates!

Application running in Andriod emulator
RSS application running in an Andriod  tablet emulator




0 comments:

Spring security & subdomains

10:41 AM , , , , 0 Comments

As previously mentioned, I have been working with a Spring MVC app that has had to deal with multiple subdomains for the one app (in other words, the subdomain really needs to just be considered as part of the normal URL path in all routing/security configuration and concerns).

Having gone through the details on how to make the @Controller and @RequestMapping routing to play nicely with subdomains, here is a quick overview of how to handle subdomains in Spring security.


A custom matcher

The main thing we really need to handle with security, is how to configure Spring-security so we can define permissions for URLs that include the subdomain.

Normally, Spring MVC permissions looks something like this:

As you can see, this just specifies a URL path to authenticate.


The specific details of how you implement the matcher exactly will be dependent on your applications approach to identifying and extracting the subdomain (maybe from http request, maybe just use a regex on the request etc)

As you can see above, the matcher we have created is just a convenient wrapper around another two spring matchers to let you match easily on both the full URL and subdomain.

Now, with a little convenience method, we can make some pretty nice Spring security configuration:

As you can see, the subdomain makes a difference to the permissions and who should access the two /dashboard/ URLs in the different contexts, but with the above simple code, we can make some pretty convenient & readable configuration to take subdomains into account.

0 comments:

Subscribe to: Posts (Atom)