Thoughts on development & design: open source

Showing posts with label open source. Show all posts

Akka-Http is my preferred framework for building APIs, but there are some things I have picked up along the way. For one thing, Akka-Http is very un-opinionated in its approach, there are often lots of ways to do the same thing, and there isn't a lot of opinionated guidance about how to do things.

I have been writing Akka-Http APIs for I guess about 18 months now (not long, I know), having previously worked predominantly with libraries like Spring, and I have seen some pretty nasty code resulting from the this (by this I mean, I have written nasty code - not intentionally, of course, but from good intentions starting off trying to write, clean, idiomatic Akka-Http code, and ending up in huge sprawling routing classes which are un-readable and generally not very nice).

The routing DSL is Akka-Http is pretty nice, but can quickly become unwieldy. For example, let's imagine you start off with something like this:

This looks nice right? A simple nested approach to the routing structure that reflects the URL hierarchy and the HTTP method etc. However, as you can probably imagine, try and scale this up to a full application it can very easily become fairly messy. The nested directives make it nice to group routes under similar root URLs but as you do that you end up with very long, arrow-shaped code that actually isn’t that easy to follow - if you have several endpoints nested within the structure it actually becomes quite hard to work out what endpoints there are and what is handling what.

Another problem that needs to be managed is that with the first one or two endpoints you might put the handling code directly in the routing structure, which is ok for very small numbers, but it needs to be managed sensibly as the endpoints grow and your routing structure starts to look more and more sprawling.

It is of course personal preference, but even with the simple example above, I don’t like the level of nesting that already exists there to simply define the mapping of the GET HTTP method and a given URL - and if you add more endpoints and start to break down the URL with additional directives per URL section then the nesting increases.

To simplify the code, and keep it clean from the start I go for the following approach:

Make sure your Routing classes are sensibly separated - probably by the URL root (e.g. have a single UserRoutes class that handles all URLs under /users) to avoid them growing too much
Hand off all business logic (well, within reason) to a service class - I use Scala’s Self-Type notation to handle this and keep it nicely de-coupled
Use custom directives & non-nested routings to make the DSL more concise

Most of these steps are simple and self explanatory, so its probably just step 3 that needs some more explanation. To start with, here is a simple example:

You can see points 1 and 2 simply enough, but you will also notice that my endpoints are simple functions, without multiple levels of nesting (we may need some additional nesting at some point, as some endpoints will likely need other akka-http directives, but we can strive to keep it minimal).

You might notice I have duplicated the URL section “users” rather than nesting it - some people might not like this duplication (and I guess risk of error/divergence of URLs - but that can be mitigated with having predefined constants instead of explicit strings), but I prefer the readability and simplicity of this over extensive nesting.

Custom Directives

First off, I have simply combined a couple of existing directives to make it more concise. Normally, you might have several levels of nested directives such as one or more pathPrefix(“path”) sections, the HTTP Method such as get{} another one to match pathEndOrSingleslash{} - To avoid this I have concatenated some of these to convenient single points.

getPath, postPath, putPath, etc simply combine the HTTP method with the URL path-matcher, and also includes the existing Akka-Http directive “redirectToTrailingSlashIfMissing” which avoids having to specify matching on either a slash or path end, and instead allows you to always match exact paths - It basically squashes the three directives in the original HelloWorld example above down to one simple, readable directive.

Custom Serialisation

You may also notice, I have implemented a custom method called “respond” - I use this to handle the serialisation of the response to a common JSON shape and to handle errors. Using this approach, I define a custom Response wrapper type that is essentially an Either of our internal custom error type and a valid response type T (implementation details below) - this means in all our code we have a consistent type that can be used to handle errors and ensure consistent responses.

This respond method simply expects a Response type to be passed to it (along with an optional success status code - defaulting to 200 OK, but can be provided to support alternative success codes). The method then uses Circe and Shapeless to convert the Response to a common JSON object.

Let’s have a look at some of the details, first the custom types I have defined for errors and custom Response type:

Simple, now let’s take a look at the implementation of the respond method:

It might look daunting (or not, depending on your familiarity with Scala and Shapeless), but its relatively simple. The two implicit Encoder arguments that are included on the method signature simply ensure that whatever type A is in the provided Response[A], Circe & shapeless are able to serialise it. If you try to pass some response to this method that can’t be serialised you get a compile error. After that, all it does is wraps the response A in a common message and returns that along with an appropriate (or provided) HTTP status code.

You might also notice the final result is built using the wrap method in the ResponseWrapperEncoder trait - this allows easy extension/overriding of what the common response message looks like.

Conclusion

All of this machinery is of course abstracted away to a common library that can be used across different projects, and so, in reality it means we have a consistent, clean API with simple routing classes as simple and neat as below, whilst also handing off our business logic to neater, testable services.

All the code for my opinionated library and an example API is all on GitHub, and it is currently in progress with more ideas underway!

In my last post about evolutionary computing, I mentioned I started building the project partly just so I could have a play with Shapeless. Shapeless is a generic programming library for Scala, which is growing in popularity - but can be fairly complicated to get started with.

I had used Shapeless from a distance up until this point - using libraries like Circe or Spray-Json-Shapeless that use Shapeless under the hood to do stuff like JSON de/serialisation without boilerplate overhead of having to define all the different case classes/sealed traits that we want to serialise.

You can read the previous post for more details (and the code) if you want to understand exactly what the goal was, but to simplify, the first thing I needed to achieve was for a given case class, generate a brand new instance.

Now I wanted the client of the library to be able to pass in any case class they so wished, so I needed to be able to generically inspect all attributes of any given case class* and then decide how to generate a new instance for it.

Thinking about this problem in traditional JVM terms, you may be thinking you could use reflection - which is certainly one option, although some people still have a strong dislike for reflection in the JVM, and also has the downside that its not type safe - that is, if someone passes in some attribute that you don't want to support (a DateTime, UUID, etc) then it would have to be handled at runtime (which also means more exception handling code, as well as loosing the compile time safety). Likewise, you could just ask clients of the code to pass in a Map of the attributes, which dodges the need for reflection but still has no type safety either.

Enter Shapeless

Shapeless allows us to represent case classes and sealed traits in a generic structure, that can be easily inspected, transformed and put into new classes.

A case class, product and HList walk into a bar

A case class can be thought of as a product (in the functional programming, algebraic data type sense), that is, case class Person(name: String, age: Int, living: Boolean) is the product of name AND age AND living - that is, every time you have an instance of that case class you will always have an instance of name AND age AND living. Great. So, as well as being a product, the case class in this example also carries semantic meaning in the type itself - that is, for this instance as well as having these three attributes we also know an additional piece of information that this is a Person. This is of course super helpful, and kinda central to the idea of a type system! But maybe sometimes, we want to be able to just generically (but still type safe!) say I have some code that doesn't care about the semantics of what something is, but if it has three attributes of String, Int, Boolean, then I can handle it - and that's what Shapeless provides - It provides a structure called a HList (a Heterogeneous List) that allows you to define a type as a list of different types, but in a compile time checked way.

Rather that a standard List in Scala (or Java) where by we define the type that the list will contain, with a HList, we can define the list as specific types, for example:
The above example allows us to define out HList with specific types, which provides our compile time safety - the :: notation is some syntactical sugar provided by Shapeless to mirror normal Scala list behaviour, and HNil is the type for an empty HList.

Case class to HList and back again

Ok, cool - we have a way to explicitly define very specific heterogeneous lists of types - how can this help us? We might not want to be explicitly defining these lists everywhere. Once again Shapeless steps in and provides a type class called Generic.

The Generic type class allows conversion of a case class to a HList, and back again, for example:
Through compile time macros, Shapeless provides us these Generic type classes for any case class. We can then use these Generic classes to convert to and from case classes and HLists:

Back to the problem at hand

So, we now have the means to convert any given case class to a HList (that can be inspected, traversed etc) - which is great, as this means we can allow our client to pass in any case class and we just need to be able to handle the generation of/from a HList.

To start with, we will define our type class for generation:

Now, in the companion object we will define a helper method, that can be called simply without needing anything passed into scope, and the necessary Generator instance will be provided implicitly (and if there aren't any appropriate implicits, then we get our compile time error, which is our desired behaviour)

As you can see, we can just call the helper method with an appropriate type argument and as long as there is an implicit Generator in scope for that type, then it will invoke that generate method and *hopefully* generate a new instance of that type.

Next, we need to define some specific generators, for now, I will just add the implicits to support some basic types:
All simple so far - if we call Generator.generate[Int] then it gets the implicit Generator[Int] and generates a random int (not that useful, perhaps, but it works!)

Now comes the part where our generate method can handle a case class being passed in, and subsequently the Shapeless HList representations of a case class (we will recursively traverse a HList structure using appropriate implict generators). First up, lets look at how we can implicitly handle a case class being passed in - at first this may start to look daunting, but its really not that bad.. honestly:
So, what is going on there? The implicit is bound by two types: [T, L <: HList] - which is then used in the implicit argument passed into the method implicit generic: Generic.Aux[T, L] and argument lGen: Generator[L] - this matches for all types T that there exists in scope a Shapeless Generic instance that can convert it into some HList L, and for which L an implicit Generator exists. Now we know from our earlier look at Shapeless that it will provide a Generic instance for any case class (and case classes will be converted into a HList), so this implicit will be used in the scenario where by we pass in a case class instance, as long as we have an implicit Generator in scope that can handle a HList: So lets add that next!

Let's start with the base-case, as we saw earlier, HNil is the type for an empty HList so lets start with that one:
Remember, the goal is from a given case class (and therefore a given HList) is to generate a new one, so for an empty HList, we just want to return the same.

Next we need to handle a generator for a non-empty HList:
So this case is also relatively simple, because our HList is essentially like a linked list, we will handle it recursively - and the type parameters [H, T <: HList] represent the type of the head of the HList (probably a simple type - in which case it will try to resolve the implicit using our original Int/Boolean/Double generators) and the tail of the HList which will be another HList (possibly a HNil if we are at the end of the list, otherwise this will resolve recursively to this generator again). We grab the implicit Generator for the head and tail then simply call generate on both and that's really all there is to it! We have defined the implicit for the top level case class (which converts it to the generic HList and then calls generate on that), we have the implicit to recursively traverse the HList structure and we have finally the implicits to handle the simple types that might be in our case class.

Now we can simply define a case class, pass it to our helper method and it will generate a brand new instance for us:

I was using this approach to generate candidates for my evolutionary algorithm, however the exact same approach could be used easily to generate test data (or much more). I used the same approach documented here to later transform instances of case classes as well.

Here is the completed code put together:

Footnote

I have mentioned so far that Shapeless provides the means to transform case classes and sealed traits to generic structures, but have only really talked about case classes. The sealed trait is a slightly different shape to a case class (it is analogous to the algebraic data type co-product) and accordingly, Shapeless has a different structure to handle it, rather than a HList it has a type called Coproduct. The above code can be extended to also handle sealed traits, to do so you just need to add the implicit to handle the empty coproduct (CNil) and recursively handle the Coproduct.

This is intended to be quite a simple high level approach to using some of these ideas and techniques for searching solution spaces. As part of my dissertation back when I was in university, I used this approach to find optimal configurations for Neural Networks. More recently, I have applied this approach to finding good configurations for training an OpenNLP model, with pretty good results - which is really what made me think to write a little about the topic.

So what is it?

Genetic algorithms are a subset of evolutionary computing that borrow techniques from classic evolution theories to help find solutions to problems within a large search space.

So, what does that mean? It sounds a little like a bunch of academic rhetoric that make nothing any clearer, right? But actually, the concept is quite simple, and works like the evolutionary theory of "Survival of the fittest" - that is, you generate a large set of random possible solutions to the problem and see how they all perform at solving the problem, and from there, you "evolve" the solutions allowing the best solutions to progress and evolve, cutting out the worst performing solutions.

How it works

In practical terms, at a high level, it involves the following steps:

Create an encoded representation for the problem solution - This might sound complicated, but this just means capture all the possible inputs or configuration for a solution to the problem.
Randomly generate a set of possible solutions - that is, a set of configurations/inputs (this is called a "population")
Asses the performance of these solutions - to do this, you need a concept of "fitness" or how well a solution performs - and rank the population
Evolve the population of solutions - this can involve a range of techniques of varying complexity, often keeping the top n candidates, dropping the bottom m candidates and then evolving/mutating the rest of the new population
Repeat the rank & evolve cycle (depending on how it performs)

To understand it lets first consider a simple, hypothetical numeric problem, let's say you have some function f(x, y, z) that returns an integer, and we want to find a combination of x, y and z that gives the a value closest to 10,000 (e.g. you want a cuboid with volume of 10,000 and you want to find dimensions for the width, depth and height that achieve this - like I said, this is just hypothetical). The first step would be to encode the solution, which is pretty simple - we have three inputs, so our candidate looks like this:

case class Candidate(x: Int, y: Int, z: Int)

We will also need a fitness function, that will asses how a candidate performs (really, this will just be evaluation of the inputs against our function f)

def fitness(candidate: Candidate): Int = Maths.abs(10000 - (x * y * z))

Now we have a representation of the solution, in generic terms, and we have a function that can measure how well they perform, we can randomly generate a population - the size of the population can be chosen as suits, the bigger the population the longer an evolution cycle will take, but of course gives you a wider gene pool for potential solutions which improves your chances of a better solution.

Once we have our initial population, we can evaluate them all against our fitness functions and rank them - from there, we can evolve!

Survival of the fit, only the strong survive

So how do we evolve our candidates? There are a few techniques, we will look at the simplest here:

Survival of the fittest - the simple and sensible approach that the top performing candidate in your population always survives as is (in the scenario where by we have stumbled upon the optimal solution in the first random generation, we want to make sure this is preserved)
Mutation - much like you might imagine genes get mutated in normal evolution, we can take a number of candidates and slightly adjust the values in our candidate - for numeric "genes" its easy - we can just adjust the number, a simple popular technique for doing this is Gaussian mutation, which ensures that in most cases the the value will be close to the original value, but there are chances that it could be a larger deviation (it's a bell curve distribution, whereby the peak of the bell curve is the original value).
Cross breeding - again, like normal reproduction, randomly select two parent candidates (probably from the top x% of the candidate pool) and take some attributes from each parent to make up a child

Between mutation and cross breeding you can create a new pool of candidates and continue the search for a solution.

Some code

Partly because I wanted to learn a bit more about the Shapeless library (more of that in another post) and partly for fun, I created a simple GitHub project with some code that has very simple evolutionary search machinery that allows you to try and use these techniques.

The shapeless stuff was to support generic encoding of our solution - so the library can easily be used to define a solution representation and a fitness function.

Following on from the example above to find three dimensions that have a volume of 10,000 I have the following configuration:
In the above, we first define the case class ThreeNumbers - this can be any case class you like, but must have Gene[A] as its attributes - these are custom types that I created that have defined generation and mutation methods, and have additional configuration (such as max and min values for numbers etc). We then pass all the configuration in, along with a fitness function of the form (a: B => Double) and run it.

It varies on each run, as the initial pool of candidates is generated at random, but in most cases this example solves the problem (finds an error rate of zero) within 10 generations:

As mentioned, I have used this approach to train OpenNLP NER models - the different training data generation parameters and the training configuration was all encoded as a candidate class and then used this method to evolve to find the best training configuration for named entity recognition, but it can be applied to all sorts of problems!

Feel free to try it out and let me know if it works well for any problem domains - you can also checkout the code and make it more sophisticated in terms of evolution/mutation techniques and introducing a learning rate.

"Cash Rules Everything Around Me" - Method Man

A few years ago I was fed up of how useless all the banks were at helping me understand my transactions/spending patterns with their online banking. Basically, they all just seemed to let you log on, and then see a list of transactions (often just referencing codes) of payments in and out by date. Nothing else. It seemed like there were lots of simple things missing that could make it a lot more useful, to name a few:

Smarter tagging/categorisation - where some direct debit was just listed by the code of the recipient rather than meaningful info, or being able to group all the spending in coffee shops
Alerting to changes/unexpected behaviour - for example when a fixed term-price contract comes to an end and the price changes - detecting a change in a regular fixed aount/recipient should be easy

So this is what I started building - a lot of the functionality has since been built into banks such as Monzo, the aim of this webapp was simply to allow you to import transaction histories (most online banks I use provide the ability to export transactions in csv) so you could view/filter/tag any existing bank data. It only supports Santander at the moment, and the alerting stuff never actually got built in, but I thought I would chuck it on Github rather than have it just sit around gathering dust in a private repo.

I have written about the project a while ago, when it was first underway, but as it has been not worked on for a while, I thought I would move it to GitHub - hence the post here today. The app and all the code can be downloaded here: https://github.com/robhinds/cream

Building & Running

Its a simple java, maven app (it was several years ago, so not migrated to Spring Boot, Gradle, Groovy etc) - and it builds a WAR file. It also uses LESS for style stuff, which is also built by maven (although watch out for silent failures if you make changes!). If you are using an eclipse based IDE you can follow the guide here: to get incremental build support for LESS building (e.g. just change the LESS source and go refresh in the browser and it will reload).

You can run the WAR file in any tomcat/IDE server, and the backend is currently expecting a MySql DB (but can easily be switched for a differed DB driver if required)

Roadmap

Migrate to Spring Boot, Groovy, Gradle
Add further bank transaction importers
Add alerting framework

Screenshots

(I loaded in a generated transaction history - which is why some of the charts look a little weird)

Honestly, I still need convincing on microservices.

I can see that they are a compelling argument compared to a monolithic application, but I think I need to get my head around some of the challenges they face - the first one that comes to mind being how to effectively define the microservice boundaries - as it seems to me a lot of the applications I have ever worked with are monolithic because these boundaries are so blurred.

Anyway, I wanted to do some tech stuff, so decided to start building out an application using the microservice architectural pattern and Spring Boot seemed like a good place to get started.

This is very much a work in progress, and I am continuing to progress through different aspects of the application and at the moment there is very little actual code written (in part that is due to the simplicity that Spring-Boot provides). All code is being kept up to date in GitHub so feel free to have a look at that.

There are lots of great blogs covering this stuff already, so won't re-cover their work, the following article gives a great write up of the Netflix OSS and the Spring integration which is worth reading:

http://callistaenterprise.se/blogg/teknik/2015/04/10/building-microservices-with-spring-cloud-and-netflix-oss-part-1/

(Image from: Building Microservices with Spring-Cloud and Netflix OSS)

Getting started: A service registry - Eureka

One of the first things that is needed is a central Service Registry to allow service-discovery - this is not a new concept to microservices and is an approach used by SOA. Straight out of the box, Spring-Boot provides integration with Netflix's OSS application Eureka, that provides this. I opted to have a dedicated application for my registry (code can be seen here) and it really is as simple as adding the relevant dependencies to the build.gradle file, adding an @EnableEurekaServer annotation to our application config then a simple config file defining the server port/name etc and its done! You can just run gradle assemble in that project to build the JAR file, then run java -jar [the new JAR file] and the application will spin up - you should then be able to go to http://localhost:1111 and you will see the Eureka dashboard (with no microservices registered of course).

My first microservice

So, I had Eureka up and running, but it was looking pretty lonely with no services registered.

A microservice in Spring is also very simple - as really, all it is is a simple web application that runs in its own process with a limited domain - so spinning up a Spring Boot MVC RESTful webapp with a single controller/endpoint is enough to get me a microservice (even just a tweet would do it..)

@Controller class ThisWillActuallyRun { @RequestMapping("/") @ResponseBody String home() { "Hello World!" } }
— Rob Winch (@rob_winch) August 6, 2013

So we can create our new microservice to do anything we like, in my case I created a QuoteService (the application is slowly evolving into an insurance engine). Just having the standalone app isn't helping much, so we need to add some configuration to tell the service to register with our Eureka server - this will make our new microservice discoverable by other services wanting to use it.

Again this is quite simple: we need to tell our application it should try and register with Eureka, and we should add the config to do so:

You can see that we simply annotate our application config in java, and then add some properties that define where the Eureka server is hosted and that's basically it.

Now if we build the project JAR and start up again (and we still have our Eureka service registry running) then after 30seconds or so you should see the Quote-Service registered and ready to use.

On to the next one..

Now, we have a microservice, and we have a registry that makes it discoverable, but still - just one microservice is pretty lonely. So next I created another dummy RESTful microservice, this time called ProductService which just followed the same pattern as the first.

Once that was started up then the Eureka dashboard started looking a bit happier with the two services registered - the obvious next challenge is seamless interaction between the two: splitting the services into their own processes is all good an well, but meaningless if you can't easily integrate them. The way I look at it is when reading the application code of a service (or application using microservices) then it should just look like a normal application with sevice classes - there shouldn't be any fanfare around the fact that my service class actually gets the data from a dedicated microservice over HTTP/AMQP rather than just getting directly from the DB in the traditional way.

So, still just stubbing out the endpoints, I updated my QuoteService endpoint to make a call to my ProductService, and then just jammed that response into the JSON response I was returning anyway:

As you can see, it could be a standard controller in a normal monolithic application from this point, we are just calling a method on our autowired ProductService class and returning that.

So the really interesting part is in the ProductService class - at the moment this isn't a really elegant, abstracted class yes, so there is still some boiler plate, but that will have the advantage of making it clear what is going on:

As you can see, it's just making a REST call to the Product microservice and returning the response cast as a Map - but the really nice part of this is that the service url is just the service name (in this case "PRODUCT-SERVICE", that is injected to the class) and with the RestTemplate annotated with Spring-Cloud's @LoadBalanced that microservice will be looked up in Eureka (and load balanced if there is more than one PRODUCT-SERVICE running).

So our setup is starting to take shape now - we have two microservices, both registered with Eureka and able to interact with each other in a fairly clean, loosely coupled way.

Don't push me, 'cos I'm close to the edge..

As your microservices start to proliferate, you will get different levels of service granularity, and undoubtedly you won't just want to expose all your microservices as a public API. One option would be to create a RESTful application and define nicely named endpoints you want to expose and then use the standard integration described above to integrate it.

Fortunately, there is an easier way - Netflix provides a library called Zuul that can be simply configured to map URL patterns to given defined service names (and again looks up in the Eureka service registry). Much like Eureka, this is super easy to setup and just needs an annotation and the config again:

And the config is pretty easy to understand:

As you can see, we just define service names against URL patterns.

Now once all are apps are up and running, and the microservices are registered on Eureka then you have a single API interface to start interacting with the services (rather than having to access each service on its designated port etc).

Conclusion

So that's as far as I have got - I wired up the QuoteService to MongoDB so the data all gets persisted there (and have added a get quote endpoint which gets the same data from mongo) and starting to wire up the product service with JPA. So far it's been enjoyable, and things are making more sense than when I started - but there are a few questions still:

It seems like there is still duplication of service names throughout the different projects - for example the ProductService name ("product-service" - case insensitive) is proliferated throughtout - the service itself defines it, the QuoteService needs to know the name of the service, the Zuul edge server needs to know the name etc. I guess this is unavoidable as these are intrinsic dependencies but still seems a bit flaky.
It feels like the Service classes could be factored out - our ProductService class that allows HTTP REST interactions with the Product microservice would likely need to be re-used across all applications/microservices that need to use the Product microservice

I have recently been building a Spring MVC app (well, actually using the Spring Boot project - which is quite nice in parts, and crazy frustrating in others - but the underlying mechanics are the same). The application is actually a re-build of another application, so it has involved a lot of playing and exploring Spring source code to try and replicate the apps functionality like for like.

One of the first things I found, was that Spring doesn't really cater for the concept of a single app running on different sub-domains. I assume the thinking is that you would build separate applications for different sub-domains, but in this case, we just have a single app.

There are two main speed-bumps I have come across so far:

Controller routing based on subdomain
Security considerations based on subdomain

Basically, as you can probably imagine, once you have subdomains on a single web application, the URL path is no longer unique (e.g. http://automateddeveloper.blogspot.com/ is clearly not the same as http://blogspot.com/ )

The rest of this post, I will look at the routing element of it. I will do another post later in the week about the Spring security stuff (I haven't solved all of that yet - but got far enough to make it work).

Understanding the subdomain

The first thing you need to sort out is a common and consistent way to determine the subdomain of any given request. There are a variety of ways to do this, for example, you could parse it from the request in apache and set a header, so your app doesn't have to worry about it, or you could just parse it from the request object server name. I will assume you have some bean/service/helper class to do this everywhere (although for now we only need it in one place).

An annotation

First up, we need a new annotation, that we can easily apply to a Controller, just like we would use the @RequestMapping - at the moment, the nice built in Spring RequestMapping handling allows you to define a URL path (plus other bits and pieces) to map a request to a URL to a given controller & method - what we want is to also specify the subdomain element of the requested URL

Defining an annotation is simple:

For the sake of simplicity of this example, I will only allow it to be used at class level (so no method based subdomain routing - but that will be pretty easy to do once you have understood the rest of the post). You will also note that the value is defined as a String array, this will allow us to define mappings to multiple subdomains if needed.

The mapping condition

So, that was easy. Obviously, at this point the annotation doesn't actually do anything - you can add it to all the controllers you like, but it won't actually make any difference to your request routing.

To get our new annotation involved, we can implement something called a RequestCondition. This is exactly what it sounds like, Spring lets you implement additional request conditions that must be satisfied for a request mapping.

The condition could be based on any logic you like, but in our case we just simply need to check for an annotation and then examine the value provided. If the annotation value matches our incoming request then the condition is met, easy! Returning the condition indicates to Spring that the condition has been met, returning the null value indicates that the condition is not met.

Adding the condition to the mapping handler

Usually, we would use the standard Spring RequestMappingHandlerMapping to handle all the routing of requests based on the URLs, but now we need to also ask Spring to consider our new custom condition from above.

This is a simple case of extending the normal RequestMappingHandlerMapping class and adding our new condition as a custom condition. Luckily, this is really easy:

Allwe are doing is checking to see if the handler class (our controller) has our new @Subdomain annotation and if it does, we register our new custom condition for consideration.

Basically..

That's really all there is to it - we can then decorate our controller class with @Subdomain("subdomain") and have them handle the routing of requests.

As you may have noticed, this is a pretty nice pattern for any kind of custom routing you might want to use - the same template could be used for routing by any request info/header or any user info (e.g. routing requests to different controllers based on their logged in role etc)

Has there ever been a wiki that actually works, I mean, other than the obvious one?

They seemed to explode on to the scene around ten years ago as the future of on-line collaboration and communication, and quickly became a mainstay of tech companies/projects as their knowledge base and documentation. I have worked on several large tech projects as a consultant over the last decade, and almost all of them treated getting a wiki setup only second to basic infrastructure and version control.

Thing is though, they kinda suck.

A few years ago I was working as a consultant on a large, internationally based project, and found myself thinking about the fragmentation, and frankly the madness, of the variety of communication and collaborative tools that were being used. We had messaging apps, email, about three different ticketing systems and a wiki, which lead me to think more about wikis.

They became popular because they suggested collaborative writing and editing of documents, and I guess a lot of managers grabbed hold of it as a way to get developers passionate about writing documentation and capturing the teams knowledge. But at the end of the day, writing documentation is still just as dull as ever - developers still won’t enjoy it, even if it is in a wiki rather than a Word doc. And assuming you are able to get them to write the docs in the first place, there is no user engagement or incentive to keep it up to date. The second someone actually finds something on the wiki isn't correct/working, they will just go ask someone.

Personally, I like the Q’n’A site model a la StackOverflow.com. It doesn't fit across the board (still a place for more formal documentation etc), but it seems like a much better model for capturing company/project knowledge.

Wikis don’t seem to offer much over normal documentation approaches (and seem to offer less than something like Google Docs, which would seem to have superior collaborative tooling).

Forums would offer a more conversation lead approach, which I think would increase user engagement, but it can be hard to find answers amongst a sequential conversation thread.

A Q’n’A site seems to offer the best, with a more conversation based approach for a better, more engaging user experience, but with crowd sourced, peer-verified answers, making it easy to find the answer you are looking for. Throw in the gamification approach StackOverflow uses for good measure and I suspect you may see your knowledge capital growing pretty quick.

It surprises me that there isn't an established, open-source Q’n’A product.

Someone should build one.

I don't normally do side-projects using ColdFusion, but one night some time last year, after drinking some neat vodka, I ended up playing around with ColdFusion and building an MVC framework.

The code isn't by any means production ready, and is only the product of two nights (and several glasses of vodka) but it was an experiment into what kind of nice MVC framework type functionality could be exposed in CF. I have only ever used Fusebox, which as an MVC framework is found wanting in many instances, however, I know there are lots of other MVC frameworks in CF, that undoubtedly have nicer features.

Anyway, it's all over on my GitHub, but I thought I would re-post the details here. Just for funsies.

Some of the main things I wanted to experiment with were involved with simplifying of the framework and removal of boiler plate code. E.g. :

Annotation or config defined mapping for URL > Controller methods
Simple returns from controllers - just returning view name and model data (to remove the temptation of building HTML code in controllers - a clearer separation of controller and view
clearly & appropriately scoped variables in the views (everything to local, as under the hood, everything is being rendered from a CFC, so if you don't local scope the view then you may get surprises)

Controllers:

At the moment, it is dead simple to configure, just grab the framework and create a controller as follows (it assumes that there will be a directory in the root of your webapp called controllers)

/controllers/Home.cfc:

/**
 * @Controller
 **/
component output = "false"  {

    /**
     * @RequestMapping
     **/
    public Array function default( Struct resourceParams="" ){
        return [ "homepage" ];
    }
}

The above code will direct all requests to the application root ("/") to the default() function. The URL pattern for either the @Controller or @RequestMapping can be left blank (or populated) - the values provided are concatenated to produce the URL mapping for a function.

You can also have variable sections of the URL: /controllers/User.cfc

/**
 * @Controller /user
 **/
component output = "false"  {

    /**
     * @RequestMapping /:username
     **/
    public Array function default( Struct resourceParams="" ){
        return [ "userpage" ];
    }
}

In the above example, you can see that @Controller and @RequestMapping have a URL pattern defined. You will also note the @RequestMapping has a ":" before username - this indicates a variable section of the URL. This code will handle requests that hit the URL /user/rob etc - Any named variable section of the URL will be available to the controller function in teh resourceParams struct (as seen above).

Views:

View definition and resolution is also real simple (hopefully). Firstly, you can define a view layout by name - this defines a named view and CFM template, along with any templates that make up the overall view. This is all configured in JSON notation.

At startup, the framework scans the /views directory for any "*.config" files - any it finds it loads up, and then the views defined will be available for use in the application.

/views/view-layout.config:

[
    {
        "name": "homepage",
        "template": "home.pagetemplate",
        "include": {
            "header": "home.header",
            "body": "home.body",
            "footer": "home.footer"
        }
    },
    {
        "name": "userpage",
        "template": "user.pagetemplate",
        "include": {
            "header": "user.header",
            "body": "user.body",
            "footer": "user.footer"
        }
    }
]

The above defines two views, "homepage" & "userpage" - both are made up of similar components. The convention for the templates is relative to the /views directory, and uses "." instead of "/" (e.g. "template": "user.pagetemplate" will expect a file /views/user/pagetemplate.cfm ).

As you can hopefully work out from the above, the "homepage" view is based on /views/home/pagetemplate.cfm and is made up of the three templates /views/home/header.cfm, body.cfm, footer.cfm.

All nested templates are simply placed in the local scope (for the template being rendered), so our pagetemplate cfm looks like this:

/views/home/pagetemplate.cfm:

<html lang="en">
    <cfoutput>#local.header#</cfoutput>
    <body>
        <cfoutput>#local.body#</cfoutput>
        <cfoutput>#local.footer#</cfoutput>
    </body>
</html>

Once we have built a view definition, and have created our templates, a controller simply has to return the viewname as the first argument of an array. If we look again at our Home controller:

/controllers/Home.cfc:

/**
 * @Controller
 **/
component output = "false"  {

    /**
     * @RequestMapping
     **/
    public Array function default( Struct resourceParams="" ){
        return [ "homepage" ];
    }
}

We see that we just want to render the "homepage" view for all requests. Simple controller & simple view composition. Pretty boss.

Models:

Obviously, most of the time we will want to pass data (in the form of a model) to our views for rendering, this is also really simple!

We have seen that in the controller, if there are variable names in the URL that is passed to the controller function in the arguments.resourceParams struct - That data is also automatically passed through to the view template, but can be supplemented by any data accessed in the controller.

Let's look again at out User controller, but lets say we want to pass some additional data to the view for rendering:

/controllers/User.cfc:

/**
 * @Controller /user
 **/
component output = "false"  {

    /**
     * @RequestMapping /:username
     **/
    public Array function default( Struct resourceParams="" ){
        return [ "userpage", { greeting: "Yo!"} ];
    }
}

This time, we are returning a second element in the array - this is a Struct of the model data, this Struct could contain any data you have fetched in the controller that you want to pass to your view template for rendering. As we have a variable URL section (username) and are passing some model data back to the view, both of these can be used.

/views/user/body.cfm:

<h1><cfoutput>#local.model.greeting#</cfoutput> <cfoutput>#local.model.username#</cfoutput>!</h1>

The URL variable sections and the returned model data are all available to the templates in "local.model".

A while ago I came across a dashboard webapp made by Shopify called Dashing. It's a fairly simple concept, it's a web dashboard using Ruby & some slick js libraries to create your own custom dashboard with stats/graphs/etc. The Shopify guys had made it for use internally showing their own stats and had open sourced.

I had a look around, and it looked pretty nice, and I was interested in how they had put it together, so decided to have a go at making my own version of it.

Obviously the real work is in making all the different widget adaptors so they can be re-used & fed any data needed to make it highly customisable - the kind of work that can only be done/worthwhile if working on an actual product with real data/use-cases.

You can see an example of the Shopify guys library over on Heroku (when it's running) - and their code is on their GitHub.

The version I built was originally built with ColdFusion 10 and a bunch of javascript libraries - including backbone.js. Here are some screenshots - it's pretty simple, just some drag & drop components with pretty colours and widgets - so far an RSS reading widget, a line chart, a gage and a basic text widget(latest tweet).

My code is all here. My demo is here (demo is actually the same code dropped into a quick spring app, as I don't have CF hosting) - running on AppFog - when it's running. I actually wrote this some time ago last year, but figured I might as well jam it up here, with some screenshots

My side project, formerly known as "Butterfly", is approaching completion of its first beta approach, which is an exciting time. It has been in development in one way or another for some time, so its going to be great to see it finally getting an early release out there and (hopefully) seeing some feedback from real users.

It's in the final stages of getting together some slides to kind of show it off, sorting out a final URL and hopefully putting together a stand alone WAR file that will come bundled with Jetty/HSQL so users can just fire it up and test it out locally in an attempt to lower the barrier to adoption.

In the meantime, here is a bit of a blurb about what its actually all about..

Having worked in Enterprise software, on large scale projects spread across several geographic sites (often across several countries) we have seen and used several disparate, mediocre tooling to support collaboration and communication across the teams (even across teams on the same site), as an example of this, on my latest project engagement we have the following toolset:

A web application to track defects/bugs
A web application to manage work tickets (for example, requesting infrastructure team to patch/restart servers etc)
A web application to manage internal work items/ToDo lists within the team
An internal wiki
Email (of course)
Online chats

This was not unique to this project, but the obvious problem was that with all these disparate channels and mechanisms it made collaboration more difficult than it probably should be. For example, having multiple web apps to manage common tasks meant that most users were never "always on" any one tool, so to raise a ticket/defect/todo it involved opening app, logging in etc. It also meant that knowledge was not easily centralised or searchable (searching for info across several web apps, email, and online chat archives is a pain in the ass) - The project attempts to tackle this inefficiency by building a single, user-centric web application that has these common communication and collaboration tasks at its heart whilst also borrowing metaphors and familiar mechanisms from popular social sites such as live feed "walls" for latest activity. Being user-centric it also means we can effectively capture the team/user relationships across the project.

Oh, and its called "Flutterby" now..

(there will be a more detailed write up shortly on the site once launched.. which is still pending the URL)

I have been working on a side project for what seems like forever now, but I have set up a blog dedicated to it. All it currently has on it is a few screenshots and an introductory post about what exactly it is, but you can check it out here.

The source code isn't actually available yet as its not even in beta yet, but I'm hoping it will surface before the end of the summer. Would love to hear peoples thoughts on the project!

Custom Directives

Custom Serialisation

Conclusion

Enter Shapeless

A case class, product and HList walk into a bar

Case class to HList and back again

Back to the problem at hand

Footnote

So what is it?

How it works

Survival of the fit, only the strong survive

Some code

Building & Running

Roadmap

Screenshots

Getting started: A service registry - Eureka

My first microservice

On to the next one..

Don't push me, 'cos I'm close to the edge..

Conclusion

Understanding the subdomain

An annotation

The mapping condition

Adding the condition to the mapping handler

Basically..

Thing is though, they kinda suck.

Someone should build one.

Controllers:

Views:

Models:

ROB HINDS

Popular Posts

Blog Archive

DZone MVB