The Adjacent Possible

At Christmas I began reading a book entitled “Where Good Ideas Come From” by Steve Johnson. The book, released last Autumn had a video trailer that was doing the rounds on the internet which was causing quite a buzz, and I have an interest in innovative/entrepreneurial things so picked it up.

The book is supposed to be a history of innovation, and after a brief introduction Steve looks into 7 different “phenomenon” or patterns in innovation, drawing on examples from throughout history.

The first topic Steve opens with is the idea of the “Adjacent Possible” – this theory was originally proposed by Stuart Kauffman,  and relates to evolution of life. Essentially the theory is that evolution happens around the “edges” of existence and evolution takes small steps, for example, in the primordial soup the evolution was more like singular cellular organisms evolving in to multi-cellular organisms, rather than making a bigger leap to evolve an eye for example.  The analogy Steve used was of a huge building of interconnected rooms, with doors on every side of the room, and to progress, or to get somewhere new, you can only use a door to connect to the next room. You can never jump many rooms ahead, but are limited to progress one room at a time via the “Adjacent Possible”.

The point that Steve draws from this is that innovation is similar in nature, and many of the big innovations have been a small change to existing thoughts, or a combination of existing thoughts, pushing the benefit of working and mixing with a range of cultures/arts/technologies so you place yourself on the edge of innovation, and therefore closer to the adjacent possible. Whilst saying it out loud or thinking about it makes is sound quite obvious, I think it’s something that is easy to miss, and as technologists(read nerds) it’s easy to focus on immersing ourselves in different technologies/frameworks for all our side projects, but actually, focussing on innovation, it’s probably more beneficial to spend a bit more time looking at other areas of industry/art/science to see where innovations and thoughts from different ares could come together to form something new and not spend so much time trying to build projects driven by technology.

I really like this idea, if you normalise all current thought/processes/etc in to a square, and inside this is everything that has been created/exists right now, so really the next innovation is going to be right around the border of the square. As it happens, I have found this to be true personally (without any intention), as coming from a professional work environment delivering enterprise software solutions, two of the projects I am currently focussing most of my effort on are about bringing together some of the value, benefits and human behaviours from web2.0/social media applications to the more traditional enterprise project management tasks - which makes a lot of sense, I work in enterprise software delivery, and I see all the processes/software that exists for this job, and my ideas for new projects are around the "edges" of these things, and are coming from looking at what is the adjacent possible to what we have already in this space.


How Testing Improves Design

Recently in work we have been looking at our testing strategies for a web app we are currently working on. The application itself is a Spring MVC java web app, and there are several challenges that we have faced around things like how to best test the persistence layer, testing the UI, mocking HTTP requests to test the controller calls, but today I was looking at testing a Service class that was used by one of our controllers.

The first challenge (as I was going for a bottom-up approach, starting with the most granular methods to test) was what is the best way to private methods. A little bit of initial research showed many people suggesting using reflection to test private methods, however, having spoken to the team's "test architect" he expressed a dislike for this approach as it resulted in failures at runtime (read test run time during build/package with maven) as due to the nature of reflection, if the contract of the class changes this is not flagged up with compile errors in your IDE. His suggested approach was just to make the methods public.  Clearly a bad idea and goes against basic OO principles, so I decided to investigate further..

On further reading, I started seeing a few people making claims such as "When I have private methods in a class that is sufficiently complicated that I feel the need to test the private methods directly, that is a code smell" - and I thought on this for a while, and looked at the code that I was testing and agreed.

Whilst the test architect had been wrong to suggest making all the methods public, I think it's fairer to say that you should be able to fully unit test a class with confidence only using public methods, these after all are your entry points, or contracts with the rest of the application, and your private methods are just implementation details - As the poster above states, if the private methods are so complex that you feel you should test them, then your design is wrong!

We all know about Principle of Single Responsibility and all that, but due to the constraints under which the application is being delivered, the service class had lost its way and many of the large, cumbersome, private methods shouldn't really have been there at all and needed to be refactored in to other relevant objects, where they could be exposed as public methods in themselves (obviously also helping reuse).  This is a common problem and easy to slip in to, you start off creating a Service class to perform some function, and as you develop it, the scope widens and more and more functionality is needed for this task, so it is easy to think that this functionality should be included in the same class, it is after all core to what the class is trying to achieve. However, you need to take a step back, as in this case during the testing, and analyse the class, does the functionality really belong in the class? Is it directly and singley contributing to the goal of the class?

With a quick nip and tuck the daunting task of mountains of private methods had all but disappeared and as well as that I could leave happy that the code was cleaner and more maintainable for it!


Online Lectures

8:15 PM , 0 Comments

On the topic of online lectures, as well as checking out the MIT Open CourseWare lectures, I highly recommend checking out what iTunes U has to offer - there are loads of free lectures, but my favourite it the Entrpreneurial Thought Leaders Series from Stanford.. the range of speakers and lectures that they have archived is incredible, and it really is amazing that you can get it all for free.

Details of the ETL series and the archive can be found here


Quote of the Day

On the way home from work today I was listening to a lecture by Thomas Friedman recorded for MIT entitled "The World is Flat: 3.0". The lecture was essentially about the changing of the world's economic and innovative landscapes, and he has documented three "changes" :

  1. 1492 - ~1880s - This period changed the "size of the world from large to medium - this change was driven by countries (Spain discovering America, Britain colonizing India etc)
  2. ~1880s - 2000 - This period changed from medium to small - this time driven by companies, as companies operated on international scales and above and beyond the geographical limits of the countries
  3. 2000 - Ongoing - this has changed the size of the world to tiny - and is now driven by individuals, primarily changed by the introduction of the PC and the internet, allowing users to connect/upload/work/create with people from all corners of the world.

My favourite quote was on speaking of how the Brits owned the 19th Century, the Americans the 20th and the Chinese will own the 21st:

"As my grandma always used to say, sitting in her rocking chair beside the fire, with her blanket over her legs, never seed a century over to a country that censored Google.."

You can check the lecture yourself over at the MIT open courseware site


Android RSS Tutorial Revisited - Some Questions Asked..

I was going to write a post about unit testing Spring web apps, but I have had a few questions recently about the Android RSS parser that I posted a while back, so I thought I should answer those quickly now (the explanation seemed a little long winded to add as a comment).

The questions were specifically about how to extend the example code provided to parse additional tags. These questions are actually just core SAX questions, and so the answer is basically more detail about SAX and can actually be applied to non RSS/Android parsers as well.

The two questions I wanted to tackle was
1) How to extend the code to handle additional tags
2) How to extend the code if there are tags that should be processed inside other tags (for example, an tag inside the tag

Both tasks require an understanding of the three core methods in the SAX Parser:
- startElement() – this is called every time an opening XML tag is found of any nature
- endElement() – This is called every time a closing XML tag is found of any nature
- characters() – this is called between start and end tags, but not necessarily at the end of the text being processed (this is very important to remember!)

In our very simple example, we are assuming that we are only ever interested in the lowest level leaf nodes of the XML feed, and they will only ever contain text. From this assumption, in the startElements() method, we don’t need to process which tag we are in, we can just reset the StringBuffer (the StringBuffer is used to gather text in between tags). By resetting the buffer every time we start an element, we know that when we close an element that we are interested in (as it is always only the leaf node) we have the text contents of that node in the buffer.

If you inspect the endElements() method from the original example, you will a nested IF block- this block examines the name of the tag being closed and decides if we are interested in the content of the current closing node. If we are interested, then we take the text from the buffer and save it in our context object, otherwise we just ignore it, and the buffer will be cleared when the next xml node is opened.

Therefore, to extend just to add another node, we can just add another condition to the IF block with the name of the new tag. Easy!
public void endElement(String uri, String localName, String qName) throws SAXException {

        if (localName.equalsIgnoreCase("title"))
            Log.d("LOGGING RSS XML", "Setting article title: " + chars.toString());

        else if (localName.equalsIgnoreCase("description"))
            Log.d("LOGGING RSS XML", "Setting article description: " + chars.toString());
        else if (localName.equalsIgnoreCase("the name of any new node goes here!"))
            //here you can handle the node contents as you like.. chars.toString() will give you all
            //the contents of the node

The second issue is a little more challenging, as it requires some additional processing, and we have to turn to our startElement() method.

Lets imagine we are processing an XML that looks as follows:

        example feed title
        example description of feed, and here is a picture i hope you like the feed!

The title and date are processed by our existing code easily, as when the parser reaches the opening title or pubDate tag it clears the buffer, and then when it reaches the closing tag for each node, it checks the IF block and saves the text in the node into our context object.

The description tag is more complex, as we want to keep the text, but currently, as soon as the inner img tag is reached by our parser, the buffer is cleared, and this clears all the description text we have up until that point.

To handle this scenario, the easiest way is to process the opening node in the startElement() method, and have a secondary buffer that is used specifically for the description node – this way, when we open the description node, we reset the description buffer, but it will not get reset at the img node, then in the endElement() block we add the condition to handle the closing of the tag. The full code to handle the above scenario is below: (note, we also need to update our characters() method to consume the text between the nodes – for performance, I have set a flag so it knows only to write to our new description specific buffer when inside the node)

public class RSSHandler extends DefaultHandler {

    // Feed and Article objects to use for temporary storage
    private Article currentArticle = new Article();
    private List
articleList = new ArrayList
(); // Number of articles added so far private int articlesAdded = 0; // Number of articles to download private static final int ARTICLES_LIMIT = 15; //Current characters being accumulated StringBuffer chars = new StringBuffer(); /** * THIS IS A NEW BUFFER SPECIFICALLY FOR DESCRIPTION **/ StringBuffer descriptionChars = new StringBuffer(); boolean processingDescription = false; /* * This method is called everytime a start element is found (an opening XML marker) * here we always reset the characters StringBuffer as we are only currently interested * in the the text values stored at leaf nodes * * (non-Javadoc) * @see org.xml.sax.helpers.DefaultHandler#startElement(java.lang.String, java.lang.String, java.lang.String, org.xml.sax.Attributes) */ public void startElement(String uri, String localName, String qName, Attributes atts) { chars = new StringBuffer(); //IF DESCRIPTION THEN SET FLAG TO START PROCESSING SPECIFIC BUFFER if (localName.equalsIgnoreCase("description")) { descriptionChars = new StringBuffer(); processingDescription = true; } } /* * This method is called everytime an end element is found (a closing XML marker) * here we check what element is being closed, if it is a relevant leaf node that we are * checking, such as Title, then we get the characters we have accumulated in the StringBuffer * and set the current Article's title to the value * * If this is closing the "Item", it means it is the end of the article, so we add that to the list * and then reset our Article object for the next one on the stream * * * (non-Javadoc) * @see org.xml.sax.helpers.DefaultHandler#endElement(java.lang.String, java.lang.String, java.lang.String) */ public void endElement(String uri, String localName, String qName) throws SAXException { if (localName.equalsIgnoreCase("title")) { Log.d("LOGGING RSS XML", "Setting article title: " + chars.toString()); currentArticle.setTitle(chars.toString()); } else if (localName.equalsIgnoreCase("description")) { /** * ALSO PROCESS HERE THE DECRIPTION SPECIFIC STRING BUFFER **/ Log.d("LOGGING RSS XML", "Setting article description: " + descriptionChars.toString()); currentArticle.setDescription(descriptionChars.toString()); //ALSO SET DESCRIPTION FLAG TO FALSE processingDescription = false; } else if (localName.equalsIgnoreCase("pubDate")) { Log.d("LOGGING RSS XML", "Setting article published date: " + chars.toString()); currentArticle.setPubDate(chars.toString()); } else if (localName.equalsIgnoreCase("encoded")) { Log.d("LOGGING RSS XML", "Setting article content: " + chars.toString()); currentArticle.setEncodedContent(chars.toString()); } else if (localName.equalsIgnoreCase("item")) { } /** * THIS IS NEW TO HANDLE IMAGE TAG **/ else if (localName.equalsIgnoreCase("img")) { Log.d("LOGGING RSS XML", "Setting article description image: " + chars.toString()); currentArticle.setDescription(chars.toString()); } else if (localName.equalsIgnoreCase("link")) { try { Log.d("LOGGING RSS XML", "Setting article link url: " + chars.toString()); currentArticle.setUrl(new URL(chars.toString())); } catch (MalformedURLException e) { Log.e("RSA Error", e.getMessage()); } } // Check if looking for article, and if article is complete if (localName.equalsIgnoreCase("item")) { articleList.add(currentArticle); currentArticle = new Article(); // Lets check if we've hit our limit on number of articles articlesAdded++; if (articlesAdded >= ARTICLES_LIMIT) { throw new SAXException(); } } } /* * This method is called when characters are found in between XML markers, however, there is no * guarante that this will be called at the end of the node, or that it will be called only once * , so we just accumulate these and then deal with them in endElement() to be sure we have all the * text * * (non-Javadoc) * @see org.xml.sax.helpers.DefaultHandler#characters(char[], int, int) */ public void characters(char ch[], int start, int length) { chars.append(new String(ch, start, length)); if (processingDescription){ descriptionChars.append(new String(ch, start, length)); } } /** * This is the entry point to the parser and creates the feed to be parsed * * @param feedUrl * @return */ public List
getLatestArticles(String feedUrl) { URL url = null; try { SAXParserFactory spf = SAXParserFactory.newInstance(); SAXParser sp = spf.newSAXParser(); XMLReader xr = sp.getXMLReader(); url = new URL(feedUrl); xr.setContentHandler(this); xr.parse(new InputSource(url.openStream())); } catch (IOException e) { Log.e("RSS Handler IO", e.getMessage() + " >> " + e.toString()); } catch (SAXException e) { Log.e("RSS Handler SAX", e.toString()); } catch (ParserConfigurationException e) { Log.e("RSS Handler Parser Config", e.toString()); } return articleList; } }