Building a Twitter style autocomplete for Android

I have recently been working on an android app and found myself needing a Twitter style input box that allowed certain words to be looked up using an API driven autocomplete but also supporting freetext (if you have used the compose tweet input on android, like this - free text, but if you type @ then you get a user name autocomplete).



I google'd for a while, and stumbled through a few StackOverflow answers but none of there were 100% clear, and further more, just copy-pasting the code into a dummy project I was working in didn't work. So here is a break down of the what and how this works.

There are four main components:

The XML Layout: MultiAutoCompleteTextView


This is simple - just add the MultiAutoCompleteTextView just like you would any input. The only point of interest here is the "completionThreshold" - this is just the number of characters that have to be typed before auto-complete kicks in. We have set this to one char so it kicks in early.


We then just setup the auto-complete in our Activity onCreate

SetTokeniser: UsernameTokenizer

This is a custom class that implements the Tokenizer interface. This will be used by our autocomplete imput box to work out whether or not it should be displaying a dropdown menu. If you are using a static list for lookups (countries, fixed codes from your app, etc) then this is the only thing you really need to do - then you can just set the fields ArrayAdapter as the list of Strings etc and it will automatically kick in.

In our case, we are using the @ character to identify the start of pieces of text that should be lookedup and spaces to determine the end of the look up token.


The three methods are relatively straight forward - and will drive when your app presents the drop down:
  • terminateToken(CharSequence text) - this basically just provides a presentable version of our token with a proper terminator at the end (e.g. makes sure a trailing single space in this case)
  • findTokenStart(CharSequence text, int cursor) - Just finds the position of the start of the token. It does this by iterating backwards through the provided CharSequence (the text input in the input box) starting from the position of the cursor, which is just the position of the last character edited (this means if you go back and edit text in the middle of a block it still finds the correct token).  If no valid token start is found (e.g. we go backwards and we can't find a @ character) then the current position is returned - no dropdown is displayed.
  • findTokenEnd(CharSequence text, int cursor) - As above, but finds the end position. Iterates forward until a token terminator (in our case a space) or the end of the text is found

As long as you implement these to support your token identification pattern then you will get a dropdown appearing appropriately.


Adding a text changed listener

For ease of use on this one, I have just set the activity to implement the TextWatcher interface - the reason for this is just convenience - the implementation is going to handle calling our API asyncronously, so it is easier if it has the activity context.

There are three methods that need to be implemented - four our case there will be two no-op methods and just one implementation:


The method onTextChanged is implemented - unfortunately the tokenizer will only indicate to the application when to display the dropdown - it doesn't actually call the API, so we have to slightly repeat ourselves here in handling the API invocation.  In this method we need to again check for and find the relevant valid token in the input, and if a valid token found, then pass it to our API to lookup the dataset.

To help with that, I also added some additional methods to help check for the existence of a valid token (reading them should be self explanatory, but comments included inline)


In this case, as it was just an experiment, and I didn't actually have a user lookup API I have just used the LinkedIn skills API (this code also taken in part from a StackOverflow answer).

The Async task should also be straight forward, and can be implemented in whatever pattern you are using for Async calls - but the key point to note is how we are updating our dropdown list.


The API driven, multi-autocomplete/free text field should then be working as expected



GitHub as a CV (GaaC)

Over the recent year or so, as it has become more popular throughout the tech industry, there has been a growing amount of discussion around the idea of "GitHub as a CV" - the idea of using your online tech footprint, primarily for most people in the form of your GitHub profile, as a CV and a better representation of a potential employees ability/preference/mindset etc. There has even been a GitHub project that can automatically create a CV for you based on your profile: http://resume.github.io/ (here is mine - depressing that Coldfusion features so highly in the stats though!).  Over at NerdAbility we have really taken that idea forward (incorporating other sites like BitBucket, GoogleCode, Coursera, LinkedIn, StackOverflow, etc) and is obviously something that we think is a good idea.

But its a bad idea..

A big argument against it is that it furthers the already engrained bias towards white men. If you look at the demographic of the most active GitHub profiles there is no denying the common pattern.

I agree completely - that using GitHub as a filtering mechanism or pre-requisite for a candidate sucks. You really shouldn't do that. I have had conversations with agents where they have told me that a client only wants to see candidates who contribute to OSS, and I have declined. It doesn't qualify a candidate as being half-way competent and rules out lots of very competent people who don't have spare time to work on OSS (multiple jobs, family responsibility).


Just another data point..

I guess this is really the point here. You shouldn't rule out candidates because they don't have GitHub accounts, just like you shouldn't rule out a candidate for not having a degree etc.  I think we can all agree tech recruiting is hard, and its really hard to assess whether someone is actually a good developer and not just blagging it - so a computer scientist, I'm grateful for as much data to help with this decision as possible.

It's not that a candidate with an active GitHub profile trumps one without, but it means that its another point in an interview that we can try and use to tease out  a little bit more of an insight into the candidates skills, interests, passions.

GitHub is another point on your CV (if you are fortunate enough to have the time to setup and contribute to a GitHub profile) - just like your academic achievements and career history or anything else you choose to put on there.  If you have an interesting project it can be a talking point for an interview, in much the same way an interesting role on your CV would be.  From my point of view, I love going into interviews and hearing that the interviewer has checked out my projects on GitHub, and when the inevitable "tell us about an interesting/challenging/etc project/problem you have had" comes up, it's great to be able to talk about projects on GitHub that they have seen - not just because I know the project well, but also it will inevitably be a project that I am passionate about (otherwise I wouldn't be doing it in my spare time!).


We shouldn't be demanding OSS contributions, or set online profiles, or StackOverflow credibility - but we probably shouldn't be dismissing it as irrelevant.



Securing your mobile API - Spring Security

Having recently posted an article on an approach to securing an API for a mobile app to use, here are the details of how something similar can be configured using Spring-MVC and Spring-Security (this post uses Java config - if you are un-familiar with Java config rather than XML then check out my previous post on the topic).


Basic webapp registration and authentication

The first thing we need is a basic web app that supports user registration and authentication.  I am not going to go into the details of how to build a Spring MVC app with basic security and users etc, so will assume you are familiar with basic Spring MVC architectures. For an example Spring webapp that supports user security and registration, you can check out one of my several random webprojects on GitHub.


As I am only looking at an API application - no web client right now - All I need to start with is to set up the security for registration and login:



Ok, so the above is my web security config - the configure() method is what is defining my URLs and the security - it's pretty basic, I state that anyone can access the resources (for css etc), the log-in and the sign-up and everything else should be authenticated.

The only parts of interest are:
  1. In the registerAuthentication() method, as well as the normal setting of UserServices and password encryptor, I am also adding a Remember Me Authentication provider
  2. In the configure() method on the formLogin component we are also setting a RemeberMeService
  3. We are setting @Order annotation on the class with value 2

This class basically just sets up the web part of the security, so all pretty straight forward.


API security configuration

For clarity and readability of code, I like to create separate configuration classes for the different aspects of security - Below is the security config for the API endpoints:



This file is a little more interesting, let's have a look at what is going on:
  1. First of all, we define @Order with value 1 - as you have probably guessed, this defines the order in which the security config is loaded/considered - we want the API security to kick in first
  2. In our configure() method, we first specify an ant matcher so the rules in this config only apply to URLs that start /api/
  3. We state that all requests to this pattern must be authenticated and apply a RememberMeAuthenticationFilter to run before the Basic authentication chain runs
  4. We also set the session management to be stateless - this refers to serverside sessions, as we don't want to retain server session state
  5. We then configure the Remember Me beans - this is all standard and as per the Spring guidelines here.  The only difference is that we have a custom TokenBasedRememberMeService - that we will look at later.


Security flow

Below is the basic security chain that we get as a result of the above confirmation
  1. Request comes in - we check if it matches /api/**
  2. If it does match, then all requests must be authenticated - and run the RememberMeAuthenticationFilter 
  3. The filter simply uses the RememberMeService and AuthenticationProvider to check the token/cookie value to see if it can pre-authenticate the request.
  4. If it has matched the URL but doesn't authenticate on the filter, then it will 403
  5. If the URL does not match /api/** then the request starts the second security config
  6. If the url matches the resources, sign-up, sign-in then allow the request to complete, otherwise require authentication, so (default spring behaviour) direct all other requests to the registered login page ("/")


Customising remember me

 So the good news is we have been able to re-use core Spring Security functionality so far, so no dubious hand-rolled solutions - We use the standard security chain, username/password authentication provider and Remember Me filter/provider. The only thing that we have to modify slightly is the RememberMeTokenService - the default behaviour for Remember Me assumes that the token will always be provided in a cookie (designed for auto-login websites if you tick "remember me"), and as we are extracting the token from the cookie and passing it in the request header instead, we need to modify the service class to grab it from the header.

As you can see, it's pretty simple - all we are doing is overriding the default extract cookie method so it looks in the request header instead.  I would really recommend you read through the source code for the RememberMe classes - starting with the filter - and seeing what is actually happening between the Filter, token service and auth provider, it's pretty clear to read and will help get a good understanding of what is going on.


Caveats

There are a few things to consider here:
  • This is fairly basic security - requests should be made over https to prevent man-in-the-middle attacks, alternatively look at a solution like Amazon use whereby the token isn't passed, rather it is used to hash the request body which is then checked on the server side (prevent the need for sending the user token every request)
  • This is currently open to anyone to use/build against - there aren't any checks for app key/secret etc - so you would probably want to build that in to prevent other parties building apps against the api and getting a user token in the same way.
  • I'm not a security expert - this is just my interpretation of the Google guidelines using Spring Securities RememberMe classes. This is really just a thought experiment, getting up to speed with how Spring Security works in more details and experimenting with mobile app fun. Use at your own risk..

Securing your API for mobile access

So I have been thinking about potential options regarding securing a server side API for use by mobile apps. No matter what question you ask on the stack exchange network, the answer will always come back as "you should really use oauth", which I guess we should consider as a half truth.

Having used OAuth on several occasions as a client (accessing Facebook/Twitter/LinkedIn/GitHub etc), it was naturally the first option that came to mind for securing my API.


My requirements right now are simple:
  • I am experimenting building an Android mobile app that will use a server API (webapp & API will be built using Java & Spring stack)
  • The app I am experimenting is mobile-only (no web client)
  • It is not going to become a public API (e.g. won't be supporting other apps/sites)


I guess the key point is that I only intend to expose the API to my mobile app, and not have lots of comsumers of the API - and given the complexity of implementing OAuth, it seemed like it would be overkill as a solution.  A big benefit of OAuth is that it supports client/application keys.


So I started looking around for other ideas as to how to do this - I was (and still am) curious as to how Twitter/Instagram secure their APIs for their own applications - I know that the Twitter API is secured for public use using OAuth, but the interesting question is how they secure their own application key/secret. The problem with mobile apps is that an application key needs to be stored on the device - and whilst Twitter offers severely throttled API access using OAuth (register an application, get a secret app key etc), their own apps are obviously not throttled, so it seems like it would be a challenge to keep their application key secured, as if its in the app code, then its subject to de-compilation.

Anyway, getting close to resigning to the fact that I was going to have to use OAuth I came across this one-pager from Google. The guide suggests a much simplified approach with the following basic steps:

  • Build an webapp on your server that has a login page 
  • From your mobile app, embed a WebView of the login page
  • Upon login to your app, return a cookie with a token value that can be used to log in your user on future requests - ideally re-using your frameworks Remember-Me type services so not to have to hand-roll your own
  • On every API request include the token in your request header to authenticate (again using the Remember Me services)

The fact that it was a suggested approach from Google is pretty good, and it's a tonne simpler than negotiating the OAuth dance, so I thought I would have a look at getting this setup.


This is all good - but really, the advice to use OAuth is a half-truth, as if you ever have to scale your application, and think you might one day have/want to open it up do other developers/clients then OAuth is the way to go - You won't get much love from third-party developers with a non-standard approach to securing your API.


So.. how can it be done with Spring? I will cover that in my next post later this/next week!

Everything is Technology

So, having pondered the merits and likely outcome of the UK government's Year of Code initiative, you could rightly be left to ponder, "who cares?". A lot of the UK's population may not consider themselves hugely tech savvy, and certainly not formally trained in tech, and we are getting by. The country has naturally produced enough nerds/tech folk to keep us as a respectable tech power in the modern world. Plus, kids these days are so emersed with tech/internet/mobiles they will be naturally equipped for the future world of work.

Maybe?


A little while ago, Marc Andreessen (of Silicon Valley Venture Capital firm Andreessen Horowitz) wrote an article titled "Why software is eating the world", referring to the pervasiveness of tech and how slowly, tech, and the internet, is disrupting more and more industries, changing the way they operate and where/how they compete for profit.

So the question is, will being "power users" on Facebook, CandyCrush, Twitter, etc be enough in the predicted software devoured world?

Some obvious examples of industries that are now tech industries spring to mind:
  • Amazon - completely changed the face of retail and is almost single handedly making high streets around the uk obsolete
  • iTunes et al - once again, changed the face of the music industry, record shops etc
  • AirBnB - changing the way that people think about travel and disrupting the hotel/holiday accommodation industry.


High profile and a very real, visible impact on the economy, and as Andreesen quotes in his article:
"Today, the world's largest bookseller, Amazon, is a software company—its core capability is its amazing software engine for selling virtually everything online, no retail stores necessary. On top of that, while Borders was thrashing in the throes of impending bankruptcy, Amazon rearranged its web site to promote its Kindle digital books over physical books for the first time. Now even the books themselves are software."

The important bit being "its core capability is its amazing software engine" - Amazon isn't a retail business, its a tech business - its core skills and expertise, and the majority of its employees are in tech.  The fact that it can branch out so easily into its Cloud services is testament to that. They have an amazing tech platform, with amazing tech folk, that is their business.

So that's an obvious example - we aren't all going to be working at Amazon/Google etc. But the thing is, its not just the obvious examples that are being disrupted. Let's look at an industry that is apparently yet to be disrupted: insurance. Any big online insurers coming in and turning things on their head the way amazon did retail? Nope. Look for any form of personal insurance in the UK and lots of familiar names still come up: The AA, Tesco, First Direct, Royal & Sun Alliance etc. Look a bit beneath the surface though, and you will realise the industry has really become a tech arms-race. Aggregator websites like money supermarket.com or comparethemarket.com have indirectly completely changed the industry with tech.

What was once a very personal industry, all about understanding people & risks and where an experienced actuary held the power, has become a race to the bottom in terms of latency and efficiency for quote generation.  The aggregator sites drive so much traffic to insurers (approximately 60% of all insurance quotes originate from these sites), and such high volumes (~10-11 million a month) that to stay in business, insurers have to have the tech to serve up quotes a) faster than others b) cheaper than others.

If your system costs a penny more to generate a quote than your competitor, at 11 million quotes a month, that's an extra £110,000 you need to generate every month (thats the cost just to generate the quotes - if you don't actually manage to convert any of those quotes, that's a pretty quick path to bankruptcy), which is most likely going to push up your premiums, which will in turn decrease your chance of conversion.

Honestly, with so many of these companies having to support monolithic legacy tech systems, I would be surprised if they didn't soon get replaced - all a start-up would need is financial backing to underwrite the claims, and a few decent tech folk to build a web friendly (web services, REST etc.) insurance system, built with aggregators in mind plus a slick web app to allow easy (and free) policy management, and they could probably undercut and offer better customer service than the all the big guns so severely that they would have no choice but retire.

EVERYONE, LEARN TO CODE. NOW!


No. Not really. 

Does everyone need to know how to code? Of course not. The point of introducing tech more prominently to the curriculum is obviously far from aiming to turn all of the future generations into coders. It is simply just to raise awareness/understanding of computers, and possibly slightly increase the number of coders we produce. It is after all only raising tech so its seen more alongside other key subjects like maths, science, english etc and we haven't all turned out to be mathematicians have we?

The uk has a great history of scientists, writers, artists and also computer scientists, but these days when people talk global tech Silicon Valley always comes up first. Would be good if we started to change that right?

Everything Is Everything.. Change, it comes eventually.