Thoughts on development & design

Overview

1. Connect to the Twitter streaming API (aka the firehose) filtering on world cup related key words
2. Pass the body of the tweet to AWS Comprehend to get the sentiment score
3. Update the in memory store of stats (count and average sentiment) for each country

Detecting Sentiment

Because I wanted to be able to stub out the sentiment analysis without being tied to AWS, you will notice I am using the self-type annotation on my twitter class above, which requires a SentimentModule to be passed in at construction - I am using a simple cake pattern to manage all my dependencies here. In the Github repo, there is also a Dummy implementation, that will just pick a random number for the score, so you can still see the rest of the API working - but the interesting part is the AWS integration:
Once again, the SDK makes the integration really painless - in my code I am simplifying the actual results a lot to a much cruder Positive/Neutral/Negative rating (plus a numeric score -100..100).

The AWSCredentials class is the bit that is going to look in the normal places for an AWS key.

Storing and updating our stats

So now we have our inbound tweets and a way to asses their sentiment score - I then setup a very simple akka actor to manage the state and just stored the API data in memory (if you restart the app, the store gets reset and the API stops serving data).

Again, very simple out of the box stuff for akka, but it allows easy and thread safe management of the in-memory data store. I also track a rolling list of the last twenty tweets processed, which is managed by a second, almost identical, actor.

The results

I ran the app during several games, below are some sample outputs from the API. The response from the stats API is fairly boring reading (just numbers) but the example tweets show two examples of a positive and neutral tweet correctly identified (apologies for the expletives in the tweet about Poland - I guess that fan wasn't too happy about being beaten by the Senegalese!) - you will also notice, the app captures the countries being mentioned, which exposes one flaw of the design: in the negative tweet from the Polish fan loosing two goals to Senegal, it correctly identifies the sentiment as negative, but we have no way to determine the subject - as both teams are mentioned, the app naively assigns it as a negative tweet to both of the teams, where as on reading, it is clearly negative with regards to Poland (I wasn't too concerned for my experiment, of course, just an observation worth noting).

Sample tweet from the latest API:

Sample response from the stats API:

When I finally did get around to starting to learn React, I just plugged in the APIs and paid no attention to styling, which is a round about way of apologising for the horrible appearence of the screenshot below (I'm really sorry about the css gradient)!

Updating our dependencies

Spring configuration

The Fabric platform is made of three modular kits that address some of the most common and pervasive challenges that all app developers face: stability, distribution, revenue and identity.

The XML Layout: MultiAutoCompleteTextView

SetTokeniser: UsernameTokenizer

• terminateToken(CharSequence text) - this basically just provides a presentable version of our token with a proper terminator at the end (e.g. makes sure a trailing single space in this case)
• findTokenStart(CharSequence text, int cursor) - Just finds the position of the start of the token. It does this by iterating backwards through the provided CharSequence (the text input in the input box) starting from the position of the cursor, which is just the position of the last character edited (this means if you go back and edit text in the middle of a block it still finds the correct token).  If no valid token start is found (e.g. we go backwards and we can't find a @ character) then the current position is returned - no dropdown is displayed.
• findTokenEnd(CharSequence text, int cursor) - As above, but finds the end position. Iterates forward until a token terminator (in our case a space) or the end of the text is found

Adding a text changed listener

Maven Configuration

	 spring-snapshot
	 Spring Maven Snapshot Repository
	 http://maven.springframework.org/snapshot



	 spring-milestone
	 Spring Maven Milestone Repository
	 http://maven.springframework.org/milestone

	org.springframework.social
	spring-social-web
	${spring-social.version}
	
		
			spring-core
			org.springframework
		
		
			spring-web
			org.springframework
		
		
			spring-webmvc
			org.springframework
		
	



	org.springframework.social
	spring-social-github
	${spring-social-github.version}



	org.springframework.social
	spring-social-twitter
	${spring-social-twitter.version}

Storing User Connection Details

public class HibernateUsersConnectionRepository implements UsersConnectionRepository { 

public class HibernateConnectionRepository implements ConnectionRepository {

@Entity
@Table(name = "CV_CONNECTION")
public class SocialConnection {

       @Id
       private Long id;
       private String providerId;
       private String providerUserId;
       private String displayName;
       private String profileUrl;
       private String imageUrl;
       private String accessToken;
       private String secret;
       private String refreshToken;
       private Long expireTime;
       private Integer rank;

Configuring the Beans

@Configuration
public class SocialConfiguration {

ConnectionFactoryLocator:

@Bean
public ConnectionFactoryLocator connectionFactoryLocator() {
	  ConnectionFactoryRegistry registry = new ConnectionFactoryRegistry();
	  registry.addConnectionFactory(new GitHubConnectionFactory(gitHubKey, gitHubSecret));
	  registry.addConnectionFactory(new TwitterConnectionFactory(twitterKey, twitterSecret));
	  return registry;
}

UsersConnectionRepository:

@Bean
@Scope(value = "singleton", proxyMode = ScopedProxyMode.INTERFACES)
public UsersConnectionRepository usersConnectionRepository() {
	  HibernateUsersConnectionRepository repository = new HibernateUsersConnectionRepository(connectionFactoryLocator(), Encryptors.noOpText());
	  return repository;
}

ConnectionRepository:

@Bean
@Scope(value = "request", proxyMode = ScopedProxyMode.INTERFACES)
public ConnectionRepository connectionRepository() {
	  Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
	  if (authentication == null) {
			 throw new IllegalStateException("Unable to get a ConnectionRepository: no user signed in");
	  }
	  ApplicationUser user = (ApplicationUser) authentication.getPrincipal();
	  return usersConnectionRepository().createConnectionRepository(String.valueOf(user.getAccountId()));
}

ThirdParty Interfaces:

@Bean
@Scope(value = "request", proxyMode = ScopedProxyMode.INTERFACES)
public Twitter twitter() {
	if (connectionRepository().findPrimaryConnection(Twitter.class) != null) {
		return connectionRepository().findPrimaryConnection(Twitter.class).getApi();
	}
	return null;
}

Using the Interfaces

@Autowired
private Twitter twitter;
public void postTweet(String content) {
	  twitter.timelineOperations().updateStatus(content);
}

@Override
public void onCreate(Bundle savedInstanceState) {
 System.setProperty("http.keepAlive", "false");
 super.onCreate(savedInstanceState);
 setContentView(R.layout.main_oauth);
 
 //check for saved log in details..
 checkForSavedLogin();

 //set consumer and provider on teh Application service
 getConsumerProvider();
 
 //Define login button and listener
 buttonLogin = (Button)findViewById(R.id.ButtonLogin);
 buttonLogin.setOnClickListener(new OnClickListener() {  
  public void onClick(View v) {
   askOAuth();
  }
 });
}

private void checkForSavedLogin() {
 // Get Access Token and persist it
 AccessToken a = getAccessToken();
 if (a==null) return; //if there are no credentials stored then return to usual activity

 // initialize Twitter4J
 twitter = new TwitterFactory().getInstance();
 twitter.setOAuthConsumer(CONSUMER_KEY, CONSUMER_SECRET);
 twitter.setOAuthAccessToken(a);
 ((TwitterApplication)getApplication()).setTwitter(twitter);
 
 startFirstActivity();
 finish();
}

private void askOAuth() {
 try {
  consumer = new CommonsHttpOAuthConsumer(CONSUMER_KEY, CONSUMER_SECRET);
  provider = new DefaultOAuthProvider("http://twitter.com/oauth/request_token", "http://twitter.com/oauth/access_token", "http://twitter.com/oauth/authorize");
  String authUrl = provider.retrieveRequestToken(consumer, CALLBACK_URL);
  Toast.makeText(this, "Please authorize this app!", Toast.LENGTH_LONG).show();
  setConsumerProvider();
  startActivity(new Intent(Intent.ACTION_VIEW, Uri.parse(authUrl)));
 } catch (Exception e) {
  Toast.makeText(this, e.getMessage(), Toast.LENGTH_LONG).show();
 }
}

@Override
protected void onResume() {
 super.onResume();
 if (this.getIntent()!=null && this.getIntent().getData()!=null){
  Uri uri = this.getIntent().getData();
  if (uri != null && uri.toString().startsWith(CALLBACK_URL)) {
   String verifier = uri.getQueryParameter(oauth.signpost.OAuth.OAUTH_VERIFIER);
   try {
    // this will populate token and token_secret in consumer
    provider.retrieveAccessToken(consumer, verifier);

    // Get Access Token and persist it
    AccessToken a = new AccessToken(consumer.getToken(), consumer.getTokenSecret());
    storeAccessToken(a);

    // initialize Twitter4J
    twitter = new TwitterFactory().getInstance();
    twitter.setOAuthConsumer(CONSUMER_KEY, CONSUMER_SECRET);
    twitter.setOAuthAccessToken(a);
    ((TwitterApplication)getApplication()).setTwitter(twitter);
    //Log.e("Login", "Twitter Initialised");
    
    startFirstActivity();

   } catch (Exception e) {
    //Log.e(APP, e.getMessage());
    e.printStackTrace();
    Toast.makeText(this, e.getMessage(), Toast.LENGTH_LONG).show();
   }
  }
 }
}

private String CALLBACK_URL =           "callback://tweeter";

Twitter twitter = new TwitterFactory().getInstance();
  List statuses = new ArrayList();
     try {
   statuses = twitter.getUserTimeline(screenName);
  } catch (TwitterException e) {
   Log.e("Twitter", "Error logging in to Twitter");
   Log.e("Twitter", e.getMessage());
  }

ArrayList JOBS = new ArrayList();
  try {
   if (statuses.size()>0){
    for (Status s : statuses){
     String avatar = "http://" + s.getUser().getProfileImageURL().getHost() + s.getUser().getProfileImageURL().getPath();
     JSONObject object = new JSONObject();
     object.put("tweet", s.getText());
     String timePosted = Utility.getDateDifference(s.getCreatedAt());
     object.put("tweetDate", timePosted);
     object.put("author", s.getUser().getName());
     object.put("avatar", avatar);
     object.put("userObj", s.getUser());
     object.put("tweetId", s.getId());
     
     JOBS.add(object); 
    }
   }else{
    JSONObject object = new JSONObject();
    object.put("tweet", "You have not logged in yet! Please log on to view latest tweets");
    object.put("author", "");
    JOBS.add(object);
   }

  } catch (JSONException e1) {
   Log.e("JSON", "There was an error creating the JSONObject", e1);
  }

public static String getDateDifference(Date thenDate){
        Calendar now = Calendar.getInstance();
        Calendar then = Calendar.getInstance();
        now.setTime(new Date());
        then.setTime(thenDate);

        
        // Get the represented date in milliseconds
        long nowMs = now.getTimeInMillis();
        long thenMs = then.getTimeInMillis();
        
        // Calculate difference in milliseconds
        long diff = nowMs - thenMs;
        
        // Calculate difference in seconds
        long diffMinutes = diff / (60 * 1000);
        long diffHours = diff / (60 * 60 * 1000);
        long diffDays = diff / (24 * 60 * 60 * 1000);

  if (diffMinutes<60){
   if (diffMinutes==1)
    return diffMinutes + " minute ago";
   else
    return diffMinutes + " minutes ago";
  } else if (diffHours<24){
   if (diffHours==1)
    return diffHours + " hour ago";
   else
    return diffHours + " hours ago";
  }else if (diffDays<30){
   if (diffDays==1)
    return diffDays + " day ago";
   else
    return diffDays + " days ago";
  }else {
   return "a long time ago..";
  }
 }

In theory there's not much difference between theory and practice. But in practice there is a huge difference