Creating a Twitter Search & Analytics tool

01 January, 2015 by Tom Elliott

In 2015, I launched a new Twitter Search & Analytics tool socialbearing.com. It hooks up to many of the feeds available through Twitter’s API  such as user and home timelines, public and people search and mentions, follower and friend streams.

Social Bearing Twitter Search

Social Bearing Twitter Search & Analytics

I have previously written a number of blog posts about the Twitter API including authentication and creating a custom Twitter feed for user timelines. Social Bearing has built upon a number of features that these articles have covered.

This post is intended to give a brief overview into some of the technical aspects in creating a more complex Twitter tool like Social Bearing and some of the decisions I was faced with.

Authentication: App vs User

In mid-2013, Twitter required all access to streams or ‘endpoints’ to be authenticated so one of the first things you’ll want to look at with any Twitter website or App is the authentication method.

There are two types of authentication: User and App, and both are rate limited depending on the endpoint requested. For example, the user timeline endpoint is limited to 300 requests every 15 limits for ‘App’ authentication or 180 requests every 15 minutes for ‘User’ authentication

User authentication requires visitors to your website or application to first sign in to Twitter before being redirected back to the site. Rate limits are applied on a user by user basis which is usually better for high traffic sites that may need to allow for hundreds of users and potentially thousands of requests to Twitter every few minutes. I used this excellent tutsplus tutorial to implement user sign-in without the account creation.

App based authentication on the other hand means anyone visiting your site will use the same tokens. This is usually better for lower traffic sites or individual applications that are unlikely to exceed these rate limits.

Delving into data

Anyone who sets about creating a twitter tool will be familiar with Twitter’s public API. There is so much data available here and worth checking out all the read only streams available (GET) and write based actions (POST).

When you know which stream(s) you are likely to use, you need to make an authenticated server-side request using PHP, passing the tokens and keys you have got either from the App or User. This can be quite complicated unless you use an oAuth library. I use twitteroauth by @abraham for PHP.

An example of a server side request in PHP to the GET users/show endpoint would be:

session_start();
require_once("twitteroauth/twitteroauth.php"); //Path to twitteroauth library

//Insert application or user consumer & access keys here
$consumerkey = "xxxxxxxxxxxxx";
$consumersecret = "xxxxxxxxxxx";
$accesstoken = "xxxxxxxxxxx";
$accesstokensecret = "xxxxxxxxxxxx";

if (isset($_GET['user'])) {
    $user = $_GET['user'];
}

function getConnectionWithAccessToken($cons_key, $cons_secret, $oauth_token, $oauth_token_secret) {
  $connection = new TwitterOAuth($cons_key, $cons_secret, $oauth_token, $oauth_token_secret);
  return $connection;
}

$connection = getConnectionWithAccessToken($consumerkey, $consumersecret, $accesstoken, $accesstokensecret);
$tweets = $connection->get("https://api.twitter.com/1.1/users/show.json?screen_name=".$user);
echo json_encode($tweets);

If this file were named get-user.php, you can pass the ‘user’ parameter in the URL. e.g. get-user.php?user=tomwebdev. You could do a load of stuff direct in PHP server-side to the $tweets data object if you wish, however I prefer to do most of the data manipulation and formatting within JavaScript which is why the resulting $tweets data above is echoed out in JSON format.

A simple JavaScript (jQuery) AJAX request might look something like the below:

getuserjson('tomwebdev');

function getuserjson(username) {
	var user = username;
	var url = 'get-user.php?user='+user;

	$.getJSON(url,
		function(data) {
			console.log("---- USER DATA---- ")
			console.log(data);
		});
}

This will output the following JSON data when viewed in Chrome Inspector.

chrome inspector twitter json data

JSON data for a Twitter user entity when viewed in Chrome inspector tools

I spent a lot of time exploring this data when working on Social Bearing and found pieces of information contained in tweet and user entities that are not commonly used.

Some interesting facts I learnt about Twitter data include:

  • Most of the Twitter endpoints return either user or tweet entities
  • Twitter automatically assigns a language to each tweet based on the content. This is not always correct
  • User entities can return the latest tweet under the ‘status’ node
  • Roughly 0.5 – 1% of tweets contain geo-coded data
  • Twitter have implemented a ‘possibly sensitive’ tag on tweets to allow users to tag offensive content.
  • Tweet media including images, links and videos are contained within the ‘entities’ node
  • Images in tweets not uploaded direct via Twitter such as Instagram are not separated under ‘entities’ which is why websites and applications usually don’t display instagram photos direct

Checkout www.socialbearing.com to see how some of this Twitter data can be used

 



Authenticating a Twitter Feed for OAuth API V1.1 – Timelines & streams »
Creating a Custom jQuery Twitter Feed (API v1.1) »


One Comment

  • aaron says:

    Hi Tom,

    I have been looking at your blogs, tutorials and websites. They Rock!

    aaron

  • Leave a Reply

    Your email address will not be published. Required fields are marked *

    css.php