How to Mine Twitter in Under One Hour

There are a number of books discussing using Python to mine the social web. After trying out Facebook, LinkedIn, and Twitter API's, it is my believe that Twitter provides the most friendly workflow for anybody to get started. For the past two days I have been working on a project where mining the Twitter feed was a part of the overall process. Here I want to briefly share some of the steps I took using Python. I believe the whole process shouldn't take you more than 1 hour.

1. Determine the Twitter module to use.

There were a number of Python Twitter modules, a simple search turns up two popular modules: Python-Twitter and Tweepy. Since I had the chance to listen to Elizabeth Uselton's PyCon 2016 Talk a few weeks back at a local Python meetup, I decided to use the same library, which was python-twitter.

[Update 9/5/2016] Make sure you use 'python-twitter' package, i.e. pip install python-twitter. Not pip install twitter.

2. Register a developer account at https://dev.twitter.com/ for documentation and create an app via https://apps.twitter.com/ to get all the keys for your program.

3. Follow the instruction here to pip install the module and give it a spin. Below is an example using the Python REPL (off my Raspberry Pi 3, mind you :)).

pi@raspberrypi:~/Alexa $ python

Python 2.7.9 (default, Mar  8 2015, 00:52:26)

[GCC 4.9.2] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import twitter, pprint, json

>>> api = twitter.Api(consumer_key="<you get this from the twitter app>",

...                   consumer_secret="<you get this from the twitter app>",

...                   access_token_key="<you get this from the twitter app>",

...                   access_token_secret="<you get this from the twitter app>")

>>> print(api.VerifyCredentials())

{"created_at": "Thu Dec 18 17:35:54 +0000 2008", "description": "Network Automation Nerds", "favourites_count": 16, "followers_count": 58, "friends_count": 225, "name": "ericchou", "profile_background_color": <skip>}

>>>

4. Cool, let's do some search. Since the Olympics is going on, I bet that is a hot topic:

>>> searchResult = api.GetSearch(term="Olympics", lang='en', result_type='recent', count=5, max_id='')

>>> pprint.pprint(searchResult)

[Status(ID=763393850215960577, ScreenName=Brezshun, Created=Wed Aug 10 15:17:31 +0000 2016, Text=u'RT @DragonflyJonez: Even Hitler sent Jesse Owens a nondescript, mass mailing thank you card for participating in the Olympics. FDR never ev\u2026'),

 Status(ID=763393849964294144, ScreenName=JeremyMcDoniell, Created=Wed Aug 10 15:17:31 +0000 2016, Text=u'RT @br_uk: Michael Phelps has won more men\u2019s swimming #Gold medals than all but two countries in the history of the #Olympics \U0001f64c https://t.c\u2026'),

 Status(ID=763393849930584065, ScreenName=alyajaafar, Created=Wed Aug 10 15:17:31 +0000 2016, Text=u'RT @BBAnimaIVids: Kitten Summer Olympics \U0001f63a https://t.co/EGUWnOrGsX'),

 Status(ID=763393849536372736, ScreenName=ilhamfachrul, Created=Wed Aug 10 15:17:31 +0000 2016, Text=u'RT @BBCSport: Watch live @BBCOne (UK only) and the @BBCSport website.\n\nhttps://t.co/b87rfBg0qt #RioOlympics2016  https://t.co/767BAdAmza'),

 Status(ID=763393848747888640, ScreenName=JustLandlords, Created=Wed Aug 10 15:17:31 +0000 2016, Text=u'RT @NewsLandlords: Have the London Olympics had a long-lasting effect on the property market? https://t.co/QolbtKou6a #London #property htt\u2026')]

>>>

5. There you have it, quick and simple. Here is the official API Doc from Twitter.

Happy Coding!