Showing posts with label Mining the Social Web. Show all posts
Showing posts with label Mining the Social Web. Show all posts

Tuesday, December 31, 2013

Project: Mining the Social Web 2nd Edition - Part 2

This is the part 2 of my experience trying out and experimenting with the code in Mining the Soical Web 2nd Edition. This covers Chapter 1 to Chapter 3 (Twitter, Facebook, LinkedIn).

A couple of thoughts so far:

1. The author is really dedicated to his book and audience, evident in his responsiveness in answering my question (#7 below).
2. The VM experience is totally awesome.
3. Each of the topic covered can be expanded into a whole book by itself. We are scratching only the surface.
4. For me, when writing a script on a new topic, finding the right library is typically half of the battle. Nothing is more frustrating when you are already going down the path of using a library only to find a better one (or worse, nothing at all). The book gave you a nice list of best-of-breed libraries for each topic, which in itself is worth every penny of buying the book.

Here is more detail about my experience:

1. I reran the exercise I did in previous post via IPython notebook via the VM. The three top trending corresponds to the bombing in Russia and the NFL game that was on at the time:

2013-12-29T18:48:48Z
set([u'AJ Green', u'Volgograd', u'Andy Dalton'])

2.  I ran the rest of the code from the VM because I was short on time:

+----------------+-------+
| Screen Name    | Count |
+----------------+-------+
| justinbieber   |     6 |
| Kid_Charliej   |     2 |
| Cavillafuerte  |     2 |
| touchmestyles_ |     1 |
| aliceorr96     |     1 |
| gymleeam       |     1 |
| fienas         |     1 |
| nayely_1D      |     1 |
| angelchute     |     1 |
+----------------+-------+
3. I was really glad I used virtualenv to isolate each of the chapters as each of them reuqire a different Python package. There was no problem installing any of the packages and run the scripts that I tried on the Win8.1 platform.
4. Facebook Graph API is a beast. It really require more reading and understanding in order to fully grasp the depth and what you can do with it. Here is a partial list of my friend's top interest list: 

Top likes amongst friends
+----------------------------------------+------+
| Name                                   | Freq |
+----------------------------------------+------+
| Amazon.com                             |    9 |
| Python for Network Engineering         |    5 |
| George Takei                           |    4 |
| West Seattle Rolfing                   |    4 |
| Photography on Facebook                |    3 |
| Music on Facebook                      |    3 |
+----------------------------------------+------+
5. D3 looks really awesome, but there is a learning curve to it that I will need to come back to. 

6. For the first 2 LinkedIn scripts, I have decided to break the second one into a more readable list comprehension. For me, I think if the List Comprehension goes into 2-degree deep, I'd like to break them apart:

     1. First file to get the connection file:

from linkedin import linkedin
import json

consumer_key = ''
consumer_secret = ''
user_token = ''
user_secret = ''

return_url = ''

auth = linkedin.LinkedInDeveloperAuthentication(consumer_key, consumer_secret,
     user_token, user_secret, return_url,
     permissions=linkedin.PERMISSIONS.enums.values())

app = linkedin.LinkedInApplication(auth)

# Get you own profile
#print app.get_profile()

# Getting connections as a ego point
#print app.get_connections()

# A good idea to get connections and store them in file to save API calls
connections = app.get_connections()
f = open('linkedin_connections.json', 'w')
f.write(json.dumps(connections, indent=1))
f.close()

for line in open('linkedin_connections.json', 'r'):
     print line

     2. Second file to do the parsing: 

from prettytable import PrettyTable
import json

f = open('linkedin_connections.json', 'r')
connections = json.loads(f.read())
f.close()

# PrettyTable Print
pt = PrettyTable(field_names=['Name', 'Location'])
pt.align = 'l'

for c in connections['values']:
     if c.has_key('location'):
          pt.add_row([c['firstName'] + ' ' + c['lastName'], c['location']['name']])

print pt
 7. I got stuck under the CSV file export portion, turns out the path should be ipynb/resources/<blah> instead of just resources/<blah>. More details on the book Facebook page: https://www.facebook.com/MiningTheSocialWeb/posts/594068610648372?comment_id=5203748&reply_comment_id=5204030&offset=0&total_comments=16&notif_t=feed_comment.

I am enjoying the experience very much and feel like I am learning a lot. Again, the biggest difference for me is the author's dedication to his audience as well as the VM experience.

Happy coding!




Sunday, December 29, 2013

Project: Mining the Social Web 2nd Edition - Part 1

I have been meaning to read Matthew Russell's book, Mining the Social Web 2nd Edition (ISBN 1449367615), ever since it came out in October. In fact, it has been on my radar since it was in preview. I think it is a fascinating topic and I really like the first edition. The main feedback I gave him on the first edition, presumably I was not the only one, was that there were too many dependencies to overcome before getting to the meat of the content. He really took some of the reader's suggestion to heart (Udemy, IPython notebook, VM) and knock it out of the park in the second edition by providing a awesome learning / development environment with Virtual Machine + Vagrant.

It is in my intention that for the second edition, I am going to document my experience and any caveats that I experienced. A couple of notes:

  • Mac has been my preferred platform for the last few years, but I am trying to get familiar with Windows 8.1 and recently got a Surface Pro 2. So I am going to conduct the whole project on the device, including writing the blogs.
  • I will try to do as much coding on my own machine as possible to gain muscle memory and only use the VM to compare results and when I get stuck. But I want to stress that I think having this nice frozen development VM is so cool and priceless.
  • I already have Enthought's Canopy Express installed and that is my default Python environment. I find that the least painful way to get IPython, Matplotlib, SciPy, and NumPy on a Windows box. 

Ok, enough talking, let's get started.

Step 1. Prep the VM Experience

Following the instruction on this page, I'd imagine it is pretty painless for most. I, however, ran into an issue with an error I received from Vagrant about 'VT-x not enabled' and therefore VirtualBox VM cannot be started. A bit of research yield this blog post about running Hyper-V and VirtualBox at the same time (http://derekgusoff.wordpress.com/2012/09/05/run-hyper-v-and-virtualbox-on-the-same-machine/). Hyper-V came with Windows 8.1 Pro, but I don't think it is enabled by default. It is probably worth knowing that this is an issue, however. Also a note that using PowerShell to execute the 'bcdedt' boot manager command did not work, even in administrative mode. A regular command line with admin right was needed for me.

It took longer than expected but at last, it is up for me:















Step 2. Chapter 1 Twitter

For isolation on the experiment code, I am creating VirtualEnv, more information on my steps here.

> easy_install-2.7.exe pip
> pip-2.7.exe install virtualenv
Downloading/unpacking virtualenv
  Downloading virtualenv-1.10.1.tar.gz (1.3MB): 1.3MB downloaded
  Running setup.py egg_info for package virtualenv
<skip>
Successfully installed virtualenv
Cleaning up...
>
Starting to environment, I also change the PS execution policy: 
> Set-ExecutionPolicy RemoteSigned

Execution Policy Change
The execution policy helps protect you from scripts that you do not trust. Changing the execution policy might expose
you to the security risks described in the about_Execution_Policies help topic at
http://go.microsoft.com/fwlink/?LinkID=135170. Do you want to change the execution policy?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): y
>
> virtualenv-2.7.exe twitter
Using base prefix \\Enthought\\Canopy\\App\\appdata\\canopy-1.1.0.1371.win-x86_64'
New python executable in twitter\Scripts\python.exe
Installing Setuptools..............................................................................................
...................................................................................................................
.......................done.
Installing Pip.....................................................................................................
...................................................................................................................
.....................................................................................................done.
> .\twitter\Scripts\activate.ps1
(twitter) >
(twitter) >
Step 3. Create Twitter developer account. No surprise here. 

Step 4. Finally, some code and result.
After a few hours from starting, I finally can start writing code. Here I combine the exercise 1.1 to 1.4 and received the current and world trend at this time (UFC168 was the trending topic tonight): 

import twitter
import json

consumer_key = ''
consumer_secret = ''
oauth_token = ''
oauth_token_secret = ''

auth = twitter.oauth.OAuth(oauth_token, oauth_token_secret, consumer_key, consumer_secret)

twitter_api = twitter.Twitter(auth=auth)

#print twitter_api

# Yahoo GeoID: http://developer.yahoo.com/geo/geoplanet/

world_woe_id = 1
us_woe_id = 23424977

# same api call as https:// api.twitter.com/1.1/trends/place.json?id=1
world_trend = twitter_api.trends.place(_id=world_woe_id)
us_trend = twitter_api.trends.place(_id=us_woe_id)

#print json.dumps(world_trend, indent=1)
#print ("*****")
#print json.dumps(us_trend, indent=1)

world_trend_set = set([trend['name'] for trend in world_trend[0]['trends']])
us_trend_set = set([trend['name'] for trend in us_trend[0]['trends']])

common_trends = world_trend_set.intersection(us_trend_set)

print common_trends
Here is the result: 
> python .\Ex1-1.py
set([u'Uriah Hall', u'Travis Browne', u'#UFC168'])
>
Wow, that was fun! Even just 20 pages into the book, I can tell this is going to be an enjoyable experience, I only wish I have more time to experiment.
Stay tuned. Happy coding.