Wednesday, July 2, 2014

Small Script to Fetch Bing search results

I have been playing with Tableau software for some data visualization practices. Thought it might be interesting if I can pull some Internet data. Looked around for some examples, and found a good one on http://www.guguncube.com/2771/python-using-the-bing-search-api. Unfortunately it seems that Bing changed its base URL and some of the minor formatting. Some quick edits was all it took to get the script up and running.

It is very nice that Bing gives you 50,000 queries per month for free. You can get your API key here, https://datamarket.azure.com/.

Here is the script:

#!/usr/bin/env python
# Modified from http://www.guguncube.com/2771/python-using-the-bing-search-api
#
# by Eric Chou
# eric@pythonicneteng.com

import urllib
import urllib2
import json
def main():
    query = "Python Newtork Engineering"
    response = bing_search(query, limit=10)
    #Raw print json.loads(response)['d']['results']
    #Title
    print "The top 10 results for your query " + query + ":"
    for i in json.loads(response)['d']['results']:
        print i['Title']

def bing_search(query, limit=5, **kwargs):
    key = 'YOURKEY'
    baseURL = 'https://api.datamarket.azure.com/Bing/SearchWeb/'
    limit = str(limit)
    query = urllib.quote(query)
    # create credential for authentication
    user_agent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; FDM; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 1.1.4322)'
    credentials = (':%s' % key).encode('base64')[:-1]
    auth = 'Basic %s' % credentials
    url = baseURL+'Web?Query=%27'+query+'%27&$top='+limit+'&$format=json'
    request = urllib2.Request(url)
    request.add_header('Authorization', auth)
    request.add_header('User-Agent', user_agent)
    request_opener = urllib2.build_opener()
    response = request_opener.open(request) 
    response_data = response.read()
    return response_data
if __name__ == "__main__":
    main()
Here is the output of the script:

$ python bingSearch_v1.py
The top 10 results for your query Python Newtork Engineering:
Python for Network Engineering | Facebook
Python for Network Engineers
Python for Network Engineers
Jeremy Filliben: Python Programming and the Network Engineer
Show 176 - Intro to Python & Automation for Network ...
Python For Network Engineers | Networking-Link
First Post! | Python for Network Engineers
Python For Scientific Engineering - OpenOpt
World News - Python Engineer
Network Test Engineer Python Jobs, Employment | Indeed.com

Happy Coding!

9 comments:

  1. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a .Net developer learn from .Net Core Training in Chennai. or learn thru .Net Core Training in Chennai. Nowadays Dot Net has tons of job opportunities on various vertical industry.
    or Es6 Training in Chennai. Nowadays JavaScript has tons of job opportunities on various vertical industry.

    ReplyDelete
  2. Wow what a Great Information about World Day its very nice informative post. thanks for the post. pandey baba

    ReplyDelete
  3. Try not to influence it to look all dolled into, in light of the fact that such bits of writing are taken to need in some other way. There are strict film content rules. script doctor

    ReplyDelete
  4. First You got a great blog .I will be interested in more similar topics. i see you got really very useful topics, i will be always checking your blog thanks. all in one search

    ReplyDelete
  5. Hi, am a big follower of your blog. I am really happy to found such a helpful and fascinating post.Keep it up. CFA Audit | Fixed Assets Audit

    ReplyDelete
  6. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Front end developer learn from Javascript Training in Chennai . or learn thru Javascript Training in Chennai. Nowadays JavaScript has tons of job opportunities on various vertical industry. JavaScript Training in Chennai

    ReplyDelete
  7. Your blog provided us with valuable information to work with. Each & every tips of your post are awesome. Thanks a lot for sharing. Keep blogging, digital agency

    ReplyDelete
  8. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. Keep in blogging. Duplicate Payment Review | Continuous Transaction Monitoring | Duplicate Payment Recovery

    ReplyDelete
  9. Thanks for another wonderful post. Where else could anybody get that type of info in such an ideal way of writing? buy backlinks to your site

    ReplyDelete