Wednesday, July 2, 2014

Small Script to Fetch Bing search results

I have been playing with Tableau software for some data visualization practices. Thought it might be interesting if I can pull some Internet data. Looked around for some examples, and found a good one on http://www.guguncube.com/2771/python-using-the-bing-search-api. Unfortunately it seems that Bing changed its base URL and some of the minor formatting. Some quick edits was all it took to get the script up and running.

It is very nice that Bing gives you 50,000 queries per month for free. You can get your API key here, https://datamarket.azure.com/.

Here is the script:

#!/usr/bin/env python
# Modified from http://www.guguncube.com/2771/python-using-the-bing-search-api
#
# by Eric Chou
# eric@pythonicneteng.com

import urllib
import urllib2
import json
def main():
    query = "Python Newtork Engineering"
    response = bing_search(query, limit=10)
    #Raw print json.loads(response)['d']['results']
    #Title
    print "The top 10 results for your query " + query + ":"
    for i in json.loads(response)['d']['results']:
        print i['Title']

def bing_search(query, limit=5, **kwargs):
    key = 'YOURKEY'
    baseURL = 'https://api.datamarket.azure.com/Bing/SearchWeb/'
    limit = str(limit)
    query = urllib.quote(query)
    # create credential for authentication
    user_agent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; FDM; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 1.1.4322)'
    credentials = (':%s' % key).encode('base64')[:-1]
    auth = 'Basic %s' % credentials
    url = baseURL+'Web?Query=%27'+query+'%27&$top='+limit+'&$format=json'
    request = urllib2.Request(url)
    request.add_header('Authorization', auth)
    request.add_header('User-Agent', user_agent)
    request_opener = urllib2.build_opener()
    response = request_opener.open(request) 
    response_data = response.read()
    return response_data
if __name__ == "__main__":
    main()
Here is the output of the script:

$ python bingSearch_v1.py
The top 10 results for your query Python Newtork Engineering:
Python for Network Engineering | Facebook
Python for Network Engineers
Python for Network Engineers
Jeremy Filliben: Python Programming and the Network Engineer
Show 176 - Intro to Python & Automation for Network ...
Python For Network Engineers | Networking-Link
First Post! | Python for Network Engineers
Python For Scientific Engineering - OpenOpt
World News - Python Engineer
Network Test Engineer Python Jobs, Employment | Indeed.com

Happy Coding!

20 comments:

  1. Wow what a Great Information about World Day its very nice informative post. thanks for the post. pandey baba

    ReplyDelete
  2. Try not to influence it to look all dolled into, in light of the fact that such bits of writing are taken to need in some other way. There are strict film content rules. script doctor

    ReplyDelete
  3. First You got a great blog .I will be interested in more similar topics. i see you got really very useful topics, i will be always checking your blog thanks. all in one search

    ReplyDelete
  4. Hi, am a big follower of your blog. I am really happy to found such a helpful and fascinating post.Keep it up. CFA Audit | Fixed Assets Audit

    ReplyDelete
  5. Your blog provided us with valuable information to work with. Each & every tips of your post are awesome. Thanks a lot for sharing. Keep blogging, digital agency

    ReplyDelete
  6. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. Keep in blogging. Duplicate Payment Review | Continuous Transaction Monitoring | Duplicate Payment Recovery

    ReplyDelete
  7. Thanks for another wonderful post. Where else could anybody get that type of info in such an ideal way of writing? buy backlinks to your site

    ReplyDelete
  8. I am all that much satisfied with the substance you have specified. I needed to thank you for this extraordinary article. lesmeilleursvpn.com

    ReplyDelete
  9. This is a smart blog. I mean it. You have so much knowledge about this issue, and so much passion. You also know how to make people rally behind it, obviously from the responses. https://internetprivatsphare.ch/

    ReplyDelete
  10. Awesome things you've generally imparted to us. Simply continue written work this sort of posts.The time which was squandered in going for educational cost now it can be utilized for studies.Thanks more info

    ReplyDelete
  11. This is really great work. Thank you for sharing such a good and useful information here in the blog for students.  vpnveteran.com

    ReplyDelete
  12. I would state, you do the genuinely amazing.This substance is made to a wonderful degree well. internet

    ReplyDelete
  13. Much appreciated such a great amount for this data. I need to tell you I agree on a few of the focuses you make here and others might require some further survey, however I can see your perspective. lemigliori vpn

    ReplyDelete
  14. Thanks for sharing the post.. parents are worlds best person in each lives of individual..they need or must succeed to sustain needs of the family. link building

    ReplyDelete
  15. Thank you for sharing this great post, I am very impressed with your post, the information provided is meticulous and easy to understand. I will regularly follow your next post.

    ReplyDelete
  16. useful information on topics that plenty are interested on for this wonderful post.Admiring the time and effort you put into your b!.. marketing1on1.com/professional-seo-services-company

    ReplyDelete
  17. Then you do a little website analysis to see which headline, price, offer, etc. gives you the best response. As you tweak each element of your site, you improve sales little by little. acu ac uk

    ReplyDelete