Tuesday, August 23, 2016

Curlify Your Python Requests Objects

I love things that are simple and useful. Egor's Curlify Python module is one of these gems I saw this morning.

I already have a Python script that demonstrate A10 Network's AXAPI using the Requests package:

https://github.com/a10networks/tps-scripts/blob/master/axapi_example.py

With this package, I changed exactly one line of code wherever I wanted to give an curl example of the same operation.

For example, after importing the curlify file, instead of printing out r.content in line 23, I just print out curlify.to_curl(r.reqeust) and boom, I have a curl command I can pass to people! How cool is that?

7 import requests, json, curlify
 17 # Acquire athorization token
 18 auth_headers = {'content-type': 'application/json'}
 19 auth_payload = {"credentials": {"username": "admin", "password": "a10"}}
 20 auth_endpoint = '/axapi/v3/auth'
 21 url = base_url + auth_endpoint
 22 r = requests.post(url, data=json.dumps(auth_payload), headers=auth_headers, verify=Fa    lse)
 23 print(curlify.to_curl(r.request))

Of course, you get the most out of it if you use Requests already (who isnt?) and you can just write it in Requests. When somebody else wants a simpler operation, say using just curl, that you can just easily dish it out to them in less than 5 minutes.

This is my final output from the same Python file above:
https://github.com/a10networks/tps-scripts/blob/master/axapi_curl_example.txt

The only different from the plain output is I added a '-k' to the curl operation because I have a self-signed certificate on the A10 Thunder TPS.

Nice tool, nice output, no sweat. My type of thing. :)





Monday, August 15, 2016

How-To Enable Bash on Windows 10 in 5 Steps

I don't know about you, but it really feels like a new day in technology when I see a bash shell running naively on Windows!





Windows 10 Anniversary Update has been out for almost 2 weeks now (update 1607), and by now it is relatively stable. Any major flaw should have been discovered by now, note that it is still update-at-your-own-risk kind of a thing, but I have decided to go for it since I just want to see Bash shell running on Windows natively.

Here are the steps I took (from PCWorld article):

1. Check for Windows Update. If the updater does not automatically prompt for update 1607, click on 'Learn More' on the same page and follow the link to get the executable for the update. That is what I had to do:



2. Enable Developer Mode in Settings:







4. In control panel, enable Windows Bash Shell feature:





5. Reboot, search for Bash and it will prompt you for the rest of the installation. Of course the first hing I do is to test out Python2 and Python3.







Leave me a comment or question below, let me know what you think! Happy Coding!





Wednesday, August 10, 2016

How to Mine Twitter in Under One Hour

There are a number of books discussing using Python to mine the social web. After trying out Facebook, LinkedIn, and Twitter API's, it is my believe that Twitter provides the most friendly workflow for anybody to get started. For the past two days I have been working on a project where mining the Twitter feed was a part of the overall process. Here I want to briefly share some of the steps I took using Python. I believe the whole process shouldn't take you more than 1 hour.

1. Determine the Twitter module to use.

There were a number of Python Twitter modules, a simple search turns up two popular modules: Python-Twitter and Tweepy. Since I had the chance to listen to Elizabeth Uselton's PyCon 2016 Talk a few weeks back at a local Python meetup, I decided to use the same library, which was python-twitter.

2. Register a developer account at https://dev.twitter.com/ and create an app to get all the keys for your program.

3. Follow the instruction here to pip install the module and give it a spin. Below is an example using the Python REPL (off my Raspberry Pi 3, mind you :)).

pi@raspberrypi:~/Alexa $ python
Python 2.7.9 (default, Mar  8 2015, 00:52:26)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import twitter, pprint, json
>>> api = twitter.Api(consumer_key="<you get this from the twitter app>",
...                   consumer_secret="<you get this from the twitter app>",
...                   access_token_key="<you get this from the twitter app>",
...                   access_token_secret="<you get this from the twitter app>")
>>> print(api.VerifyCredentials())
{"created_at": "Thu Dec 18 17:35:54 +0000 2008", "description": "Network Automation Nerds", "favourites_count": 16, "followers_count": 58, "friends_count": 225, "name": "ericchou", "profile_background_color": <skip>}
>>>

4. Cool, let's do some search. Since the Olympics is going on, I bet that is a hot topic:

>>> searchResult = api.GetSearch(term="Olympics", lang='en', result_type='recent', count=5, max_id='')
>>> pprint.pprint(searchResult)
[Status(ID=763393850215960577, ScreenName=Brezshun, Created=Wed Aug 10 15:17:31 +0000 2016, Text=u'RT @DragonflyJonez: Even Hitler sent Jesse Owens a nondescript, mass mailing thank you card for participating in the Olympics. FDR never ev\u2026'),
 Status(ID=763393849964294144, ScreenName=JeremyMcDoniell, Created=Wed Aug 10 15:17:31 +0000 2016, Text=u'RT @br_uk: Michael Phelps has won more men\u2019s swimming #Gold medals than all but two countries in the history of the #Olympics \U0001f64c https://t.c\u2026'),
 Status(ID=763393849930584065, ScreenName=alyajaafar, Created=Wed Aug 10 15:17:31 +0000 2016, Text=u'RT @BBAnimaIVids: Kitten Summer Olympics \U0001f63a https://t.co/EGUWnOrGsX'),
 Status(ID=763393849536372736, ScreenName=ilhamfachrul, Created=Wed Aug 10 15:17:31 +0000 2016, Text=u'RT @BBCSport: Watch live @BBCOne (UK only) and the @BBCSport website.\n\nhttps://t.co/b87rfBg0qt #RioOlympics2016  https://t.co/767BAdAmza'),
 Status(ID=763393848747888640, ScreenName=JustLandlords, Created=Wed Aug 10 15:17:31 +0000 2016, Text=u'RT @NewsLandlords: Have the London Olympics had a long-lasting effect on the property market? https://t.co/QolbtKou6a #London #property htt\u2026')]
>>>

5. There you have it, quick and simple. Here is the official API Doc from Twitter.

Happy Coding!









Tuesday, August 2, 2016

Amazon Alexa - Hello World App

Ok, so if you followed by last post, you know I was trying to get more familiar with Amazon Alexa / Echo but found Lambda to be too difficult. Did I mention Amazon is throwing all kinds of resources behind their newest toy? Oh yeah, there was a free Alexa training from Big Nerd Ranch from an ad that pushed directly in my face on Facebook. For those who does not know, Big Nerd Ranch authored some of the best selling training books on iOS and Android. So I signed up.

Truth be told, out of respect to their IP, I should not post their content here. But I can say that they helped me understand some of the basic building blocks of building a Skill. You can probably search around online and get the same material for free, and I have a feeling they will post the lessons online at some point.

The biggest surprise for me was that you should build the Lambda function and the Alexa Skill at the same time, also you don't need to publish either in order to start testing. Here are two screenshots and screencast of the hello world app I build. You can see that neither was in a 'published' state but they were happily interacting with each other and my EchoSim.








Stay tune for my further progress on playing with this new interface. I am having so much fun that I even ordered a Echo Tap. :)

Happy Coding! 







Tuesday, July 26, 2016

Introduction to Conda Workshop

I went thru the Conda Workshop this past Saturday, July 23rd sponsored by Puget Sound Programming Python Group and FlowRoute. The course was taught by Jasmine Sandhu and Chris Barker. Jasmine talked about the basics of the Conda package management system while Chris talked about Conda Recipes.

While both topics are very interesting, the introduction to Conda served one step further from my previous understanding of this tool. To be honest, I have been using Virtualenv and Pip most of the time mostly because of familiarities, but this workshop game me some new perspectives about the tool.

The introductory session notebook link below, it is worth a read if you are interested in what the tool is about:
https://github.com/sandhujasmine/CondaWorkshop/blob/master/IntroConda.ipynb

Basically, I think there are three reasons to use Conda over virtualenv/pip:

1. Separation of the binary packages such as NumPy.
2. It is both a package manager (pip) and environment manager (virtualenv). Therefore you can have one file that takes care of environment separation and package installation (think requirements.txt).
3. Channels. It is like rolling a GitHub repo natively into Python, available both in public (free) and private (paid) repo.

Here are some pictures from the session (I particularly liked the phone booth):


I wont repeat anything that is in the IPython notebook, you should check out that link since she is the expert. However, here is a sample workflow that I worked on (flask-ask):

1. Create Env

MacBook-Air:Alexa echou$ conda create -n flask-ask-temp1 python=2.7
Fetching package metadata .......
Solving package specifications: ..........

Package plan for installation in environment /Users/echou/anaconda/envs/flask-ask-temp1:

The following NEW packages will be INSTALLED:

    openssl:    1.0.2h-1
    pip:        8.1.2-py27_0
    python:     2.7.12-1
    readline:   6.2-2
    setuptools: 23.0.0-py27_0
    sqlite:     3.13.0-0
    tk:         8.5.18-0
    wheel:      0.29.0-py27_0
    zlib:       1.2.8-3

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|########################################################################| 100%
#
# To activate this environment, use:
# $ source activate flask-ask-temp1
#
# To deactivate this environment, use:
# $ source deactivate
#
MacBook-Air:Alexa echou$

2. Activate Environment

MacBook-Air:Alexa echou$ source activate flask-ask-temp1

3. Install a conda package

(flask-ask-temp1) MacBook-Air:Alexa echou$ conda install numpy
Fetching package metadata .......
Solving package specifications: ..........

Package plan for installation in environment /Users/echou/anaconda/envs/flask-ask-temp1:

The following NEW packages will be INSTALLED:

    mkl:   11.3.3-0
    numpy: 1.11.1-py27_0

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|########################################################################| 100%
(flask-ask-temp1) MacBook-Air:Alexa echou$

4. Install a package that is NOT available thru Conda:

(flask-ask-temp1) MacBook-Air:Alexa echou$ pip install flask-ask
Collecting flask-ask
Collecting PyYAML (from flask-ask)
Collecting six (from flask-ask)
  Using cached six-1.10.0-py2.py3-none-any.whl

5. Testing

(flask-ask-temp1) MacBook-Air:Alexa echou$ python
Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:43:17)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from flask_ask import Ask
>>> dir()
['Ask', '__builtins__', '__doc__', '__name__', '__package__']
>>> exit()
(flask-ask-temp1) MacBook-Air:Alexa echou$

6. Deactivate

(flask-ask-temp1) MacBook-Air:Alexa echou$ source deactivate
MacBook-Air:Alexa echou$

7. Delete the Env

(flask-ask-temp1) MacBook-Air:Alexa echou$ source deactivate
MacBook-Air:Alexa echou$ conda remove -n flask-ask-temp1 --all

Package plan for package removal in environment /Users/echou/anaconda/envs/flask-ask-temp1:

The following packages will be REMOVED:

    mkl:        11.3.3-0
    numpy:      1.11.1-py27_0
    openssl:    1.0.2h-1
    pip:        8.1.2-py27_0
    python:     2.7.12-1
    readline:   6.2-2
    setuptools: 23.0.0-py27_0
    sqlite:     3.13.0-0
    tk:         8.5.18-0
    wheel:      0.29.0-py27_0
    zlib:       1.2.8-3

Proceed ([y]/n)? Y

Unlinking packages ...
[      COMPLETE      ]|########################################################################| 100%
MacBook-Air:Alexa echou$

There you have it, one step further to manage your Python work environment.

Leave me a comment for your thoughts! Happy Coding!





Monday, July 18, 2016

The Making of Network Automation Nerds Pre-Flight Podcast

When I was at PyCon 2016, I had the chance to talk to Michael Kennedy from Talk Python To Me a few times. I have been enjoying his podcast for a while now, even was part of the panel on people who recently gotten a job as Python developers.

I tried to pick his brain about his journey and experience. The conversations were really enjoyable, we shared a common interest in Python and building life-style business (think Tim Ferriss of 4-Hour Work Week). Of course, he is waaaay ahead of me as far as progress. :)

One of the things he mentioend was how Podcasting had helped him in building up his audience and that I might give it a shot. After giving it some thoughts, I thought, 'Why not?' It wouldn't hurt to at least learn about the workflow as long as the upfront investment is not that much. Sure enough, after a day or two of planning, I was able to submit and received the approval for my Podcast from iTunes (because who else is out there, really?):

Network Automation Nerds Podcast

"THE" most useful resource that I found, was this tutorial from PodcastAnswerMan as well as this step-by-step-guide from SmartPassiveIncome. I wont repeat his instructions because he does a much better job at his methods. The videos are short and enjoyable so highly recommended that, if interested, go thru them in one setting and drill down on the parts you might need more help on.

Personally, I chose to picked the cheapest way forward:

1. The $2.99 iPhone app for recording audio file.
2. This microphone that was recommended by SmartPassiveIncome on the link above. It is a bit pricey at $79 but I figure I will be able to reuse them for other projects as well.
3. My audio file was put on my a web server that I already have.
4. The initial RSS feed was done via Blogger, which was free. As recommended by PodCastAnswerMan, this can be changed if you have a bigger audience later on.
5. The middle feed was done on Feedburner, combine with step #4, this provides the most flexibility to bootstrap the podcast quickly as well as expansion opportunity if your audience grow later on.

Overall, I was surprised at how easy the whole process was. All for less than $100. Even with the equipment I bought they can be reused later on. I love the fact that we are now all self-branded and we can deliver our message to the biggest audience possible as long as they offer value to others.

Now comes the hard part, offer value to people who give the Podcast a shot and offer them incredible amount of value!

Leave me a comment if you have any questions about the process of making this go live. :)



Tuesday, July 5, 2016

Developing Ansible Modules Troubleshooting Steps

In this post I wanted to show my troubleshooting process using print statements and interactive prompt for small programs. As network engineers starts to write code, there are inevitably times when something that does not go as planned and you need to troubleshoot. For small programs, simple print statements should be able to do the job.

I am going thru the Ansible tutorial for developing modules, http://docs.ansible.com/ansible/developing_modules.html#tutorial. The second iteration of timetest.py seems to have contained some errors. The file as-is copied and pasted into the file, timetest.py simply printed out the same result as the original timetest.py.

echou-a10:playbooks echou$ ansible/hacking/test-module -m ./timetest.py -a "time=\"March 14 12:23\""
* including generated source, if any, saving to: /Users/echou/.ansible_module_generated
* this may offset any line numbers in tracebacks/debuggers!
***********************************
RAW OUTPUT
{"time": "2016-06-29 17:48:16.553883"}


***********************************
PARSED OUTPUT
{
    "time": "2016-06-29 17:48:16.553883"
}
echou-a10:playbooks echou$

Which is clearly not the desired result indicated on the tutorial:
{"changed": true, "time": "2012-03-14 12:23:00.000307"}

Upon some troubleshooting, it seems that the syntax "WANT_JSON", even commented out in the file and not in the -a syntax, causes the args interpreted to be JSon.

 22
 23 arguments = shlex.split(args_data)
 24
 25 print("args_data: ", args_data, "arguments: ", arguments)
 26
 27 for arg in arguments:
 28     print("arg: ", arg)
 29     # ignore any arguments without an equals in it
 30     if "=" in arg:

***********************************
RAW OUTPUT
('args_data: ', '{"time": "March 14 12:23"}', 'arguments: ', ['{time:', 'March 14 12:23}'])

If you take out that line, the args_data will now be the string as it was explained in the tutorial and enter into the loop. However, it is now giving me the error:

***********************************
RAW OUTPUT
{"msg": "failed setting the time", "failed": true}


***********************************
PARSED OUTPUT
{
    "failed": true,
    "msg": "failed setting the time"
}

Upon reading the file, it looks like this error message is generated under the condition where rc != 0. Therefore if we insert a print statement in line 44:

 43             rc = os.system("date -s \"%s\"" % value)
 44             print("rc: ", rc, "value: ", value)

the output shows:

RAW OUTPUT
('rc: ', 256, 'value: ', 'March 14 12:23')

A quick interactive prompt test:

echou-a10:playbooks echou$ python
Python 2.7.11 |Anaconda 2.5.0 (x86_64)| (default, Dec  6 2015, 18:57:58)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import os
>>> value = 'March 14 12:23'
>>> rc = os.system("date -s \"%s\"" % value)
date: illegal option -- s
usage: date [-jnu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
>>> rc
256
>>> exit()
echou-a10:playbooks echou$

Which gave me an error indicating 'date' command under OS-X does not have a '-s' option. Quickly trying it out on a Ubuntu machine shows that even when the '-s' option is available, the process will still return a non-zero value because the date command cannot be executed (need to be root, also I dont want to execute the script as root that sets my date).

echou@a10-ubuntu3:~$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> value = 'March 14 12:23'
>>> rc = os.system("date -s \"%s\"" % value)
date: cannot set date: Operation not permitted
Mon Mar 14 12:23:00 PDT 2016
>>> rc
256
>>> exit()
echou@a10-ubuntu3:~$

At this point, it is probably easier to force the value of rc to be 0:

 42
 43             #rc = os.system("date -s \"%s\"" % value)
 44             rc = 0
 45

Now the output is good:

echou-a10:playbooks echou$ ansible/hacking/test-module -m ./timetest.py -a "time=\"March 14 12:23\""
* including generated source, if any, saving to: /Users/echou/.ansible_module_generated
***********************************
RAW OUTPUT
{"changed": true, "time": "2016-06-29 18:57:06.625650"}


***********************************
PARSED OUTPUT
{
    "changed": true,
    "time": "2016-06-29 18:57:06.625650"
}
echou-a10:playbooks echou$

I just wanted to show that the process that I go thru for troubleshooting. Everybody go about the troubleshooting process a little differently, this is just my process that can hopefully help other network engineers who are perhaps newer to Python.


Happy coding.