Tuesday, July 26, 2016

Introduction to Conda Workshop

I went thru the Conda Workshop this past Saturday, July 23rd sponsored by Puget Sound Programming Python Group and FlowRoute. The course was taught by Jasmine Sandhu and Chris Barker. Jasmine talked about the basics of the Conda package management system while Chris talked about Conda Recipes.

While both topics are very interesting, the introduction to Conda served one step further from my previous understanding of this tool. To be honest, I have been using Virtualenv and Pip most of the time mostly because of familiarities, but this workshop game me some new perspectives about the tool.

The introductory session notebook link below, it is worth a read if you are interested in what the tool is about:
https://github.com/sandhujasmine/CondaWorkshop/blob/master/IntroConda.ipynb

Basically, I think there are three reasons to use Conda over virtualenv/pip:

1. Separation of the binary packages such as NumPy.
2. It is both a package manager (pip) and environment manager (virtualenv). Therefore you can have one file that takes care of environment separation and package installation (think requirements.txt).
3. Channels. It is like rolling a GitHub repo natively into Python, available both in public (free) and private (paid) repo.

Here are some pictures from the session (I particularly liked the phone booth):


I wont repeat anything that is in the IPython notebook, you should check out that link since she is the expert. However, here is a sample workflow that I worked on (flask-ask):

1. Create Env

MacBook-Air:Alexa echou$ conda create -n flask-ask-temp1 python=2.7
Fetching package metadata .......
Solving package specifications: ..........

Package plan for installation in environment /Users/echou/anaconda/envs/flask-ask-temp1:

The following NEW packages will be INSTALLED:

    openssl:    1.0.2h-1
    pip:        8.1.2-py27_0
    python:     2.7.12-1
    readline:   6.2-2
    setuptools: 23.0.0-py27_0
    sqlite:     3.13.0-0
    tk:         8.5.18-0
    wheel:      0.29.0-py27_0
    zlib:       1.2.8-3

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|########################################################################| 100%
#
# To activate this environment, use:
# $ source activate flask-ask-temp1
#
# To deactivate this environment, use:
# $ source deactivate
#
MacBook-Air:Alexa echou$

2. Activate Environment

MacBook-Air:Alexa echou$ source activate flask-ask-temp1

3. Install a conda package

(flask-ask-temp1) MacBook-Air:Alexa echou$ conda install numpy
Fetching package metadata .......
Solving package specifications: ..........

Package plan for installation in environment /Users/echou/anaconda/envs/flask-ask-temp1:

The following NEW packages will be INSTALLED:

    mkl:   11.3.3-0
    numpy: 1.11.1-py27_0

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|########################################################################| 100%
(flask-ask-temp1) MacBook-Air:Alexa echou$

4. Install a package that is NOT available thru Conda:

(flask-ask-temp1) MacBook-Air:Alexa echou$ pip install flask-ask
Collecting flask-ask
Collecting PyYAML (from flask-ask)
Collecting six (from flask-ask)
  Using cached six-1.10.0-py2.py3-none-any.whl

5. Testing

(flask-ask-temp1) MacBook-Air:Alexa echou$ python
Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:43:17)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from flask_ask import Ask
>>> dir()
['Ask', '__builtins__', '__doc__', '__name__', '__package__']
>>> exit()
(flask-ask-temp1) MacBook-Air:Alexa echou$

6. Deactivate

(flask-ask-temp1) MacBook-Air:Alexa echou$ source deactivate
MacBook-Air:Alexa echou$

7. Delete the Env

(flask-ask-temp1) MacBook-Air:Alexa echou$ source deactivate
MacBook-Air:Alexa echou$ conda remove -n flask-ask-temp1 --all

Package plan for package removal in environment /Users/echou/anaconda/envs/flask-ask-temp1:

The following packages will be REMOVED:

    mkl:        11.3.3-0
    numpy:      1.11.1-py27_0
    openssl:    1.0.2h-1
    pip:        8.1.2-py27_0
    python:     2.7.12-1
    readline:   6.2-2
    setuptools: 23.0.0-py27_0
    sqlite:     3.13.0-0
    tk:         8.5.18-0
    wheel:      0.29.0-py27_0
    zlib:       1.2.8-3

Proceed ([y]/n)? Y

Unlinking packages ...
[      COMPLETE      ]|########################################################################| 100%
MacBook-Air:Alexa echou$

There you have it, one step further to manage your Python work environment.

Leave me a comment for your thoughts! Happy Coding!





Monday, July 18, 2016

The Making of Network Automation Nerds Pre-Flight Podcast

When I was at PyCon 2016, I had the chance to talk to Michael Kennedy from Talk Python To Me a few times. I have been enjoying his podcast for a while now, even was part of the panel on people who recently gotten a job as Python developers.

I tried to pick his brain about his journey and experience. The conversations were really enjoyable, we shared a common interest in Python and building life-style business (think Tim Ferriss of 4-Hour Work Week). Of course, he is waaaay ahead of me as far as progress. :)

One of the things he mentioend was how Podcasting had helped him in building up his audience and that I might give it a shot. After giving it some thoughts, I thought, 'Why not?' It wouldn't hurt to at least learn about the workflow as long as the upfront investment is not that much. Sure enough, after a day or two of planning, I was able to submit and received the approval for my Podcast from iTunes (because who else is out there, really?):

Network Automation Nerds Podcast

"THE" most useful resource that I found, was this tutorial from PodcastAnswerMan as well as this step-by-step-guide from SmartPassiveIncome. I wont repeat his instructions because he does a much better job at his methods. The videos are short and enjoyable so highly recommended that, if interested, go thru them in one setting and drill down on the parts you might need more help on.

Personally, I chose to picked the cheapest way forward:

1. The $2.99 iPhone app for recording audio file.
2. This microphone that was recommended by SmartPassiveIncome on the link above. It is a bit pricey at $79 but I figure I will be able to reuse them for other projects as well.
3. My audio file was put on my a web server that I already have.
4. The initial RSS feed was done via Blogger, which was free. As recommended by PodCastAnswerMan, this can be changed if you have a bigger audience later on.
5. The middle feed was done on Feedburner, combine with step #4, this provides the most flexibility to bootstrap the podcast quickly as well as expansion opportunity if your audience grow later on.

Overall, I was surprised at how easy the whole process was. All for less than $100. Even with the equipment I bought they can be reused later on. I love the fact that we are now all self-branded and we can deliver our message to the biggest audience possible as long as they offer value to others.

Now comes the hard part, offer value to people who give the Podcast a shot and offer them incredible amount of value!

Leave me a comment if you have any questions about the process of making this go live. :)



Tuesday, July 5, 2016

Developing Ansible Modules Troubleshooting Steps

In this post I wanted to show my troubleshooting process using print statements and interactive prompt for small programs. As network engineers starts to write code, there are inevitably times when something that does not go as planned and you need to troubleshoot. For small programs, simple print statements should be able to do the job.

I am going thru the Ansible tutorial for developing modules, http://docs.ansible.com/ansible/developing_modules.html#tutorial. The second iteration of timetest.py seems to have contained some errors. The file as-is copied and pasted into the file, timetest.py simply printed out the same result as the original timetest.py.

echou-a10:playbooks echou$ ansible/hacking/test-module -m ./timetest.py -a "time=\"March 14 12:23\""
* including generated source, if any, saving to: /Users/echou/.ansible_module_generated
* this may offset any line numbers in tracebacks/debuggers!
***********************************
RAW OUTPUT
{"time": "2016-06-29 17:48:16.553883"}


***********************************
PARSED OUTPUT
{
    "time": "2016-06-29 17:48:16.553883"
}
echou-a10:playbooks echou$

Which is clearly not the desired result indicated on the tutorial:
{"changed": true, "time": "2012-03-14 12:23:00.000307"}

Upon some troubleshooting, it seems that the syntax "WANT_JSON", even commented out in the file and not in the -a syntax, causes the args interpreted to be JSon.

 22
 23 arguments = shlex.split(args_data)
 24
 25 print("args_data: ", args_data, "arguments: ", arguments)
 26
 27 for arg in arguments:
 28     print("arg: ", arg)
 29     # ignore any arguments without an equals in it
 30     if "=" in arg:

***********************************
RAW OUTPUT
('args_data: ', '{"time": "March 14 12:23"}', 'arguments: ', ['{time:', 'March 14 12:23}'])

If you take out that line, the args_data will now be the string as it was explained in the tutorial and enter into the loop. However, it is now giving me the error:

***********************************
RAW OUTPUT
{"msg": "failed setting the time", "failed": true}


***********************************
PARSED OUTPUT
{
    "failed": true,
    "msg": "failed setting the time"
}

Upon reading the file, it looks like this error message is generated under the condition where rc != 0. Therefore if we insert a print statement in line 44:

 43             rc = os.system("date -s \"%s\"" % value)
 44             print("rc: ", rc, "value: ", value)

the output shows:

RAW OUTPUT
('rc: ', 256, 'value: ', 'March 14 12:23')

A quick interactive prompt test:

echou-a10:playbooks echou$ python
Python 2.7.11 |Anaconda 2.5.0 (x86_64)| (default, Dec  6 2015, 18:57:58)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import os
>>> value = 'March 14 12:23'
>>> rc = os.system("date -s \"%s\"" % value)
date: illegal option -- s
usage: date [-jnu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
>>> rc
256
>>> exit()
echou-a10:playbooks echou$

Which gave me an error indicating 'date' command under OS-X does not have a '-s' option. Quickly trying it out on a Ubuntu machine shows that even when the '-s' option is available, the process will still return a non-zero value because the date command cannot be executed (need to be root, also I dont want to execute the script as root that sets my date).

echou@a10-ubuntu3:~$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> value = 'March 14 12:23'
>>> rc = os.system("date -s \"%s\"" % value)
date: cannot set date: Operation not permitted
Mon Mar 14 12:23:00 PDT 2016
>>> rc
256
>>> exit()
echou@a10-ubuntu3:~$

At this point, it is probably easier to force the value of rc to be 0:

 42
 43             #rc = os.system("date -s \"%s\"" % value)
 44             rc = 0
 45

Now the output is good:

echou-a10:playbooks echou$ ansible/hacking/test-module -m ./timetest.py -a "time=\"March 14 12:23\""
* including generated source, if any, saving to: /Users/echou/.ansible_module_generated
***********************************
RAW OUTPUT
{"changed": true, "time": "2016-06-29 18:57:06.625650"}


***********************************
PARSED OUTPUT
{
    "changed": true,
    "time": "2016-06-29 18:57:06.625650"
}
echou-a10:playbooks echou$

I just wanted to show that the process that I go thru for troubleshooting. Everybody go about the troubleshooting process a little differently, this is just my process that can hopefully help other network engineers who are perhaps newer to Python.


Happy coding.








Tuesday, June 28, 2016

Python's CowSay - Cowpy

It is late and has been a long day (like when I write this post now), and you need some geeky comedy relief. In comes Cowpy. According to Wikipedia:

cowsay is a program which generates ASCII pictures of a cow with a message.[2] It can also generate pictures using pre-made images of other animals, such as Tux the Penguin, the Linux mascot. It is written in Perl. There is also a related program called cowthink, with cows with thought bubbles rather than speech bubbles. .cow files for cowsay exist which are able to produce different variants of "cows", with different kinds of "eyes", and so forth.[3] It is sometimes used on IRC, desktop screenshots, and in software documentation. It is more or less a joke within hacker culture, but has been around long enough that its use is rather widespread. In 2007 it was highlighted as a Debian package of the day.

Check out the GitHub repository:
https://github.com/jeffbuttars/cowpy

One of the YouTube demo:
https://www.youtube.com/watch?v=RnfXIbMuJh0

- Sample Result (steps below):

$ cowpy -c beavis "I like Networking"
 ___________________
< I like Networking >
 -------------------
   \         __------~~-,
    \      ,'            ,
          /               \
         /                :
        |                  '
        |                  |
        |                  |
         |   _--           |
         _| =-.     .-.   ||
         o|/o/       _.   |
         /  ~          \ |
       (____\@)  ___~    |
          |_===~~~.`    |
       _______.--~     |
       \________       |
                \      |
              __/-___-- -__
             /            _ \
(venv) echou-a10:tmp echou$


1. Step 1. I use Python 3's built in venv module for virtual environment to install:

echou:tmp echou$ python3 -m venv venv
echou:tmp echou$ source venv/bin/activate

(venv) echou:tmp echou$ pip3 install cowpy
Collecting cowpy
  Downloading cowpy-1.0.1.tar.gz
Installing collected packages: cowpy
  Running setup.py install for cowpy
Successfully installed cowpy-1.0.1
You are using pip version 7.1.2, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

2. Let the fun begins:

(venv) echou:tmp echou$ cowpy
 ____________________________________
< Cowsay | cowpy. Please seek --help >
 ------------------------------------
     \   ^__^
      \  (oo)\_______
         (__)\       )\/\
           ||----w |
           ||     ||

(venv) echou:tmp echou$ echo "Network Engineering" | cowpy
 _____________________
< Network Engineering >
 ---------------------
     \   ^__^
      \  (oo)\_______
         (__)\       )\/\
           ||----w |
           ||     ||

(venv) echou:tmp echou$

(venv) echou:tmp echou$ cowpy "Network Engineering is Cool"
 _____________________________
< Network Engineering is Cool >
 -----------------------------
     \   ^__^
      \  (oo)\_______
         (__)\       )\/\
           ||----w |
           ||     ||

(venv) echou:tmp echou$ cowpy -c ghostbusters "I like Networking"
 ___________________
< I like Networking >
 -------------------
          \
           \
            \          __---__
                    _-       /--______
               __--( /     \ )XXXXXXXXXXX\v.
             .-XXX(   O   O  )XXXXXXXXXXXXXXX-
            /XXX(       U     )        XXXXXXX\
          /XXXXX(              )--_  XXXXXXXXXXX\
         /XXXXX/ (      O     )   XXXXXX   \XXXXX\
         XXXXX/   /            XXXXXX   \__ \XXXXX
         XXXXXX__/          XXXXXX         \__---->
 ---___  XXX__/          XXXXXX      \__         /
   \-  --__/   ___/\  XXXXXX            /  ___--/=
    \-\    ___/    XXXXXX              '--- XXXXXX
       \-\/XXX\ XXXXXX                      /XXXXX
         \XXXXXXXXX   \                    /XXXXX/
          \XXXXXX      >                 _/XXXXX/
            \XXXXX--__/              __-- XXXX/
             -XXXXXXXX---------------  XXXXXX-
                \XXXXXXXXXXXXXXXXXXXXXXXXXX/
                  ""VXXXXXXXXXXXXXXXXXXV""

(venv) echou-a10:tmp echou$

3. Great! So how do I add the images I want?

     Step 1. Find an online image, such as this Targaryen sigil I found online:



     Step 2. Use this site to convert the image to ascii art
     Step 3. Instantiate your own class like below


    Step 4. Enjoy your own Cowpy art!



Check out the GitHub link for all the options.

Happy Coding!





Tuesday, June 21, 2016

Software for Linux Screencast

I bought a new Dell XPS 13 running Ubuntu 14.04 out of box with the Sputnik project. Nothing fancy, just the low end non-touch model. For those who cried foul that I am suppose to buy a cheap Laptop and install Linux from scratch, I know how you feel, sorry I sort of cheated. But I just want to have a Linux laptop without having to be my own tech support when it comes to things like fixing WiFi drivers and hooking up external monitors (although they were not as smooth as I would've expected, but that is for another post). Overall I am happy that I have a small, nicely engineered Linux laptop.

However, when it comes time to showing my work and do screencast, I remember seeing an article last year about free Linux software that includes some screen recording and video editing software. Here is the original article 20 Free Open Source Softwares I Found in Year 2015 if you are interested.

From the article I found the two tools I was looking for, Simple Screen Recorder and OpenShot Video Editor.

Here are the two most helpful articles on setting them up and getting them running. At least for Ubuntu, they are straight forward as to adding the repository and do apt-get install.

Screencasting with Simple Screen Recorder

http://www.openshot.org/ (Go to Download, and for me I chose the PPA method)

I used the default settings to record in 30 fps with the entire monitor at 1920x1080. Then use OpenShot to export it to HD 1080p 30 fps MP4 video.

Here are two screenshots of the software:







































Here is a sample of the final MP4 that I did.

video


Happy coding! Let me know in the comments if there are some settings you are interested in that I did not cover.







Sunday, June 19, 2016

Be More Serious about Blogging and Building a Community

To The Readers:

I am sorry I have been away for so long! I suck at blogging. When I look at my last post, it was dated December 2015, which means I have not written a blog for almost 6 months. Why? I think the following reasons applies:

1. Life Events. The past 6 months has been more eventful than usual for life outside of my 9-5. I quit my job at Microsoft and started working for A10 Networks again. Let's face it, anybody who have switched jobs know that you agonize about the move weeks / months before it actually happens, not to mention all the phone screens and interview loops. I also bought a house that required some renovation before moving in, then of course, have to move out of the old home and into the new. Moving is no fun and sucks the energy out of ya, I tell you.

2. Professional Direction. I have been making my living as network engineering before switching my focus to software development, particularly Python. However, I never wanted to just code all day. So I was trying to find a balance between writing code, building stuff, and relate them back to network engineering. I wrote my first online class for Internetwork Expert (https://streaming.ine.com/c/practical-python-for-network-engineers) and spent 10 weeks being a teaching assistant at University of Washington Extension Python 200 night course. They were each rewarding in their own rights, but they demanded my time and attention.

All in all, I am happy with the progress I have made this year, especially the teaching experiences, but there is one thing I regret the most: Not keeping up with my blog! This is highly unfortunate because although I started this blog as a place to do a brain dump of my code and projects, over time it proves to be one of the most valuable piece of work I have done. I can often point to this blog as a repository of projects and spur new conversations. Sometimes I look back at the post and find new inspirations. Reading thru comments that people leave allows me to build a closer relationship with people who spent their precious time reading things I have written. The list goes on and on.

So here I am, 6 months after my last post, writing a post to pledge my renewed enthusiasm for blogging. My goal is to share my knowledge that people find useful and build a community around the topic of network automation.

My renewed goals for my blog are:

1. Provide valuable information to people who wants to read about network automation, Python, and network engineering.

2. In time, earn their trust so they are open about what they want to learn and which topic they will find value in and pay for.

3. I will be able to focus on the topics that bring the most value to the audience with free and paid material.

I want to be completely honest that one of my goals is to construct paid course material that will help me sustain my effort, but not before I make sure the contents are worth every penny that someone paid for them. Also I will make sure I bring value to the world by putting free and valuable knowledge that can serve others.

So enough chatting, let's go for some Python for Network Engineer contents!

Sincerely,

Eric








Sunday, December 27, 2015

Anaconda Environment and VirtualEnv Management

Anaconda Python from Continuum is a great tool. It is easy to install and get rid of a lot of the headache of package installation and management.  It is one of the reasons Python has a big footprint in the Scientific Computing and Data Science circle, in my opinion. I enjoy their one-installer-for-all-platform especially and happily traded in my 'pip' commands for 'conda' commands (for more information click here). Anaconda is my default Python interpreter on multiple machines.

There is only one complain, working with Python VirtualEnv is different when you have Anaconda Python as your default Python interpreter. Doing a quick search for 'Anaconda Python VirtualEnv' shows a lot of frustration amongst others. Conda places emphasis on reproducibility as all scientific research should, so their recommended way of managing environment is to use the 'conda env' commands.

I don't claim the following is correct, but they work for me.

***** Using the Good Old VirtualEnv for Learning *****

Currently I am going thru RealPython's courses (which are excellent, by the way), and want to create quick VirtualEnv to separate the different packages used in different lessons.

- Find out your Python interpreter path and specify it when creating virtualenv

MacBook-Air:my_code echou$ virtualenv -p /usr/bin/python2.7 test
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in test/bin/python
Installing setuptools, pip, wheel...done.
MacBook-Air:my_code echou$ source test/bin/activate
(test)MacBook-Air:my_code echou$
(test)MacBook-Air:my_code echou$ deactivate

MacBook-Air:my_code echou$ 

***** Use Conad Env for more Permanent Projects *****

I also intend to keep a few Flask environment active for some other projects, which I have elected to use the Conda environments. 

Here are some helpful links: 

http://conda.pydata.org/docs/using/envs.html #Managing Anaconda environments



MacBook-Air:my_code echou$ conda env --help
usage: conda-env [-h] {attach,create,export,list,remove,upload,update} ...

positional arguments:
  {attach,create,export,list,remove,upload,update}
    attach              Embeds information describing your conda environment
                        into the notebook metadata
    create              Create an environment based on an environment file
    export              Export a given environment
    list                List the Conda environments
    remove              Remove an environment
    upload              Upload an environment to anaconda.org
    update              Update the current environment based on environment
                        file

optional arguments:
  -h, --help            Show this help message and exit.

my_code echou$ conda create --name TEST2 ipython (click on the link above for Anaconda package list)
Fetching package metadata: ....
Solving package specifications: ...............
Package plan for installation in environment <locaion>/anaconda/envs/TEST2:

The following NEW packages will be INSTALLED:

    appnope:          0.1.0-py27_0
    decorator:        4.0.6-py27_0
    ipython:          4.0.1-py27_0
    ipython_genutils: 0.1.0-py27_0
    openssl:          1.0.2d-0
    path.py:          8.1.2-py27_1
    pexpect:          3.3-py27_0
    pickleshare:      0.5-py27_0
    pip:              7.1.2-py27_0
    python:           2.7.11-0
    python.app:       1.2-py27_4
    readline:         6.2-2
    setuptools:       19.1.1-py27_0
    simplegeneric:    0.8.1-py27_0
    sqlite:           3.8.4.1-1
    tk:               8.5.18-0
    traitlets:        4.0.0-py27_0
    wheel:            0.26.0-py27_1
    zlib:             1.2.8-0

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|##################################################################################################################| 100%
#
# To activate this environment, use:
# $ source activate TEST2
#
# To deactivate this environment, use:
# $ source deactivate
#

my_code echou$ source activate TEST2
discarding <path>/anaconda/bin from PATH
prepending <path>/anaconda/envs/TEST2/bin to PATH
(TEST2)my_code echou$ source deactivate
discarding <path>/anaconda/envs/TEST2/bin from PATH
my_code echou$
*****

So far it works for me, if I find any problems or issues down the line I will update this post. 

Happy Coding!