Sunday, May 5, 2013

PyTapDEMon - Part 3 Pro-Active Monitoring with sFlow-RT

Originally, part 3 of my project was going to be a web front end to allow users to indicate which port to tap and monitor. However, last week I came across a few blog posts on sFlow-RT on the DEMon project that makes things very interesting.

http://blog.sflow.com/2013/04/sdn-packet-broker.html
http://blog.sflow.com/2013/05/software-defined-analytics.html

Here is a screencast of the project. Sorry about the quality, I need to go thru the tutorial of Camtasia to edit the video better. But all the commands in the screencast are all listed below:



It is interesting because, as the blog suggested, sFlow-RT allows for automatic tapping based on criteria set up by the user in a pro-active fashion. I, too, believes this changes the dynamic of distributed monitoring quit a bit. So I have decided to see if I can play around with this approach in my project a bit.

After some thought, this is the simple high level overview: 

  1. All the conditions from my prototype are still the same (PyTapDEMON - Part 2 Prototype). 
  2. Upon observing any web traffic from h1 to h2, port-based flow from h8 and h9 will be pushed down to s2. 
  3. Port 4 will be mirrored to port 7 connected from s2 to s3, and I should then see the traffic on s11 connected to s3. 

I am using my VM setup before (Part 2) plus the following:

    1. sFlow-RT on my host machine, 192.168.56.1 (VM eth1 ip is 192.168.56.101).
    2. In order to differentiate agents from the switches, the following IPs are configured:


sudo ifconfig s1-eth1 192.168.1.1 netmask 255.255.255.0
sudo ifconfig s2-eth1 192.168.2.1 netmask 255.255.255.0
sudo ifconfig s3-eth1 192.168.3.1 netmask 255.255.255.0


    3. Then sFlow export are pushed out manually:


sudo ovs-vsctl -- --id=@s create sFlow agent=s1-eth1 target=\"192.168.56.1:6343\" header=128 polling=10 -- set Bridge s1 sflow=@s

sudo ovs-vsctl -- --id=@s create sFlow agent=s2-eth1 target=\"192.168.56.1:6343\" header=128 polling=10 -- set Bridge s2 sflow=@s

sudo ovs-vsctl -- --id=@s create sFlow agent=s3-eth1 target=\"192.168.56.1:6343\" header=128 polling=10 -- set Bridge s3 sflow=@s

    4. I can see the switches on sFlow-RT:








 

    5. From the sFlow blog above, the condition of monitoring TCP flows is added:


mininet@mininet-vm:~$ curl -H "Content-Type:application/json" -X PUT --data "{keys:'ipsource,ipdestination,tcpsourceport,tcpdestinationport', value:'bytes'}" http://192.168.56.1:8008/flow/tcp/json
mininet@mininet-vm:~$






 

    6. Simple web server is added to h1 and tested on h2 (from Mininet walk thru):

mininet> h1 python -m SimpleHTTPServer 80 &
mininet> h2 wget -O - h1
--2013-05-05 10:17:45--  http://10.0.0.1/
Connecting to 10.0.0.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 320 [text/html]
Saving to: `STDOUT'
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"><html>
<title>Directory listing for /</title>
<body>
<h2>Directory listing for /</h2>
<hr>
<ul>
<li><a href="controllers.py">controllers.py</a>
<li><a href="PyPath.py">PyPath.py</a>
</ul>
<hr>
</body>
</html>

     0K                                                       100% 64.7M=0s

2013-05-05 10:17:45 (64.7 MB/s) - written to stdout [320/320]

mininet>

Ok, now I am ready to run the test. Here are steps: 

    1. Here is the Python script to grab the data from sFlow-RT and push out flows when condition is met: 

#!/usr/bin/env python

import json
import urllib2
import subprocess

data = json.load(urllib2.urlopen('http://192.168.56.1:8008/metric/ALL/tcp/json'))

# if TCP traffic is detected
if 'topKeys' in data[0]:
    info =  data[0]['topKeys'][0]['key']
    dst, src, dstPort, srcPort = info.split(',')
    # install flows to s2 for h8 and h9
    # port 4 will also duplicate traffic to port 7 mirror port
    subprocess.call(['sudo','ovs-ofctl','add-flow','s2','hard_timeout=120,in_port=4,actions=output:3,output:7'])
    subprocess.call(['sudo','ovs-ofctl','add-flow','s2','hard_timeout=120,in_port=3,actions=output:4'])

    2. The script is initiated via cron tab:

* * * * * python /home/mininet/PyTapDEMON/tests/sflow-rt_test.py 

    3. Currently h8 cannot reach h9:

mininet> h8 ping -c3 h9
PING 10.0.0.9 (10.0.0.9) 56(84) bytes of data.
From 10.0.0.8 icmp_seq=1 Destination Host Unreachable
From 10.0.0.8 icmp_seq=2 Destination Host Unreachable
From 10.0.0.8 icmp_seq=3 Destination Host Unreachable

--- 10.0.0.9 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2018ms
pipe 3
mininet>

    4. Now generate interesting traffic from h2 to h1: 

mininet> h2 wget -O - h1
--2013-05-05 15:13:09--  http://10.0.0.1/
Connecting to 10.0.0.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 320 [text/html]
Saving to: `STDOUT'
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"><html>
<title>Directory listing for /</title>
<body>
<h2>Directory listing for /</h2>
<hr>
<ul>
<li><a href="controllers.py">controllers.py</a>
<li><a href="PyPath.py">PyPath.py</a>
</ul>
<hr>
</body>
</html>

     0K                                                       100% 63.4M=0s

2013-05-05 15:13:09 (63.4 MB/s) - written to stdout [320/320]

mininet>

    5. The traffic an be viewed from sFlow-RT:















    6. The before and after flows from s2:

mininet@mininet-vm:~/PyTapDEMON/tests$ sudo ovs-ofctl dump-flows s2
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=8.197s, table=0, n_packets=0, n_bytes=0, idle_timeout=120,hard_timeout=120,in_port=1 actions=output:2,output:6,output:8
 cookie=0x0, duration=8.194s, table=0, n_packets=0, n_bytes=0, idle_timeout=120,hard_timeout=120,in_port=2 actions=output:1,output:6,output:8

mininet@mininet-vm:~/PyTapDEMON/tests$ sudo ovs-ofctl dump-flows s2
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=7.323s, table=0, n_packets=0, n_bytes=0, idle_timeout=120,hard_timeout=120,in_port=1 actions=output:2,output:6,output:8
 cookie=0x0, duration=6.431s, table=0, n_packets=5, n_bytes=378, hard_timeout=120,in_port=3 actions=output:4
 cookie=0x0, duration=6.445s, table=0, n_packets=5, n_bytes=378, hard_timeout=120,in_port=4 actions=output:3,output:7
 cookie=0x0, duration=7.323s, table=0, n_packets=0, n_bytes=0, idle_timeout=120,hard_timeout=120,in_port=2 actions=output:1,output:6,output:8
mininet@mininet-vm:~/PyTapDEMON/tests$

    7. Now h8 can ping h9:

mininet> h8 ping -c3 h9
PING 10.0.0.9 (10.0.0.9) 56(84) bytes of data.
64 bytes from 10.0.0.9: icmp_req=1 ttl=64 time=2.09 ms
64 bytes from 10.0.0.9: icmp_req=2 ttl=64 time=0.047 ms
64 bytes from 10.0.0.9: icmp_req=3 ttl=64 time=0.052 ms

--- 10.0.0.9 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.047/0.732/2.099/0.966 ms
mininet>

    8. More importantly, I am now monitoring h9 traffic as intended on h11:











Obviously this had nothing to do with the POX component that I wrote. But that is part of the beauty, they are decoupled, I can use whatever mechanism I choose to tie in sFlow-RT with my setup. The fact that we get sFlow by virtue of default is very valuable. 

I like this approach a lot, this is both simple and powerful. The next step for me is to see if I can make this into a more useful tool in the next few weeks before I need to present this in class. 

Cheers. Let me know what you think. 












1 comment:

  1. Grab the entity id for your server. Find the Label for your server and grab the ID.
    raxmon-entities-list --username=Your-User --api-key=Your-api-Key --details
    You will be given a chunk of results look for the ID, Melbourne Video Intercom Systems

    ReplyDelete