Saturday, February 2, 2013

OpenFlow Tutorial with POX - Part 1

*****

[Update 11/25/2016: I humbly invite you to check out my InternetworkExpert Class on Learn SDN with OpenFlow and Ryu Controller]

*****



The OpenFlow Tutorial is simply awesome. It is hands down the best way to gain some hands on experience with OpenFlow. Everything is encapsulated into a VirtualBox VM and you simply need to bootstrap it. Everything you need for network component is included in Mininet, which is another awesome tool in itself. The wireshark component has the OpenFlow dissect included already. And to run simple OpenFlow test you can use the built-in dpctl.

I do, however, think when it comes to actually construct a Controller + Mininet network the tutorial needs more example and hand-holding. Let's face it, for those of us coming from Cisco / Juniper background with some Python chops it is still a very steep learning curve when:

[If this is helpful to you, please leave me a note. If enough people respond, we can add a few items to the tutorial wiki. Otherwise I will just keep it here on my blog]

1. You dont know which portion is OpenFlow, which is Python, and which is Controller. 

For example, when the tutorial reference of.ofp_flow_mod() and "this instructs a switch to install a flow table entry"; it includes all three components, POX/OpenFlow/Python. The flow_mod is just an OpenFlow message, while it is implemented by POX via a Python function. But it is not until I read the POX WIki that I realize the the function is implemented in the "pox.openflow.libopenflow_01" module and the example probably skipped the potion where 'import pox.openflow.libopenflow_01 as of' at the top of the file.

2. Most of the pointers later on are task oriented, there is no end-to-end examples to build from.

So here I aim to pick up at the "Create Learning Switch" section and provide a more detailed example using POX as the controller.


[Update 04/06/2013: If you are using the new VM with mininet 2.0 (added Feb 2013) in the tutorial, the new s1 switch port 1 and port 2 now connects to h1 and h2 instead of h2 and h3. The section below used the old image, which uses h2 and h3 at the first two ports. Please adjust accordingly if you use the new image.
old:
Old:
openflow@openflowtutorial:~$ sudo mn --topo single,3 --mac --switch ovsk --controller remote
*** Adding links:
(s1, h2) (s1, h3) (s1, h4)
***
mininet> net
s1 <-> h2-eth0 h3-eth0 h4-eth0
mininet>
New:
mininet@mininet-vm:~$ sudo mn --topo single,3 --mac --switch ovsk --controller remote
*** Adding links:
(h1, s1) (h2, s1) (h3, s1)
***
mininet> net
c0
s1 lo:  s1-eth1:h1-eth0 s1-eth2:h2-eth0 s1-eth3:h3-eth0
h1 h1-eth0:s1-eth1
h2 h2-eth0:s1-eth2
h3 h3-eth0:s1-eth3
mininet>
]


1. Clone the POX controller from Git repository, Git is already pre-installed on the VM:
openflow@openflowtutorial:~$ git clone http://github.com/noxrepo/pox
Cloning into pox…
2. Verify and change into the directory:
openflow@openflowtutorial:~$ ls
mininet nox oflops oftest openflow openvswitch pox
openflow@openflowtutorial:~$ cd pox
openflow@openflowtutorial:~/pox$ ls
COPYING debug-pox.py doc ext pox pox.py README setup.cfg tests tools
openflow@openflowtutorial:~/pox$
3. Open up a separate window and start the simple Mininet topology:
openflow@openflowtutorial:~$ sudo mn --topo single,3 --mac --switch ovsk --controller remote
*** Adding controller
*** Creating network
*** Adding hosts:
h2 h3 h4
*** Adding switches:
s1
*** Adding links:
(s1, h2) (s1, h3) (s1, h4)
*** Configuring hosts
h2 h3 h4
*** Starting controller
*** Starting 1 switches
s1
*** Starting CLI:
mininet>
4. Test your POX controller by starting the POX, look for the switch registration message. You can optionally open up wireshark to verify:
openflow@openflowtutorial:~/pox$ ./pox.py
POX 0.0.0 / Copyright 2011 James McCauley
DEBUG:core:POX 0.0.0 going up...
DEBUG:core:Running on CPython (2.7.1+/Apr 11 2011 18:05:24)
INFO:core:POX 0.0.0 is up.
This program comes with ABSOLUTELY NO WARRANTY.  This program is free software,
and you are welcome to redistribute it under certain conditions.
Type 'help(pox.license)' for details.
DEBUG:openflow.of_01:Listening for connections on 0.0.0.0:6633
INFO:openflow.of_01:[Con 1/1] Connected to 00-00-00-00-00-01
Ready.
POX>
POX> 

5. If you go back to the Mininet window and try to have H2 ping H3, nothing will happen because the controller is not doing anything. This is pretty boring.
mininet> h2 ping h3
<nothing>
<ctrl+c to kill>

6. At this point, exit the interactive prompt with exit(). You can leave the Mininet topology running or kill it in another window. I find it easier to kill it and just 'up arrow + enter' every time to see my controller messages.
POX> exit()
7. The way POX initiate components, is to use the component name as the argument. To use a pre-built L2 learning switch component, you can use 'forwarding.l2_learning' as the first argument.
openflow@openflowtutorial:~/pox$ ./pox.py forwarding.l2_learning
POX 0.0.0 / Copyright 2011 James McCauley
DEBUG:core:POX 0.0.0 going up...
DEBUG:core:Running on CPython (2.7.1+/Apr 11 2011 18:05:24)
INFO:core:POX 0.0.0 is up.
This program comes with ABSOLUTELY NO WARRANTY.  This program is free software,
and you are welcome to redistribute it under certain conditions.
Type 'help(pox.license)' for details.
DEBUG:openflow.of_01:Listening for connections on 0.0.0.0:6633
Ready.
POX>
POX> INFO:openflow.of_01:[Con 1/1] Connected to 00-00-00-00-00-01
DEBUG:forwarding.l2_learning:Connection [Con 1/1]
7. At this point, if you go back to Mininet window, you can see that as a learning switch, h2 can now poing h3:
mininet> h2 ping -c5 h3
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_req=1 ttl=64 time=17.4 ms
64 bytes from 10.0.0.3: icmp_req=2 ttl=64 time=0.298 ms
64 bytes from 10.0.0.3: icmp_req=3 ttl=64 time=0.041 ms
64 bytes from 10.0.0.3: icmp_req=4 ttl=64 time=0.050 ms
64 bytes from 10.0.0.3: icmp_req=5 ttl=64 time=0.000 ms
--- 10.0.0.3 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 0.000/3.559/17.407/6.924 ms
mininet> 
And your POX controller prompt should show the controller installing the flows:
POX>
POX> DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.1
DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:02.1 -> 00:00:00:00:00:03.2
DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.1
DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.1
DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:02.1 -> 00:00:00:00:00:03.2
POX> 
You can also use the dpctl tool to verify that the flows are installed:
openflow@openflowtutorial:~/pox$ dpctl dump-flows tcp:127.0.0.1:6634
stats_reply (xid=0xad1245f1): flags=none type=1(flow)
  cookie=0, duration_sec=4s, duration_nsec=993000000s, table_id=0, priority=32768, n_packets=1, n_bytes=42, idle_timeout=10,hard_timeout=30,arp,dl_vlan=0xffff,dl_vlan_pcp=0x00,dl_src=00:00:00:00:00:02,dl_dst=00:00:00:00:00:03,nw_src=10.0.0.2,nw_dst=10.0.0.3,nw_tos=0x00,nw_proto=2,tp_src=0,tp_dst=0,actions=output:2
  cookie=0, duration_sec=9s, duration_nsec=1000000s, table_id=0, priority=32768, n_packets=7, n_bytes=686, idle_timeout=10,hard_timeout=30,icmp,dl_vlan=0xffff,dl_vlan_pcp=0x00,dl_src=00:00:00:00:00:02,dl_dst=00:00:00:00:00:03,nw_src=10.0.0.2,nw_dst=10.0.0.3,nw_tos=0x00,icmp_type=8,icmp_code=0,actions=output:2
  cookie=0, duration_sec=4s, duration_nsec=994000000s, table_id=0, priority=32768, n_packets=1, n_bytes=42, idle_timeout=10,hard_timeout=30,arp,dl_vlan=0xffff,dl_vlan_pcp=0x00,dl_src=00:00:00:00:00:03,dl_dst=00:00:00:00:00:02,nw_src=10.0.0.3,nw_dst=10.0.0.2,nw_tos=0x00,nw_proto=1,icmp_type=0,icmp_code=0,actions=output:1
  cookie=0, duration_sec=10s, duration_nsec=32000000s, table_id=0, priority=32768, n_packets=9, n_bytes=882, idle_timeout=10,hard_timeout=30,icmp,dl_vlan=0xffff,dl_vlan_pcp=0x00,dl_src=00:00:00:00:00:03,dl_dst=00:00:00:00:00:02,nw_src=10.0.0.3,nw_dst=10.0.0.2,nw_tos=0x00,icmp_type=0,icmp_code=0,actions=output:1
openflow@openflowtutorial:~/pox$
Notice the first ICMP reply is really slow, waiting for the flow to be installed. But later packets are fast.

8. So now we know POX works, let's start with some boiler plate code so we can learn. Toward the end of the POX Wiki, there is a link for the code prepared by William Emmanuel Yu (not sure if all the codes were written by him). Get his codes from Git as well.

openflow@openflowtutorial:~/pox$ git clone https://github.com/hip2b2/poxstuff.git
Cloning into poxstuff...
remote: Counting objects: 70, done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 70 (delta 39), reused 59 (delta 28)
Unpacking objects: 100% (70/70), done.
openflow@openflowtutorial:~/pox$ 
Note. There is also the interactive version of the same of_sw_tutorial_oo.py that is listed in the POX Wiki. But it did not work for me. Gave me the error below. I also prefer to do the 'dumb' thing and just type out code to help me learn. Just FYI.
openflow@openflowtutorial:~/pox$ ./pox.py of_sw_tutorial_oo
POX 0.0.0 / Copyright 2011 James McCauley
DEBUG:ext.of_sw_tutorial_oo:Initializing switch SW_IDEALPAIRSWITCH.
Traceback (most recent call last):
  File "./pox.py", line 352, in main
    if doLaunch():
  File "./pox.py", line 152, in doLaunch
    f(**params)
  File "/home/openflow/pox/ext/of_sw_tutorial_oo.py", line 327, in launch
    core.Interactive.variables['MySwitch'] = MySwitch
  File "/home/openflow/pox/pox/core.py", line 346, in __getattr__
    raise AttributeError("'%s' not registered" % (name,))
AttributeError: 'Interactive' not registeredopenflow@openflowtutorial:~/pox$
9. POX controller will automatically look for the ext/ directory for components. You can copy the of_switch_tutorial.py file to the ext/ folder if you want the POX controller to be able to find it automatically.
openflow@openflowtutorial:~/pox$ cp poxstuff/of_sw_tutorial.py ext/
openflow@openflowtutorial:~/pox$ ./pox.py of_sw_tutorial
POX 0.0.0 / Copyright 2011 James McCauley
INFO:ext.of_sw_tutorial:Switch Tutorial is running.
DEBUG:core:POX 0.0.0 going up...
DEBUG:core:Running on CPython (2.7.1+/Apr 11 2011 18:05:24)
INFO:core:POX 0.0.0 is up.
This program comes with ABSOLUTELY NO WARRANTY.  This program is free software,
and you are welcome to redistribute it under certain conditions.
Type 'help(pox.license)' for details.
DEBUG:openflow.of_01:Listening for connections on 0.0.0.0:6633
Ready.
POX>
POX> INFO:openflow.of_01:[Con 1/1] Connected to 00-00-00-00-00-01
10. To build muscle memory, I manually create a Python file, can start typing in line-by-line in order to learn. Here is the link I look at, https://github.com/hip2b2/poxstuff/blob/master/of_sw_tutorial.py.

11. For example, to build a dumb hub, follow the code and only write out what is required. Here is a block-by-block:
  1 #!/usr/bin/env python
  2
  3 # Copy and paste https://github.com/hip2b2/poxstuff/blob/master/of_sw_tutorial.py
  4 # step by step to understand the process
  5 
- The pox.core and pox.openflow.libopenflow_01 are the packages being in use in the functions:
  6 from pox.core import core
  7 import pox.openflow.libopenflow_01 as of
  8 
- Logging for output:
  9 log = core.getLogger()
 10
 11 table = {}
 12 
- This is a base function to allow sending packets out:
 13 def send_packet(event, dst_port = of.OFPP_ALL):
 14     msg = of.ofp_packet_out(in_port=event.ofp.in_port)
 15     if event.ofp.buffer_id != -1 and event.ofp.buffer_id is not None:
 16         msg.buffer_id = event.ofp.buffer_id
 17     else:
 18         if event.ofp.data:
 19             return
 20         msg.data = event.ofp.data
 21     msg.actions.append(of.ofp_action_output(port = dst_port))
 22     event.connection.send(msg)
 23 
- Here is the dumb hub function itself:
 24 def _handle_dumbhub_packetin(event):
 25     packet = event.parsed
 26     send_packet(event, of.OFPP_ALL)
 27
 28     log.debug("Broadcasting %s.%i -> %s.%i" %
 29         (packet.src, event.ofp.in_port, packet.dst, of.OFPP_ALL))
 30 
- Here is the launch() function to specify which function to use:
 31 # launch whichever implementation you want via function
 32 def launch():
 33     core.openflow.addListenerByName("PacketIn", _handle_dumbhub_packetin)
 34
 35     log.info("Switch Tutorial is running.")
 36
12. Let's start our simple dumb hub controller: 
openflow@openflowtutorial:~/pox$ ./pox.py of_sw_tutorial_myTestPOX 0.0.0 / Copyright 2011 James McCauley
INFO:ext.of_sw_tutorial_myTest:Switch Tutorial is running.
DEBUG:core:POX 0.0.0 going up...
DEBUG:core:Running on CPython (2.7.1+/Apr 11 2011 18:05:24)
INFO:core:POX 0.0.0 is up.
This program comes with ABSOLUTELY NO WARRANTY.  This program is free software,
and you are welcome to redistribute it under certain conditions.
Type 'help(pox.license)' for details.
DEBUG:openflow.of_01:Listening for connections on 0.0.0.0:6633
Ready.
POX>
POX> INFO:openflow.of_01:[Con 1/1] Connected to 00-00-00-00-00-01

mininet> h2 ping -c5 h3
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_req=1 ttl=64 time=74.3 ms
64 bytes from 10.0.0.3: icmp_req=2 ttl=64 time=33.4 ms
64 bytes from 10.0.0.3: icmp_req=3 ttl=64 time=44.9 ms
64 bytes from 10.0.0.3: icmp_req=4 ttl=64 time=21.6 ms
64 bytes from 10.0.0.3: icmp_req=5 ttl=64 time=6.10 ms
--- 10.0.0.3 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4026ms
rtt min/avg/max/mdev = 6.101/36.093/74.358/23.053 ms
mininet> 
Notice the packet time are longer on the first packet, and subsequent packets have longer time compare to the L2 learning switch. 

You will also see the broadcast packets on the POX console: 
POX> DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:02.1 -> ff:ff:ff:ff:ff:ff.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:02.1 -> 00:00:00:00:00:03.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:02.1 -> 00:00:00:00:00:03.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:02.1 -> 00:00:00:00:00:03.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:02.1 -> 00:00:00:00:00:03.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:02.1 -> 00:00:00:00:00:03.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.65532
DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:02.1 -> 00:00:00:00:00:03.65532
13. You can now start to experiment with the functions. Say, you can insert a print statement to see what the parsed packet look like: 
 23
 24 def _handle_dumbhub_packetin(event):
 25     packet = event.parsed
 26     print packet 27     send_packet(event, of.OFPP_ALL)
 28
 29     log.debug("Broadcasting %s.%i -> %s.%i" %
 30         (packet.src, event.ofp.in_port, packet.dst, of.OFPP_ALL))
 31 
Restart the controller, you will now see the packet: 
POX> INFO:openflow.of_01:[Con 1/1] Connected to 00-00-00-00-00-01
[00:00:00:00:00:02>00:00:00:00:00:03:IP]|([v:4hl:5l:84t:64]ICMP cs:26a5[10.0.0.2>10.0.0.3]){t:ECHO_REQUEST c:0 chk:23e7}{id:2835 seq:3}DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:02.1 -> 00:00:00:00:00:03.65532
[00:00:00:00:00:03>00:00:00:00:00:02:IP]|([v:4hl:5l:84t:64]ICMP cs:dfc5[10.0.0.3>10.0.0.2]){t:ECHO_REPLY c:0 chk:2be7}{id:2835 seq:3}DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.65532
[00:00:00:00:00:02>00:00:00:00:00:03:IP]|([v:4hl:5l:84t:64]ICMP cs:26a5[10.0.0.2>10.0.0.3]){t:ECHO_REQUEST c:0 chk:4fd1}{id:2835 seq:4}
14. Or if you want to look at the packet out message: 
 13 def send_packet(event, dst_port = of.OFPP_ALL):
 14     msg = of.ofp_packet_out(in_port=event.ofp.in_port)
 15     print msg 16     if event.ofp.buffer_id != -1 and event.ofp.buffer_id is not None:
 17         msg.buffer_id = event.ofp.buffer_id
 18     else:
 19         if event.ofp.data:
 20             return
 21         msg.data = event.ofp.data
 22     msg.actions.append(of.ofp_action_output(port = dst_port))
 23     event.connection.send(msg)
 24 
POX> ofp_packet_out  header:     version: 1    type:    13 (OFPT_PACKET_OUT)    length:  8    xid:     None  buffer_id: -1  in_port: 1  actions_len: 0  actions:DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:02.1 -> 00:00:00:00:00:03.65532
ofp_packet_out  header:     version: 1    type:    13 (OFPT_PACKET_OUT)    length:  8    xid:     None  buffer_id: -1  in_port: 2  actions_len: 0  actions:DEBUG:ext.of_sw_tutorial_myTest:Broadcasting 00:00:00:00:00:03.2 -> 00:00:00:00:00:02.65532
15. Let's try the lazyhub function, just create the lazyhub function and change the launch() function: 
 23
 24 def _handle_lazyhub_packetin (event): 25     packet = event.parsed 26      27     msg = of.ofp_flow_mod() 28     msg.idle_timeout = 10 29     msg.hard_timeout = 30 30     msg.actions.append(of.ofp_action_output(port = of.OFPP_ALL)) 31     event.connection.send(msg) 32      33     log.debug("Installing %s.%i -> %s.%i" % 34         ("ff:ff:ff:ff:ff:ff", event.ofp.in_port, "ff:ff:ff:ff:ff:ff", of.OFPF_ALL)) 35          36      
 37 # launch whichever implementation you want via function
 38 def launch():
 39     core.openflow.addListenerByName("PacketIn", _handle_lazyhub_packetin)
 40  
 41     log.info("Switch Tutorial is running.")
 42 
mininet> h2 ping -c5 h3
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_req=2 ttl=64 time=0.674 ms
64 bytes from 10.0.0.3: icmp_req=3 ttl=64 time=0.060 ms
64 bytes from 10.0.0.3: icmp_req=4 ttl=64 time=0.041 ms
64 bytes from 10.0.0.3: icmp_req=5 ttl=64 time=0.286 ms
--- 10.0.0.3 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4014ms
rtt min/avg/max/mdev = 0.041/0.265/0.674/0.255 ms
mininet>
POX>
POX> INFO:openflow.of_01:[Con 1/1] Connected to 00-00-00-00-00-01
DEBUG:ext.of_sw_tutorial_myTest:Installing ff:ff:ff:ff:ff:ff.1 -> ff:ff:ff:ff:ff:ff.65532
POX> 
I am not sure if this is helpful for others, but at least for me, this provides me with a good entry point to start understanding the different aspects of the POX controller and to proceed. 

Cheers. Leave me comments to let me know how to improve. 














78 comments:

  1. HI Eric Chou, i have some question, hope you'll help me:
    -when i created topo like you first: & sudo mn --topo single,3 --mac --switch ovsk --controller remote , i received message :Unable to contact the remote controller at 127.0.0.1:6633
    -when i tried connect to POX: ~/pox$ ./pox.py, i only saw display: INFO:openflow.of_01:[Con 1/1] Connected to 00-00-00-00-00-01 and no more. I saw you have POX> in below, i think this is place to make code and can controlle in POX.
    This is 2 my question. I'm the beginner at mininet.my email: tuanhai.bk@gmail.com

    ReplyDelete
    Replies
    1. Hi Hai Pham, thanks for reading! I took down my VM at the time to save some laptop resources. But both of those messages you saw are normal. The mininet topology is created with 1 switch and 3 hosts, the switch is trying to connect to the kernel switch at port 6633. If you havn't started the POX controller at port 6633, then it would not connect.

      When the POX controller is started, the switch immediately connects to the controller. The mininet sets the switch mac address as 00:00:00:00:00:01 (hosts are 0::02, 0::03, 0:004, etc IIRC). But by default, there is no applications associated with the POX, so you should proceed to step 6, stop the POX controller after you make sure the switch can register. Then proceed to use the L2 Learning switch app to make sure everything is working.

      If you are happy with the boiler plate applications (dumb hub, L2 learning switch), then you can stop there. But if you are interested in learning how to make your own application, then you should proceed.

      Hope it helps. :)

      Delete
    2. Hey Eric,

      I have a quick question:
      This is referring to step 7.

      So I have POX running simple l2_forwarding. I ping from h1 to h3. I can see the flows being added from the Debug of POX CLI, but I don't see any flows being dumped when I use dpctl.

      Here is the output:
      mininet> h1 ping h3
      PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
      64 bytes from 10.0.0.3: icmp_req=1 ttl=64 time=15.9 ms

      POX> DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:03.3 -> 00:00:00:00:00:01.1
      DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:01.1 -> 00:00:00:00:00:03.3
      DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:03.3 -> 00:00:00:00:00:01.1
      DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:03.3 -> 00:00:00:00:00:01.1
      DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:01.1 -> 00:00:00:00:00:03.3

      mininet@mininet-vm:~/pox/pox$ sudo dpctl dump-flows tcp:127.0.0.1:6634
      stats_reply (xid=0xf4171fd9): flags=none type=1(flow)

      mininet@mininet-vm:~/pox/pox$ sudo ovs-ofctl dump-flows tcp:127.0.0.1:6634
      NXST_FLOW reply (xid=0x4):

      mininet@mininet-vm:~/pox/pox$ sudo ovs-ofctl dump-flows s1
      NXST_FLOW reply (xid=0x4):

      mininet@mininet-vm:~/pox/pox$ sudo ovs-ofctl dump-flows c0
      ovs-ofctl: c0 is not a bridge or a socket


      mininet> net
      c0
      s1 lo: s1-eth1:h1-eth0 s1-eth2:h2-eth0 s1-eth3:h3-eth0
      h1 h1-eth0:s1-eth1
      h2 h2-eth0:s1-eth2
      h3 h3-eth0:s1-eth3


      Any help would be greatly appreciated
      Thanks,
      Daniel

      Delete
    3. Hi Daniel, sounds like the flows is installed and you can see the ping flowing from the hosts, just not from the other monitoring tool. I think there are two things that we can try:

      1. if you enable x-forwarding as taught by the openflow tutorial, clear flow tables, and do tcpdump on all hosts, do you see what you would expect? (i.e. broadcast first packet then just h1-h3).
      2. this is an assumption, but worth checking, prior to enable any constroller, the tutorial explained how to install flows manually by dpctl, it is probably worth checking again that dpctrl itself actually works.

      I have also found that the VM / Mininet sometimes would give a hick up, for me it happens mostly when the VM is still running and I hibernate the machine, when I come back to it later there seems to be some small quirks and I have to restart mininet. I dont know if this just happens to me and my machine or not.

      Let me know what you find out. tcpdump is really the next step for verification in my humble opinion.

      Delete
    4. HI Eric, now i want using Pox to control my topo. Example, when i connected to POX, i can ping h1,h2. But now i want h1 cant ping h2, can you tell me how can i do that by coding on POX, i dont know working on POX now.thanks!!!

      Delete
    5. Hi Hai Pham, if I understand correctly, you have successfully implement whatever pre-made packages from POX to control your topology such as the dumb hub and L2 learning switch, now you would like to make 1 small tweak to make sure you understand how everything ties together?

      If the answer is yes, the short answer is I was in the same boat when I started to explore POX and my goal for the blog is to share my experience so others can build on top of it. So for example, in step 13, you can see the packet structure that was sent to the controller. So to do what you want to do, you can simply insert an if statement to look for (ip.src==10.0.0.2 and ip.dst==10.0.0.3) and of action to drop. Then everything besides those packets will pass, but the packets matching your description will drop, just like an access list. :)

      Hope it helps.

      Delete
    6. Thanks very much, Eric can you send me your mail to my inbox, im studying at python code. If i have your mail, its very easily to contact you when i have a trouble.My mail: tuanhai.bk@gmail.com

      Delete
    7. @Daniel, doing more work with OF / Mininet again. Wondering if you still cant see the flows in dpctl? Just tried it again with the new image with Mininet 2.0:

      1. Initial state with no flow:
      mininet@mininet-vm:~$ date
      Sat Apr 6 18:25:08 PDT 2013
      mininet@mininet-vm:~$ dpctl dump-flows tcp:127.0.0.1:6634
      stats_reply (xid=0x906ef4d2): flags=none type=1(flow)
      mininet@mininet-vm:~$

      mininet@mininet-vm:~/pox$ date
      Sat Apr 6 18:25:19 PDT 2013
      mininet@mininet-vm:~/pox$

      2. Start POX in debug

      mininet@mininet-vm:~/pox$ ./pox.py --verbose forwarding.l2_learning
      POX 0.1.0 (betta) / Copyright 2011-2013 James McCauley, et al.
      DEBUG:core:POX 0.1.0 (betta) going up...
      DEBUG:core:Running on CPython (2.7.3/Sep 26 2012 21:51:14)
      DEBUG:core:Platform is Linux-3.5.0-17-generic-x86_64-with-Ubuntu-12.10-quantal
      INFO:core:POX 0.1.0 (betta) is up.
      DEBUG:openflow.of_01:Listening on 0.0.0.0:6633
      INFO:openflow.of_01:[00-00-00-00-00-01 1] connected

      3. h1 poing to h2

      mininet> h1 ping -c 3 h2
      PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
      64 bytes from 10.0.0.2: icmp_req=1 ttl=64 time=6.88 ms
      64 bytes from 10.0.0.2: icmp_req=2 ttl=64 time=39.4 ms
      64 bytes from 10.0.0.2: icmp_req=3 ttl=64 time=0.456 ms

      --- 10.0.0.2 ping statistics ---
      3 packets transmitted, 3 received, 0% packet loss, time 2005ms
      rtt min/avg/max/mdev = 0.456/15.608/39.481/17.083 ms
      mininet>

      4. Flow gets installed:

      DEBUG:forwarding.l2_learning:Connection [00-00-00-00-00-01 1]
      DEBUG:forwarding.l2_learning:Port for 00:00:00:00:00:02 unknown -- flooding
      DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:02.2 -> 00:00:00:00:00:01.1
      DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:01.1 -> 00:00:00:00:00:02.2
      DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:02.2 -> 00:00:00:00:00:01.1
      DEBUG:forwarding.l2_learning:installing flow for 00:00:00:00:00:01.1 -> 00:00:00:00:00:02.2

      5. dpctl shows flows:

      ininet@mininet-vm:~$ dpctl dump-flows tcp:127.0.0.1:6634
      stats_reply (xid=0xcdc223b6): flags=none type=1(flow)
      cookie=0, duration_sec=6s, duration_nsec=220000000s, table_id=0, priority=65535, n_packets=3, n_bytes=294, idle_timeout=10,hard_timeout=30,icmp,in_port=2,dl_vlan=0xffff,dl_vlan_pcp=0x00,dl_src=00:00:00:00:00:02,dl_dst=00:00:00:00:00:01,nw_src=10.0.0.2,nw_dst=10.0.0.1,nw_tos=0x00,icmp_type=0,icmp_code=0,actions=output:1
      cookie=0, duration_sec=5s, duration_nsec=185000000s, table_id=0, priority=65535, n_packets=2, n_bytes=196, idle_timeout=10,hard_timeout=30,icmp,in_port=1,dl_vlan=0xffff,dl_vlan_pcp=0x00,dl_src=00:00:00:00:00:01,dl_dst=00:00:00:00:00:02,nw_src=10.0.0.1,nw_dst=10.0.0.2,nw_tos=0x00,icmp_type=8,icmp_code=0,actions=output:2
      cookie=0, duration_sec=1s, duration_nsec=181000000s, table_id=0, priority=65535, n_packets=1, n_bytes=42, idle_timeout=10,hard_timeout=30,arp,in_port=2,dl_vlan=0xffff,dl_vlan_pcp=0x00,dl_src=00:00:00:00:00:02,dl_dst=00:00:00:00:00:01,nw_src=10.0.0.2,nw_dst=10.0.0.1,nw_proto=1,actions=output:1
      cookie=0, duration_sec=1s, duration_nsec=179000000s, table_id=0, priority=65535, n_packets=1, n_bytes=42, idle_timeout=10,hard_timeout=30,arp,in_port=1,dl_vlan=0xffff,dl_vlan_pcp=0x00,dl_src=00:00:00:00:00:01,dl_dst=00:00:00:00:00:02,nw_src=10.0.0.1,nw_dst=10.0.0.2,nw_proto=2,actions=output:2
      mininet@mininet-vm:~$ date
      Sat Apr 6 18:25:48 PDT 2013
      mininet@mininet-vm:~$

      Have you tried it again?

      Delete
    8. how to check flow entry is full or not

      Delete
  2. Hi Eric,

    First of all it was very helpful in understanding it and cleared many doubts. I have one question. inside misc.of_tutorial.py file I wrote parser which uses Scapy.
    def parseEachPacket(pkt):
    if(pkt.haslayer(Ether)):
    print "Ethernet::"
    ether = pkt[Ether]
    print " Src:: " + str(ether.src)
    print " Dst:: " + str(ether.dst)

    and i have imported required scapy packages. but when I run Error pop up:
    AttributeError: 'ethernet' object has no attribute 'haslayer'

    Any idea ? please help

    ReplyDelete
    Replies
    1. Hi, I am new to Scapy and OF as well, just a fair warning.. :)

      How are you calling the function? Because the error seems to indicate that pkt in the function is not understood as a Scapy packet. Here is a simple program that I took from the Scapy tutorial with your function:

      http://www.secdev.org/projects/scapy/doc/usage.html#simplistic-arp-monitor

      ** begin**

      #!/usr/bin/env python
      from scapy.all import *

      def parsePacket(pkt):
      if pkt.haslayer(Ether):
      print "Found Ethernet packet"
      print str(pkt[Ether].dst)

      if __name__ == "__main__":
      sniff(prn=parsePacket, store=0)

      **end**

      mininet@mininet-vm:~/pox$ sudo ./scapyTest.py
      WARNING: No route found for IPv6 destination :: (no default route?)
      Found Ethernet packet
      :ac
      Found Ethernet packet
      :5a
      ....

      The haslayer() function resides under packet.Packet:

      In [1]: from scapy.all import *
      WARNING: No route found for IPv6 destination :: (no default route?)

      In [2]:
      In [2]: packet.Packet.ha
      packet.Packet.hashret packet.Packet.haslayer

      In [2]:

      So perhaps search for a way to make the pkt into a Scapy packet object would be the logical next step?

      Delete
    2. Also, wanted to mention that it almost seems easier to parse out the packet in OF then reconstruct the packet in Scapy. For example, in step 13, you already know the IP source, then you can use Scapy to construct a new packet with the same source:

      from scapy.all import IP

      ipSource = packet.src
      newPacket = IP(src=ipSource)

      something like that might fit your needs?

      Delete
  3. i want to start up a dumb hub. however, when i type it out it gives me this "Module not found: of_sw_tutorial_myTest" how can i solve this?

    ReplyDelete
    Replies
    1. By default it searches the ~/ext directory, maybe check if the file is in that directory?

      Delete
  4. This is very helpful, many thanks!

    ReplyDelete
  5. Thanks Eric for this tutorial. It is very helpful.

    I've this question when I run:

    ./pox.py flow_stats l3_learning #### flow_stats.py from poxstuff/
    with mn --controller remote

    I've faced this error:


    POX> INFO:openflow.of_01:[Con 1/1] Connected to 00-00-00-00-00-01
    Task caused exception and was de-scheduled
    Traceback (most recent call last):
    File "/home/mininet/pox/pox/lib/recoco/recoco.py", line 276, in cycle
    rv = t.execute()
    File "/home/mininet/pox/pox/lib/recoco/recoco.py", line 94, in execute
    return self.gen.send(v)
    File "/home/mininet/pox/pox/lib/recoco/recoco.py", line 751, in run
    rv = self._callback(*self._args,**self._kw)
    File "/home/mininet/pox/ext/flow_stats.py", line 42, in _timer_func
    connection.send(of.ofp_stats_request(body=of.ofp_port_stats_request()))
    File "/home/mininet/pox/pox/openflow/of_01.py", line 572, in send
    data = data.pack()
    File "/home/mininet/pox/pox/openflow/libopenflow_01.py", line 2136, in pack
    packed += struct.pack("!HH", self.type, self.flags)
    error: cannot convert argument to integer
    DEBUG:ext.flow_stats:FlowStatsReceived from 00-00-00-00-00-01: []
    INFO:ext.flow_stats:Web traffic from 00-00-00-00-00-01: 0 bytes (0 packets) over 0 flows

    ReplyDelete
    Replies
    1. Hi Ali, I am afraid that might be a bit out of my knowledge based on the information given. Is flow_stats your own code or one of the stock ones?

      Have you tried the some of the email alias such as opeflow-discuss@ and mininet-discuss-requests@lists.stanford.edu? I find the list to be very helpful and knowledgeable, sometimes answered by creator of the tutorials/tools themselves.

      Best of luck! :)

      Delete
  6. hey Eric,

    Great thanks for your great job!
    It is a pity that there are no any references to this tutorial out there, I was having a tough time trying to understand the code before I got here. I think a small link to this tutorial in POX wiki and OpenFlow tutorial would be really helpful for newcomers.

    Nevertheless, I hope I am not too late to a party! :)
    I am trying to do a simple VLAN-translation function that would assign VLAN tags depending on given criteria.
    It would be very interesting to hear your opinion and maybe suggestions on realization of such function.

    Looking forward to hearing from you!

    Regards,
    Abe

    ReplyDelete
  7. Hi Abe, thank you for reading and the encouragement. :)

    I am not sure I qualify to answer that VLAN-translation question. I think in today's implementation, it is already a bit complex to determine packet forwarding based not the existing fields; having the promise to swap headers is certainly possible, but would take more work. It is certainly fun to have this option, right? :)

    ReplyDelete
  8. Hi Eric,
    actually I am new to python and POX.
    This tutorial helped me so much to understand the basic functions of POX and how it is working.
    would you mind tell me for understanding these codes in PYTHON, which reference would be useful?
    and another question is about lazyhub.
    I changed the dumbhub to lazyhub but it gives me erorr.
    i will be thanksful if you can help me.
    the error is as follow:
    POX> INFO:openflow.of_01:[Con 1/1] Connected to 00-00-00-00-00-01
    ERROR:core:Exception while handling OpenFlowNexus!PacketIn...
    Traceback (most recent call last):
    File "/home/mininet/pox/pox/lib/revent/revent.py", line 233, in raiseEventNoErrors
    return self.raiseEvent(event, *args, **kw)
    File "/home/mininet/pox/pox/lib/revent/revent.py", line 280, in raiseEvent
    rv = event._invoke(handler, *args, **kw)
    File "/home/mininet/pox/pox/lib/revent/revent.py", line 158, in _invoke
    return handler(self, *args, **kw)
    File "/home/mininet/pox/ext/of_sw_tutorial_myTest.py", line 33, in _handle_lazyhub_packetin
    ("ff:ff:ff:ff:ff:ff", event.ofp.in_port, "ff:ff:ff:ff:ff:ff", of.OFPF_ALL))
    AttributeError: 'module' object has no attribute 'OFPF_ALL'

    thanks a lot for providing this tutorial for others

    ReplyDelete
  9. hi eric,
    I have just started using pox
    can you please guide how to handle "www" request if their is no DNS server?

    thanks

    ReplyDelete
    Replies
    1. Hi Gurjot, sorry for the late reply, I have been thinking about doing a Part 2 of the tutorial. The OpenFlow project has come very far and POX has a ton of new features as well. Let me bundle this in with other questions for the Part 2. Stay tuned! :)

      Delete
  10. I'm working with forwarding_l2.multi..
    1. Whether forwarding.l2_multi.py is for switch only?
    2. That uses discovery.py to know the topology of the network and picks the shortest one, so where is the role of controller?

    Waiting eagerly for your reply...
    Thank you...
    Bondankit07@gmail.com

    ReplyDelete
    Replies
    1. Hi bondankit07, thank you for reading. I have been thinking about doing a part 2 for the tutorial with how different both OpenFlow and POX are now. I will probably bundle in the questions I have received and answer them together. Stay tuned! :)

      Delete
  11. hi eric,
    I am new in SDN. Can you please help me how to block ip addresses in mininet using pox controller?
    thanks

    ReplyDelete
    Replies
    1. Hi Divam, sorry for the late reply, I have been thinking about doing a part 2 of the tutorial with so much change in OpenFlow, SDN, and POX. I will bundle in the questions I have received and answer them together. Stay tuned! :)

      Delete
  12. Hi Eric,

    Great job! It helps me a lot trying to understand the differences and limits between OpenFlow/POX/Python, I've been struggling a while the original tutorial.

    Any clues on how to start designing and implementing a basic load balancer?

    Thanks a lot!!

    ReplyDelete
    Replies
    1. Hi Rous, thanks for reading! Yeah, when I first looked at POX the documentation were a bit lacking so I wrote this one after I bump my head many, many times. Glad it was helpful to you.

      To be honest, for basic load balancer, I will probably take a serious look at Google's project Seesaw, http://google-opensource.blogspot.com/2016/01/seesaw-scalable-and-robust-load.html. It is not Python, but I have heard good things about Go so it would be probably be a good opportunity to play with Go, I guess. :)

      Delete
  13. Hi Eric, great article for beginners It helped me a lot.
    I've a confusion ho can I build something in a way that there're 6 host one switch and one controller, where 3 host (Hx) and other 3 host(Gx) can only ping to each other but not to others.
    Like h1 ping g1 shouldn't ping but h1 ping h2 or g1 ping g2 should work.
    Any help is appreciated
    Thanks

    ReplyDelete
    Replies
    1. Hi, thanks for reading. I think so.. you just in the rules for the traffic that you want to go thru but not the ones you don't. With OpenFlow that is the level of control you have. Have you tried it and it does not work somehow? :)

      Delete
  14. how to check network performance in pox or mininet.
    i want to check performance metrics like throughput, delay,packet loss etc.

    ReplyDelete
    Replies
    1. Hi Mohammed, at least for BigSwitch controllers they report the switch stats at the controller level. Some of the switches also expose SNMP (I think). Do you have any particular vendor in mind?

      Delete
  15. This is a great tutorial. Help me a lot.
    Can anyone help me i want to drop packets from these two sources. but i am unable to run this. i am also new to python

    def _handle_dumbhub_packetin(event):
    packet = event.parsed
    print packet
    src_mac = packet.src
    dst_mac = packet.dst
    print src_mac
    print dst_mac
    if (packet.src == "00:00:00:00:00:01") or (packet.dst == "00:00:00:00:00:03"):
    print "hello" # drop();
    else:
    send_packet(event, of.OFPP_ALL)

    log.debug("Broadcasting %s.%i -> %s.%i"%(packet.src, event.ofp.in_port, packet.dst, of.OFPP_ALL))

    if condition is not going to execute correctly.

    ReplyDelete
    Replies
    1. Hi Sincer, just a note that I will look at this question when I spin up my OpenFlow lab again.. although I am not sure when I will get to it... :(

      Delete
  16. Hi,
    can you tell me how I can use POX with wireless interfaces ?

    ReplyDelete
    Replies
    1. Yeah, if you use the OpenWRT54G Linksys with OpenFlow 1.0 load here is the instruction:
      http://archive.openflow.org/wk/index.php/OpenFlow_1.0_for_OpenWRT#LinkSys_WRT54GL

      edit the /etc/config/wireless file, etc.

      Delete
  17. HI Eric,

    Can you tell me how can I communicate between two SDN controllers?

    ReplyDelete
    Replies
    1. Hi Sonal, from what I can tell the orchestration of controllers has always been fragmented and differs from implementer to implementer. The switch can always connect to multiple controllers, it is the automatic determining role of master/slave/dual-head that required some sort of controller-to-controller communication. This was defined in OF 1.2 and carried over to 1.3, https://www.opennetworking.org/images/stories/downloads/sdn-resources/onf-specifications/openflow/openflow-spec-v1.3.1.pdf but as you can see not much in terms of standardization.

      So... in short, if I were to do it, I would probably build my own using API by the controller itself instead of relying on the standard body. I think POX has not moved beyond OF 1.0, which was why I chose Ryu for my INE course. That would be my suggestion, use Ryu and see if you think the 1.3 spec fits your need, else use its included HTTP API to build your own.

      Cheers and good luck. :)

      Delete
  18. This comment has been removed by the author.

    ReplyDelete
  19. Hi Eric,

    When I run sudo python ip_loadbalancer.py in my terminal I get :

    File "ip_loadbalancer.py", line 23, in
    from pox.core import core
    ImportError: No module named pox.core

    Please Help.

    ReplyDelete
    Replies
    1. Hi Nitesh, core.py is still there by way of project code https://github.com/noxrepo/pox/blob/carp/pox/core.py and documentation, https://openflow.stanford.edu/display/ONL/POX+Wiki#POXWiki-ComponentsinPOX. If your ip_loadbalancer.py is your own code outside of the pox directory, then you need to add the POX directory to your Python module search path. This can be done using 'import sys' and 'sys.path.append("")'.

      Delete
    2. Hi Eric,none of my pox programs are running.Pox is installed in /home/pox. All my codes are in some folders in /home/pox/pox. For example, /home/pox/pox/forwarding/hub.py. So what should I add in sys.path.append("") ???

      Delete
  20. Hi Eric,
    I am working on an overlay topology (where all switches are connected) and I have a controller connected to one of the switches (In band controller). In my controller, I need to know the IPs of the hosts (not only the switches). Is there a way to know that at the controller level? or should I import these IPs as an input to the controller?
    Thanks in advance.

    ReplyDelete
  21. Can you please tell how to communicate between two pox controllers?

    ReplyDelete
  22. Great Article please keep writing, It helps people alot who love to read good articles.

    ReplyDelete
  23. I have h1 and h2 as host and c1 as controller.
    I want to change the POX controller so that if I ping between h1 and h3 it is h2 that receives the packets sent because h3 does not exist.this is my problem.I want to have ideas on that.
    Thank you

    ReplyDelete
  24. hi Eric can you help me please, i created my topology which includes two mobiles staions and one access point and i want to see the messages received by the controller i followed the instructions but i have always this line [dump:00-00-00-00-00-01|4096] [ethernet][ipv4][udp][dhcp] i didn't understand why pox just work fine with the predefine cmd "sudo mn --wifi --topo linear,5 --controller=remote,ip=10.0.2.15,port=6633" but not if i create my file.py

    ReplyDelete
  25. This comment has been removed by the author.

    ReplyDelete
  26. This comment has been removed by the author.

    ReplyDelete
  27. Great post, Please visit Courseya for Mysql free Courses

    ReplyDelete
  28. thanks for this excellent and nice post, its a good one. i'm working on attack and normal traffic generation using scapy, i actually created the SDN topology but having issue with traffic generation and traffic statistic collection. could you please help me out?

    ReplyDelete
  29. Nice post and good to know about network automation nerds. Thanks for sharing.
    psychometric test for employees
    swift interview questions

    ReplyDelete
  30. Hey everyone, “India is a place where color is twice as bright. Pinks that make your eyes scorch, blues that you can drown in." Foreign citizens can come to India and see this country's beauty. But you don't know how to apply for an India e visa? You can get Indian tourist visa info via our Indian visas website.

    ReplyDelete
  31. Very interesting post and I want more updates from your blog. Thanks for your great efforts...
    Divorce in Virginia with Child
    Divorce in Virginia with Child

    ReplyDelete
  32. If you are planning to travel to Equatorial Guinea, you will need a visa. A visa is a special document that allows you to enter and stay in a country for a specific period of time. To apply for an Equatorial Guinea visa , you will need to fill out an application, pay a fee, and provide some personal information and supporting documents, such as your passport and an itinerary. The type of visa you will need will depend on the purpose of your visit, be it for tourism, business or study. Equatorial Guinea offers a range of visa types, including short-term visas and long-term visas for those planning to stay for an extended period of time. Before planning your trip to Equatorial Guinea it is important to check the visa requirements for your specific state.

    ReplyDelete
  33. We value the information you gave. Thank you for providing this information. Top 30 Best Places To Live In The World In 2023. These sought-after destinations captivate with their unique charm, high quality of life, and promising opportunities. From the picturesque cities of Switzerland, known for their pristine landscapes and efficient infrastructure.

    ReplyDelete
  34. Greetings, fellow adventurers from the Dominican Republic! Dreaming of an Indian escapade? India Visa for Dominican Republic Citizens – you can conveniently apply for an e-Visa online. Whether it's for tourism, business, or medical purposes, the process is hassle-free. Just make sure your passport holds a validity of at least six months and carry a recent photo. Get set to embrace the enchanting wonders of India!

    ReplyDelete
  35. It is truly remarkable and highly appreciated that you are willing to impart knowledge and insight. I have greatly appreciated your guidance. Ghanaian adventurers! Planning an Indian getaway? India Visa for Ghana Citizens. you can apply for an e-Visa online. Whether for tourism, business, or medical purposes, the process is straightforward. Just ensure your passport has six months' validity and a recent photo. Get ready to experience the vibrant culture of India!

    ReplyDelete
  36. Hello! Your content captivates and informs effectively. Eagerly anticipating further insights from you in the times ahead! Planning a trip? Discover the essentials of Egypt visa for Bulgaria citizens. Ensure a seamless journey by understanding the Egypt visa requirements for a memorable experience.

    ReplyDelete
  37. Good morning everyone, Many people also ask about , 5 Things to Know Before Traveling to Turkey ? Planning a trip to Turkey? This captivating country offers a blend of rich history, vibrant culture, and stunning landscapes. To ensure a smooth and enjoyable journey, here are five essential things you should know before you go.

    ReplyDelete
  38. Hello! I wanted to express my gratitude for the valuable information shared on your blog. Your dedication to providing in-depth insights is truly commendable. Are you a Malaysian citizen planning a trip to Saudi Arabia? Check out Saudi Arabia Visa for Malaysia Citizens and explore the visa requirements, application process, and essential tips for a smooth travel experience.

    ReplyDelete
  39. Hlo sir, Discover the convenience and simplicity of the e-Visa system for Azerbaijan. With the e visa of Azerbaijan, travelers can easily obtain their travel authorization online, making their journey to this beautiful country hassle-free and efficient.

    ReplyDelete
  40. "This post is a gem! Your effort in sharing such valuable information is truly appreciated." Solomon Islands citizens can obtain an Indian visa for travel, business, or medical reasons. India Visa for Solomon Islands Citizens. Apply online or at designated centers. Ensure you have the necessary documents, pay the fees, and follow processing guidelines, which may vary. Be sure to check eligibility before applying and look forward to a memorable trip to India!

    ReplyDelete
  41. Obtaining a Turkey visa from Palestine is a crucial step for Palestinian travelers seeking to explore Turkey's rich cultural heritage, breathtaking landscapes, and historical treasures. This process ensures that Palestinian passport holders can embark on their Turkish adventures with ease. Travelers from Palestine can apply for their visas through the Turkish consulate or embassy in Palestine, adhering to the required documentation and procedures. Alternatively, the convenient e-Visa system offers a streamlined and user-friendly application process, simplifying travel planning to Turkey. With the appropriate visa, Palestinian adventurers can immerse themselves in the enchanting experiences that Turkey has to offer, creating lasting memories.

    ReplyDelete
  42. Hello! Great post, I found it very informative and helpful. Looking forward to more content! Getting an Azerbaijan visa for Indians is straightforward. Ensure a hassle-free visit to this beautiful country by checking the latest requirements and application process for Azerbaijan visa for Indians.

    ReplyDelete
  43. Hello everyone, How are you all of you. Exploring India's wonders has never been easier. Learn about the Indian Visa cost and plan your unforgettable journey to this diverse and captivating country without breaking the bank.

    ReplyDelete
  44. Excellent post! Concise, informative, and highly valuable content. Well done! Requirements to enter Saudi Arabia include a valid visa, passport, and adherence to their specific regulations, such as dress code and customs guidelines.

    ReplyDelete
  45. It's wonderful that you're willing to share these with us. I'm enthusiastic about joining the event in the future, as this topic resonates with me. Diwali celebrations across the globe

    ReplyDelete
  46. I genuinely appreciate your inclination to share these with us. In the days ahead, I'll be actively participating in the event, as this subject matter deeply interests me.Here is the important news morocco visa for Indians. Are you excited to know more about it? I am sharing this here because everyone benefits from it.

    ReplyDelete
  47. Your blog consistently delivers valuable insights! The way you articulate complex topics is truly impressive. Your dedication to quality content is appreciated. Keep up the excellent work! Explore the convenient Saudi on Arrival visa for Indian Passport holders

    ReplyDelete