Category Archives: NSX

Cloning VM’s from powercli then automatically add to a NSX load balancer.


This environment will be built out from the script.  It is a simple powershell script from powercli.  I am not that great with powercli I was able to learn a lot about powercli this week with some given down time.  It really is not that hard.

The script can be found on my github here the script basically first clones a vm from a template as many times as it is in the range.  After cloning it will then return back the vsphere mob id for the VM’s as NSX load balancer uses either a mob reference or it can use an IP. Im not sure how to power on the VM and then grab the IP?  I guess I could have done that but I really do not have the time and this was just for a small fun project.

After the VM’s are created the NSX portion takes over.  I could not figure out for the life of me how to get powershell to communicate properly with the NSX api.  Thankfully Chris Wahl had a really good blog post on how to do just that followed up with hit github on how to do so.

There are three effective API calls.  The first is to create the application profile.  I put in 443 but that can be changed, the second is to create the back end pool I am also assuming that snat is being used.  The third then creates a virtual IP with that Pool inside of it.  I need to add more to this logic.  Like returning the pool in a string etc… I just ran out of useable time today unfortunately.   But for the most part the script should work within powershell.

NSX Python automated build

I figured I would share this NSX python script to build an entire environment.  I worked with VMware a few months ago to get a lot of this going.  Some of this is my code some VMware’s.  But generally it builds out this environment.  This is taking into assumption that the NSX manager appliance is built out connected to vcenter, clusters are prepared for NSX vibs and controllers are deployed.  The click click click click has always killed me.  This should take up to 5-10 minutes to run all three scripts total.


This is all within my Github page Here

Here are some screen shots of how the script works..

Create the distributed logical router

Screenshot from 2016-04-18 08:57:46

Create routing on the distributed logical router for bgp purposes

Screenshot from 2016-04-18 09:07:42

Create 4 Edge service gateways with BGP peering to the top of rack switches and DLR.  BGP is created, syslog is created and firewall is disable on the edges.

Screenshot from 2016-04-18 11:00:49

Everything you need other than the network build can be found within the vsphere MOB https://vcenterip/mob – > content – > root folder.


Everything is within the github repository.

There is a A side VLAN 1085(only on ToR1) B side VLAN 1086(only on TOR2)

Each Logical switch is unicast mode

Edge firewall is disable,ECMP is enabled, Syslog is created.

This was setup in three scripts because I had to use each of these all the time to either convert a VLAN backed environment to VXLAN or swap over a OSPF environment over to a BGP environment.

VCP-NV 610 passed


On my second shot November 27th I passed the VCP-NV 610 exam. The test was all multiple choice and tested my knowledge of NSX in all aspects. For this test I walked in and took it the first time not expecting much. I failed by a lot. I had to go back and study things I was weak on. For my production / lab environments I focused a lot on the routing aspect of NSX and not enough on all the other services because routing is generally what I felt most comfortable with. Once I went back and made sure I understood all concepts of NSX I passed the exam on the second try.

To be honest the exam is not structured very well. There were a few questions that seemed like they were unrelated to the entire blue print. It is a good think I did not need high marks to pass the exam. Otherwise, if someone had some hands one experience with NSX ie the hands on labs or home lab it would be a easy pass. I would recommend this test to all my network friends as if you have a valid CCNA you do not need the normal VMware class to get the certification.

NSX troubleshooting commands.

NSX Controller related commands
show control-cluster status – Shows if a controller is connected to the cluster

This command is ran on every NSX controller to make sure that each controller is added to the 3 node cluster. For some reason or another if the NSX controller is not enabled for all processes it either has to be deleted or rebooted then re added.

If for some reason the join is not completed then do the following.
1.) Ping the other NSX controllers for connectivity
2.) Reload controller.
3.) Check NSX install management to see if the controller is setup.

show control-cluster logical-switch vni xxx – This command shows which one of the NSX controllers handles all the functionality for a particular VXLAN/VNI.
In my experience if you do not see a logical switch /VNI associated with a specific controller please do the following.
1.) Make sure the right VNI is being used
2.) Find the logical switch change its mode to multicast then back to unicast quickly.

show controler-cluster logical-switches vtep-table xxx – Discover what hosts participate in a VXLAN

1.)You do not see VTEPs showing up on the controller who owns that VNI/VTEP – Restart the NetCPA agent by logging into a ESXi host and issuing the following command /etc/ini.d/netcpad restart
2.)Netcpa did not resolve the issue the only way to fix it at this point is a reboot of the host.

show control-cluster logical-switches arp-table xxx – Discover VM’s arp address in a VXLAN
Connection-ID shows the Host where it belongs to. If we look at the previous command.

1.)If a IP address does not show up in a controller issuing the arp-table command for its VXLAN/VNI chances are that VM will not be able to communicate to the outside world due to an issue with the host where it lives. Take that VM and migrate it to another host that has a working VTEP.
2.) IP address shows up but cannot ping its default gateway. Check to see the default gateway of the host and make sure it matches the default gateway of the LIF same goes with hosts OS.

show control-cluster logical-switches mac-table xxx – Discover VM’s mac addresses in a VXLAN

Same thing as the Arp-table the connection-ID directly maps to the VTEP table.

1.)Mac does not show up in the controller. Chances are there is an issue with the host. Check that the hosts VTEP interface shows up when issuing the command to see all the VTEPs that participate within a VXLAN/VNI. VMotion the VM to another host and reboot the non functional host.
2.)Check to make sure that the mac address is correct in the guest operating system.

show control-cluster logical-routers instance all – Shows each edges association with each host.
This command like the other controller commands will look different per controller. The LR-ID number will be needed for future commands.

show control-cluster logical-routers interface-summary – Provides all the interfaces for the LDR / Edge associated

show control-cluster logical-routers interface routerID interface – Provides the default gateway IP / MAC and MTU

show control-cluster logical-routers routes routerID – Shows all the routes for a given ESG. Note this is different per controller.

NSX edge commands
show ip route
Show ip route ospf/static/bgp
Show ip ospf
Show ip ospf neighbors
Show ip ospf database
show firewall flows – Will show every single flow going through the Edge router at that time. Similar to a iptables –L
show firewall flows top 10 – Provides the top 10 largest sessions
show firewall flows top 10 sort-by-pkts – Provides the top 10 by the amount of packet
show flowtable – will show all flows.
show ip forwarding – Displays the FIB as show ip route will show the rib
show system uptime – Shows the uptime of a device.

ESXi Related troubleshooting commands
esxcli network vswitch dvs vmware vxlan list – Lists the VTEP segment and default gateway for the VTEP with MTU
net-vdr -l –instance – Will list the routers along with their associated LIFs etc.
Esxcli software vib list | grep vxlan – This is the installed vib that needs to be installed on each host. If the vib is not installed the host cannot participate in VXLAN.

NSX Controllers east to west traffic.

In the typical network space with VLANs where we have our core,distribution and access layer. We typically run the default gateway as a HSRP address. Whichever router/switch is the active n its pair the mac address is sent to each individual hosts and they arp for their default gateway which would be the HSRP address. Otherwise, for east to west traffic in the same VLAN a arp request is handled by the VM to the physical network to ask how to find another mac address it is trying to talk to.

Thinking differently in the network overlay world since we are overlaying networks and tunneling all traffic something has to sit off to the side and tell us how to reach our default gateway and how to get to guests within the same subnet. This concept is known as a controller. A controller will build routes and tell a ESXi host how to find the mac address of a corresponding VM or how to reach its default gateway which is otherwise known as a LIF or logical interface that lives on either a Logical distributed router or a edge service gateway. The controller is responsible for telling the ESXi host where to send the traffic next hop wise.

In this example I am showing two physical hosts, two VM’s in the same VXLAN and two controllers.


So lets first start a ping from Tenant-A-1 to Tenant-A-2.


So everything is pinging YAY.

Now how does this work? If you recall I said the controllers are whats making this possible. One of the controllers has to tell the ESXi host where Tenant-A-1 lives how to get to Tenant-A-2.

So lets log onto the controllers and run some commands to figure out how it knows this is possible.


What the command show control-cluster logical-switches vtep-table 5000 is telling me is I have 2 hosts that participate in that VXLAN/VNI 5000. and The first host will be referenced as Connection-ID 3. The second Connection-ID 2. This will make sense in a few seconds.

Okay, so that is great thats how that works now how does the ESXi host know where to send traffic to while these are pinging?Arp-5

The following two commands explain it all here. The first shows the mac addresses of both VM’s on VNI/VXLAN 5000. The second will explain how to resolve arp to ip the same way a normal Layer 3 router or switch would. So the controller tells the host how to get to each corresponding VM it needs to get to. So lets VMotion everything to live off of and see what happens.


There you have it the controller knows how to get to the two mac addresses of either tenant-a-1 or tenant-a-2.