Building Virtual networks with VMwares NSX

I have had the time over the last 3 weeks to start setting up NSX along with some help from VMware. I myself have been looking forward to something similar to this for a long time. The chance to do networking on a broad scale of deployment where I do not have to use physical networking gear. I will look at this from a network engineers perspective and not a system admin / virtual administrator. I will quickly highlight some information on NSX terms that will be used.

-ESG Edge Services gateway. This is the edge of the NSX network that allows NSX to reach out to the physical network ie BGP,ISIS,OSPF or Static
-LDR Logical Distributed Router. This is similar to a DVs. This is a router that spans multiple hosts inter or intra clusters. Which allows for Logical interfaces / Distributed DFGW.
-VXLAN A VXLAN is similar to a VLAN in the Layer 2 world. VXLAN is where most of the magic happens where we can virtualize our networks.
-VTEP VLAN tunnel end point. A VTEP is a IP address that each individual ESXi host receives. They will build tunnels between each ESXi host in order to overlay networks.
-VXLAN Bridge Allows bare metal devices to participate in the same subnet as NSX.
-Transport zone. A transport zone allows a large overlay so that ESG and LDR can talk to each other similar to running a VLAN between multiple routers or switches.
-NSX manager The manager speaks back and forth to Vcenter.
-NSX Controllers There are three NSX controllers that will push routes down to each VTEP telling each VTEP how to get to each server.

Alright I am glad that is over. So I will go over the design I decided to use. Mine is a bit complex I was lucky enough to use Nexus 7700s and Nexus 56128s in a leaf and spine setup.


So physically this is how my setup looks. I am using 2 ESG’s for redundancy. Each ESG peers with a respective 7ks. Between edge routers and LDR’s I am running OSPF as a dynamic routing protocol. This is extremely similar to how we are doing networking today there is not much of a change. Except for the way I am doing eBGP between the edge routers to the 7ks. I will explain that in a later blog post but I am using OSPF as a recursive lookup.

This design also pushes Layer 3 out to the edge. Which is great because us network people like layer 3 over layer 4.

Logically this is what my design would look like taking out the underlay out of the equation.


Logically everything is the same. The idea here is that we are decoupling the physical network and overlaying it. This makes for a great idea as I can spin up as many edge routers as I want to. The ESG and LDR’s are simply VM’s which reside in a cluster.

So how does everything work within NSX from a data flow perspective?

If the VM’s I have posted within want to talk to each other the flow is relatively simple. Each VM will be forwarded up to the LDR. The LDR then checks through the VTEP over to the NSX controller to see which VTEP it would traverse for east west traffic. For traffic that is on a different subnet similar flows will happen. Traffic will hit the LDR and be routed across its respective VXLAN.

Some known gotchas for anyone deploying NSX in the future.
Controller and VTEP has to have connectivity to each other.
Manager has to have connectivity into VCenter and use a SSO account
Never ever try to firewall VTEP traffic it wont work out so well
VTEP tunnels will not work with multipathing. ie if I have two VTEP tunnels per ESXi host I will only use one for forwarding within 6.0.4 release of NSX.

Post a comment or leave a trackback: Trackback URL.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: