NSX Distributed Switch and Edge Services GW OSPF adj

wintermute000 · June 19, 2015, 01:37:15 AM

Having a ridculously hard time bringing up OSPF adjacency between the NSX DLR and ESG, anyone got any tips?

I turned off both FWs. Initially that fixed it. Upon closer inspection, the FW rules had allow any any, so why was this blocking traffic? I turned the FW back on, dropped the adjacency again. But the next time I turned if off, the adjacency failed to come back. Its stuck on Init/DR.

I'm merely peering between 10.0.0.2 and 10.0.0.1 (can ping OK) with the 'fake' protocol address of 10.0.0.3 on the DLR.
On advice of a vmware forum post I removed my mgt address on the DLR (as its apparently not needed!??!) and put it on the logical switch segment same as 10.0.0.x, no dice.

https://communities.vmware.com/thread/499932

Makes me suspicious theres some kind of bug or state sync issue?

deanwebb · June 19, 2015, 08:55:08 AM

Were the firewalls trying to participate in OSPF routing and failing at it?

NetworkGroover · June 19, 2015, 12:05:40 PM

Anytime you have a peering issue that just doesn't make sense, I'd want to do a packet capture and take a look.

Granted it's a pair of Cisco boxes in this example, if you need something to reference:

http://aspiringnetworker.blogspot.com/2013/10/ospf-adjacency-building-process.html
http://aspiringnetworker.blogspot.com/2013/09/exploring-ospf-messages-between-new.html

So going by my article, it looks like you're stuck around step 3. So an OSPF HELLO is at least getting through in one direction, but possibly not the other - preventing transition to 2-way state? EDIT - Or you ARE getting a HELLO back, but it doesn't include it's own RID.... a packet capture would be really good here.

NetworkGroover · June 19, 2015, 12:13:21 PM

Also if this is a point-to-point link, if there's any way to do so, I'd specify it as such. There's no need for a DR on a point-to-point and you'll eliminate the associated unnecessary multicast communication (224.0.0.5,224.0.0.6).

EDIT - http://www.cisco.com/c/en/us/support/docs/ip/open-shortest-path-first-ospf/13702-7.html

burnyd · June 19, 2015, 12:30:58 PM

We can review on hangouts if you want.

wintermute000 · June 19, 2015, 08:52:17 PM

Thanks all. Burnyd this is amazing stuff, so glad you and others gave me the push to get onto it, I might still take you up on that hangouts session in the future once I get over the basics! (my exam is on July 4th).

Conclusion: software bug

(as other colleagues deploying prod/PoCs have told me already, OSPF is buggy and BGp is plain broken... what's your experience burnyd?).

I turned off the lab on Friday afternoon. Just turned it on again and behold.... I've changed NOTHING from when it was 'broken'.
The good thing is that this has nudged me towards labbing some packet captures so moving onto that topic.

AspiringNetworker, there are no such knobs to tune with NSX at least on the surface, your options are very limited compared to a Cisco or Juniper router. Heck AFAIK you HAVE to redistribute your DLR LiFs, for example/

nsx-esg1-1> show ip ospf nei
Neigbhor ID Priority Address Dead Time State
10.0.0.1 128 10.0.0.3 30 Full/BDR
nsx-esg1-1> show route
% Unknown command.
nsx-esg1-1> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 6

S 0.0.0.0/0 [1/1] via 192.168.0.1
C 10.0.0.0/24 [0/0] via 10.0.0.2
O E2 10.1.1.0/24 [110/1] via 10.0.0.1
O E2 10.1.2.0/24 [110/1] via 10.0.0.1
C 169.254.1.8/30 [0/0] via 169.254.1.10
C 192.168.0.0/24 [0/0] via 192.168.0.248
nsx-esg1-1>

burnyd · June 19, 2015, 11:23:10 PM

its prolly not broken. If this is your first time its prolly related to distributed routing. On the ESXi cli please type the following.

net-vdr -l --instance

If you get nothing back then distributed routing is busted. Try to restart the netcpad its the process that controls a lot of the distributed routing functionality. ./etc/init.d/netcpad restart.

wintermute000 · June 20, 2015, 07:33:05 AM

well I managed to break other things (ROFL) and then it gets weird when you uninstall network components off one host (to try to get back to a clean slate), but NSX decides to put a host in mgt mode, destroy its networking config and shove its management onto the wrong dvswitch port and wrong vlan. Even recovering the host by using fall back 'put the mgt vmkernel back onto a VSS' isn't working so I thought stuff it might as well start over again - since I'm rebuilding one host anyway...

So I'm starting again with a clean slate and actually drawing it all out (as I'm nesting there's an unholy tangle of vswitches and dvswitches), will report back if I find anything else funny.
Its kind of hard keeping it all in your head when nested and I am charging ahead much much quicker than IRL (with no planning either lol) but its certaiinly been educational.

FWIW before I broke everything I had OSPF going correctly, but it all started to go pear shaped when the controllers were all reporting disconnected and none of the ESXis were connecting to them when I issued net-vdl2, even after service restarting

NetworkGroover · June 20, 2015, 11:40:00 AM

Quote from: wintermute000 on June 19, 2015, 08:52:17 PM
Thanks all. Burnyd this is amazing stuff, so glad you and others gave me the push to get onto it, I might still take you up on that hangouts session in the future once I get over the basics! (my exam is on July 4th).

Conclusion: software bug (as other colleagues deploying prod/PoCs have told me already, OSPF is buggy and BGp is plain broken... what's your experience burnyd?).

I turned off the lab on Friday afternoon. Just turned it on again and behold.... I've changed NOTHING from when it was 'broken'.
The good thing is that this has nudged me towards labbing some packet captures so moving onto that topic.

AspiringNetworker, there are no such knobs to tune with NSX at least on the surface, your options are very limited compared to a Cisco or Juniper router. Heck AFAIK you HAVE to redistribute your DLR LiFs, for example/

nsx-esg1-1> show ip ospf nei
Neigbhor ID Priority Address Dead Time State
10.0.0.1 128 10.0.0.3 30 Full/BDR
nsx-esg1-1> show route
% Unknown command.
nsx-esg1-1> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 6

S 0.0.0.0/0 [1/1] via 192.168.0.1
C 10.0.0.0/24 [0/0] via 10.0.0.2
O E2 10.1.1.0/24 [110/1] via 10.0.0.1
O E2 10.1.2.0/24 [110/1] via 10.0.0.1
C 169.254.1.8/30 [0/0] via 169.254.1.10
C 192.168.0.0/24 [0/0] via 192.168.0.248
nsx-esg1-1>

Gotcha. Yeah I was wondering since you were mentioning firewall issues if eliminating unnecessary comms would help, but if your FW was allowing everything, that wouldn't have made sense. Getting a packet capture of that communication probably would have been very interesting and educational.

burnyd · June 20, 2015, 12:29:15 PM

A lot of times the LDR/DLR firewall gets in the way of things working which is anoiying. But by default it allows ospf in I think? In my script I have the dlr firewall turned off by default.

wintermute000 · June 20, 2015, 05:08:27 PM

New lab runs smoothly, no controllers dropping out or vteps disconnecting etc there must have been something messed in my initial deployment. I suspect all the messing around with vswitches and constant disconnecting eventually corrupted something, I was getting cryptic internal errors and the like to the point where I couldn't even create a logical switch for example.

But now that I've deployed it again, this time with some idea about what to do and a working topology off the bat, everything seems to hum along..... FOR ICMP

I can ping everything - across the DLR, to my real network, back and forth all fine. But I can't actually pass any production (TCP) traffic - SSH, HTTP, dead as a doornail.

I did a packet capture on the ESXis and I can see packets passing, all VXLANs up and running, controllers up, ARP/MAC tables look kosher and MTU1600 ping fine. I STRONGLY suspect its something to do with my nested arrangement as I can't hit my 'real' linux box via SSH from a nested ESXi either (but strangely I can ssh TO the nested ESXi from my PC... weird)

Confused, investigating (I'm sure any and all firewalls, NSX or host, are off)

wintermute000 · June 20, 2015, 09:36:38 PM

After going really deep.... I think I have basically the same problem as this guy.

https://www.google.com.au/url?sa=t&rct=j&q=&esrc=s&source=web&cd=6&cad=rja&uact=8&ved=0CD8QFjAF&url=https%3A%2F%2Fcommunities.vmware.com%2Fthread%2F488683&ei=DSKGVf2MO4a6mAWI2IPYBg&usg=AFQjCNFtCl3Gi1qGPSaIAgFI6gDZXUCKJg&sig2=UW3RhY7HgH8gU6hcT09oFA

Packet captures reveal I see the syn but the synack never makes it back.

The weird thing is that ICMP works fine, including replies (duh), so its not MAC/ARP/VTEP etc. related (I can do MTU 1580 pings from host VTEP to host VTEP).

I have gone into the dvswitch used for VXLAN and turned on promiscuous mode on all VXLAN segment dvportgroups, no dice.

wintermute000 · June 22, 2015, 01:44:54 AM

Problem solved. TCP OFFLOAD IS BAD when you're nesting!
Noticed checksum errors when tcpdumping on a destination guest. Turned off TCP offload and everything is pure gravy.

Oh well, at least now I know the NSX L2 troubleshooting tools really really well....