hey other siders !!!

[icecream-guy]

Hanging out doing an all night maintenance, get to say hi!! live to you guys on the other side of the planet..    wet rainy night here. ..

[icecream-guy]

snore....snore...snore

[deanwebb]

I would have responded, I was dealing with wannacry and was up... but I was doing a screen share... boss doesn't like it when I do non-work stuff on a screen share...

[LynK]

I was working on the 13th. Started 6509 upgrade form 12.2 to 15.1

Started at 10:30 and finished by 1:30.

[icecream-guy]

I was working on the 13th. Started 6509 upgrade form 12.2 to 15.1

Started at 10:30 and finished by 1:30.

I had to roll back,  2 hours upgrade, 1/2 hour of troubleshooting, 2 hours rollback.
(5k has like 20 FEX connected)

For some reason the antique StoneGate we run wasn't able to populate it's ARP table when we went from NX-OS 5.1 to NX-OS 5.2 - odd.
5K switch would see everything in it's MAC address table (l2 switch) and the upstream 6500 Catalyst (l3 switch) would see everything in ARP too,
but the Stonegate not. Through troubleshooting we found that configuring a static entry for the SVI on the StoneGate wouldn't hold through reboots.
Also found that when one of the 5K's goes down, the active/active StoneGate looses connections and everything breaks.
Something to due with our dual homed architecture and VLANs suspending when one 5K switch goes down.

Plan is to rehome the StoneGate outside connections to the 6500 Catalyst and try the upgrade again in a few weeks.

I also learned from Cisco that the upgrades go quicker when you shut the dual homed FEX connections on the secondary 5K. this way they upgrade and finish and don't forever reload (though doesn't fix the VLAN suspension issue). Without saving the config, upgrade the secondary, and all the links come back up proper.

[LynK]

For some reason the antique StoneGate we run wasn't able to populate it's ARP table when we went from NX-OS 5.1 to NX-OS 5.2 - odd.

okay... interesting

active/active StoneGate looses connections and everything breaks.

I think I found your issue. active/active is an ARP NIGHTMARE. Can you change this to A/P?

I also learned from Cisco that the upgrades go quicker when you shut the dual homed FEX connections on the secondary 5K.

Mental note taken. Are you using VPC+? Any considerations to changing this, to make it easier on yourself?

[icecream-guy]

For some reason the antique StoneGate we run wasn't able to populate it's ARP table when we went from NX-OS 5.1 to NX-OS 5.2 - odd.

okay... interesting

active/active StoneGate looses connections and everything breaks.

I think I found your issue. active/active is an ARP NIGHTMARE. Can you change this to A/P?

I also learned from Cisco that the upgrades go quicker when you shut the dual homed FEX connections on the secondary 5K.

Mental note taken. Are you using VPC+? Any considerations to changing this, to make it easier on yourself?

will have to check that A/P think,  but when we roll back to 5.1 all is good.  don't know why A/P should work better in 5.2

DC is closing, NOBODY is interested in spending time or effort to make changes to the End of Service architecture.
more than to run a few cables as work around to passify the security folks,

[weasleman]

Wet and rainy. Manchester UK always wet and raining 

We get a 25c weather warning here once every 10-15 years followed by the news telling everyone the elderly are going to die in the sweltering heat.

So be grateful you have good weather 

[deanwebb]

25c causes your elderly to perish in sweltering heat?

We issue cold weather advisories for 25c or below here in Texas... 

[weasleman]

So true.

[deanwebb]

Maybe the rain is best for the UK. From the look of it, you lot seem to have trouble figuring out snow:



Best to leave that business to the Scandinavians, I suppose.

[icecream-guy]

Hey again, making another all night attempt as a redo for that failed maintenance on May 13th.

[icecream-guy]

freakin' weird, 

on the 6500 where the VLAN SVI is
I see the mac address of the firewall in the CAM,
I see the network in the CEF table
I see the IP address in the ARP table as INCOMPLETE.

I can't ping the firewall A/A VIP.

firewall doesn't see MAC address of the SVI gateway and cant ping it either.

as we roll back again, all works normally/.

firewall is connected directly to the 5K's, we moved it from the FEX to the 5K.

[deanwebb]

freakin' weird, 

on the 6500 where the VLAN SVI is
I see the mac address of the firewall in the CAM,
I see the network in the CEF table
I see the IP address in the ARP table as INCOMPLETE.

I can't ping the firewall A/A VIP.

firewall doesn't see MAC address of the SVI gateway and cant ping it either.

as we roll back again, all works normally/.

firewall is connected directly to the 5K's, we moved it from the FEX to the 5K.

So I look up incomplete IP addresses... https://supportforums.cisco.com/document/11216/packets-are-not-being-forwarded-due-incomplete-entries-arp-table

But that's from, like, 8 years ago... and then I read another article that complains about how Cisco implements the RFC for ARP... and that's from 7 years ago... Has this been a Cisco issue for a very long time?

[icecream-guy]

typically ya can't do layer 3 without layer 2