I Did a Good Thing, and I Am Proud

NetworkGroover · December 09, 2015, 10:11:40 AM

Quote from: dlots on December 09, 2015, 07:19:02 AM
Like you wouldn't believe!!! It's kind of a weird cloud setup. We have 40+ areas using the same single IP space, and I am the only one who does up-keep on it in my spair time, and it's kinda a hack job right now just cause I don't have time .

I would have been fine if we were doing PVST, but we have way to many vlans for that so the MSTP got me :-(

Woooo wee - that sounds like fun times. Not that you ever have time, but it sounds like the network is really touchy - it may behoove you, although I understand likely completely impossible, to re-evaluate things and see if you can make it more stable, understandable, and behave in a manner that makes sense and doesn't take a simple port change to break so much stuff. Easier said than done, I know. MSTP can be tricky sometimes, ESPECIALLY if you're running multi-region - please tell me you're not doing that...

dlots · December 09, 2015, 10:30:16 AM

The changing a port screwed me up because years ago when we set this up we had a vlan, lets say 10, vlan 10 was used for tones of crap in the old setup and we pruned the hell out of it and we said "you can use any vlan you want other than vlan 10, vlan 10 is a mess and we don't want it in our new pretty network area... Needless to say a couple weeks ago I was told "we have to have vlan 10". So I went though and got rid of vlan 10 on all the old stuff and permited it though on most of the new stuff... but I missed a port that was still pruning vlan 10.

Not multi-region MSTP... because in the other region we are using PVST, but there is a L3 router between them so as long as nothing crazy happens it's still good.

Honestly this is by far the best setup I could come up with. We are a development lab for new products, each team wants their own area (group of servers) to do their development working around other teams scheduled and other stuff. However we have a limited amount of hardware that everyone wants to use. So we were given the task of coming up with a network where you unplug from one teams system and plug it into the other teams system and it all just works with no differences, no spending hours re-configuring the gear, and it works great. We actually get complaints that they can't tell which system they are actually working in (can't make everyone happy it seems). So everyone's VM servers are just a clone of a "Perfect" system, if they screw it up to much we just re-clone the system.

dlots · December 09, 2015, 10:38:02 AM

It's a crazy cool system, and I have a Visio document that describes how each peace of odd gear works and how it fits in the big picture and documented the hell out of this thing, but I still pity the person who comes after me. I have a tool that generates a new VRF, all you have to do is type in some IP address that are unique to each system. Templates for each system's DMVPN spokes for the people doing off site work.

It really is (IMO) a great setup given the requirements I was given.

NetworkGroover · December 09, 2015, 10:42:20 AM

Ah - sounds great then. I incorrectly assumed it was something set up already that you walked into. Dynamic labs are always challenging environments I imagine.

dlots · December 09, 2015, 10:49:23 AM

What I walked into was way worse

No account to get into any of the gear, or Radius or anything like that

No documentation or CDP

No config backups or network monitoring (still don't have any good network monitoring

)

Nothing labeled

Over lapping subnets (No VRFs: as this wasn't intended when it happened, but no one made the guy who took the over lapping IP spaces stop using them)

4 EIGRP ASs redistributed between one another

Everything crapping out due to stuck in active issues

The core of the network was a 2800 with so much NATing thrown in there with no plan for the directions it wasn't even funny.

Nerm · December 09, 2015, 01:09:25 PM

Quote from: dlots on December 09, 2015, 10:49:23 AM
What I walked into was way worse

No account to get into any of the gear, or Radius or anything like that

No documentation or CDP

No config backups or network monitoring (still don't have any good network monitoring )

Nothing labeled

Over lapping subnets (No VRFs: as this wasn't intended when it happened, but no one made the guy who took the over lapping IP spaces stop using them)

4 EIGRP ASs redistributed between one another

Everything crapping out due to stuck in active issues

The core of the network was a 2800 with so much NATing thrown in there with no plan for the directions it wasn't even funny.

icecream-guy · December 09, 2015, 03:06:41 PM

found a terminal server connected to console ports on devices, but has no Ethernet uplink,

so I'm re-configuring the terminal server and reconnecting it to the network, we will soon have console access into those switches, since who knows how many years.....

deanwebb · December 09, 2015, 03:09:24 PM

Quote from: ristau5741 on December 09, 2015, 03:06:41 PM
found a terminal server connected to console ports on devices, but has no Ethernet uplink, so I'm re-configuring the terminal server and reconnecting it to the network, we will soon have console access into those switches, since who knows how many years.....

NetworkGroover · December 09, 2015, 03:38:00 PM

Quote from: ristau5741 on December 09, 2015, 03:06:41 PM
found a terminal server connected to console ports on devices, but has no Ethernet uplink, so I'm re-configuring the terminal server and reconnecting it to the network, we will soon have console access into those switches, since who knows how many years.....

Lol.... that's hilarious.

It's like... "SCORE!"

NetworkGroover · December 09, 2015, 03:40:02 PM

Quote from: dlots on December 09, 2015, 10:49:23 AM
What I walked into was way worse

No account to get into any of the gear, or Radius or anything like that

No documentation or CDP

No config backups or network monitoring (still don't have any good network monitoring )

Nothing labeled

Over lapping subnets (No VRFs: as this wasn't intended when it happened, but no one made the guy who took the over lapping IP spaces stop using them)

4 EIGRP ASs redistributed between one another

Everything crapping out due to stuck in active issues

The core of the network was a 2800 with so much NATing thrown in there with no plan for the directions it wasn't even funny.

Lol wow.... damn.... even when I was just barely learning what a VLAN was, one of the first things I did when I was a desktop support guy wanting to learn networking was map out all the ports and diagram it. It turned out to help when a port on a switch was link-flapping repeatedly, and I found the culprit was an unused desk phone constantly rebooting. Would have taken forever to find that had I not had it mapped out. Granted, this was a smaller company.

deanwebb · March 04, 2016, 05:24:36 PM

I just wrote thank-you notes and follow-up conversation requests to the interesting people I met at RSAC.

Now to put together my RSAC 2016 reading list. There's some good stuff on it, and I hope you lot will enjoy at least some of it.

deanwebb · March 18, 2016, 10:29:02 AM

We turned on NAC yesterday.

We caught a haxxor today.

Well, it was more just a consultant plugging his laptop into the LAN, BUT THAT IS HOW THEY START!