Applications and networking

[wintermute000]

So I have another consulting gig where the customer wants help with, you guessed, it, stretched L2 between DCs.... RAGE RAGE RAGE
re-reading my ipspace.net notes and came across this hilarious comic (yes, Ivan says stretched L2 is bad LOL)

[that1guy15]

For the longest time I never understood the meaning of "The customer does not know what they want", But I see it all the time now. And most of the time shitty-ass applications are dictating the BS such as above. .

But this is the real world and shitty applications are not going away. And usually design starts with the application so you have no kick back on it. The best you can do is layout your design with all the failure scenarios and pitfalls. Have there leadership sign-off on it and be done. At the end of the day its their network.

[wintermute000]

the problem is that a lot of them don't understand what they're signing off on. What pointy head is going to care about stretched L2 single failure domain or hairpin traffic flow or possible split brain? All they know is that it all works - until it doesn't and their in house guys just claim innocence 'the consultant put this together we have no idea', conveniently forgetting they signed off on the complexity and/or lack of robustness.

[deanwebb]

Tell them that's how the US OPM databases got hacked, through a stretched L2 line that the Chinese easily compromised. It's not in the news, but you know this guy... Then, push a rather thick waiver of liability across the table and ask the client to sign off on it so that you're not responsible when the security breach happens.

Maybe that will work?

[that1guy15]

the problem is that a lot of them don't understand what they're signing off on. What pointy head is going to care about stretched L2 single failure domain or hairpin traffic flow or possible split brain? All they know is that it all works - until it doesn't and their in house guys just claim innocence 'the consultant put this together we have no idea', conveniently forgetting they signed off on the complexity and/or lack of robustness.

This should be handled by your sales or customer relations team. Yeah a few customers are ass-holes but they burn their bridges pretty quick. If this is not the case then maybe its a red-flag and time to find someone who treats you better.

[Dieselboy]

I'll ALWAYS say this "Horses for courses.".

Stretched L2 between DCs may be bad in a big scheme of things, but when a company doesn't have much $$$ and they're just setting up then this may be a good fit for them to get things up and running. Then later down the line as things expand, upgrades / modifications could be made. The benefit with this model is that the customer could get what they want for a lower cost but with increased risks (their financial status might outweigh the risks), and another benefit is that your consultant gig will need your expertise a bit down the line when they expand and their network needs to be moulded into something more robust. Everyones a winner.

In my old job back in the UK, we took space in two datacentres. We set up a shared multi-tenanted hosted platform with really basic kit and stretched VLANs for an active / standby set up per customer. Initially we had one customer on the hosted platform. The network was built with a few spare 3750G switches. I think 4 in total as the core of the network. We also had some 2960's connecting the internet circuits to the ASA firewalls. This was real basic stuff and there were risks involved and the bases were covered as best as possible. We even did the QinQ on our equipment at both sides and so we had mini (10cm or so) crossover cables going out and in the same switch. This was back in 2010 ish. Now the network has grown quite a bit and is nothing like the original design. Around 100 customers now. But if they had of initially planned to house 100 customers from the beginning, the cost was astronomical, not only that but the design was extensive and would have been very timely to put together. After all that, what if no customers signed up?

My feeling is that make all the risks known, and plan / work out diagrams how your network can migrate to better things as the need arises.

The last bit is, sometimes the shit needs to hit the fan before the customer takes action. But if you've already laid all the details out in front of them you can clearly show that those events can be avoided etc etc. They usually get the work done.

good luck

[deanwebb]

Dieselboy is right. Although we want to keep the poo from coming into contact with the rotary oscillator, the fact is that much of our job is keeping the sanitary wipes handy so that we can clean up after such an event - and in so doing, we'll have justification for fixing a few things to keep more frequent risks from happening again.

[Dieselboy]

Ideally that wouldn't be the case though. I've frequently spent a few hours discovering SNMP MIB's so that I can make custom thorough SNMP checks to poll things like default route, HSRP state, OSPF route's etc. I used to get into a routine that the big works which were done on site and the failover testing that you do after to QA it all, the next working day I'd remote back in to the network and check everything over and then set up monitoring. Sometimes it would take ages though, as the monitoring was only a basic SNMP poller.
To everything there's advantages and disadvantages. With this, we were alerted before there was any actual issue and so we could fix it before the customer even noticed anything. The down side to this is that they never see any justification to spend money to get things working a better way. On the other hand, any work is billable anyway so I'd hope the CRM would approach them with the costs and say something like "spend X amount and these frequent recurring costs will disappear".