Internet routing issue from Sri Lanka to OZ...again - but this one's weirder

Started by Dieselboy, November 15, 2018, 12:19:40 AM

Previous topic - Next topic
Print
Go Down Pages 1 2
User actions

Dieselboy

#15
December 07, 2018, 02:44:16 AM
One of the ISPs confirmed they done the change overnight. Have not heard anything at all from the other, so I don't know if they were successful in coordinating it. This morning I still had pending alerts and investigation showed the original UDP issue still present but now also ESP traffic being blocked from SL to Australia. So, no VPN at all. Phase 1 was working, phase 2 came up but one side transmitting packets and they were not being received.
I emailed them to give the status. After a short while they said they done something and can I check... All issues are resolved now. Both VPNs working and TCP packet loss issue also resolved.

They said they are still working with vendor tac for an RCA. They hinted at a bug. If it's confirmed bug I'll try and get the bug ID.  :twitch:

deanwebb

#16
December 07, 2018, 07:12:26 AM
If it's not a bug "Try it now" and 100% function indicates somebody fat-fingered a setting that usually doesn't get mis-set.
Take a baseball bat and trash all the routers, shout out "IT'S A NETWORK PROBLEM NOW, SUCKERS!" and then peel out of the parking lot in your Ferrari.
"The world could perish if people only worked on things that were easy to handle." -- Vladimir Savchenko
Вопросы есть? Вопросов нет! | BCEB: Belkin Certified Expert Baffler | "Plan B is Plan A with an element of panic." -- John Clarke
Accounting is architecture, remember that!
Air gaps are high-latency Internet connections.

Dieselboy

#17
December 09, 2018, 07:42:50 PM
 :XD:

I was watching this over the weekend. Seems they have done some more changes but havent heard from them. During the "try it now, I've tried something" scenario, RTT between sites was 400ms. Then 24 hours later, had a bunch of "site down" alerts for a brief moment. Once everything came up again, latency is back down to around 250ms however the issue is still resolved.

So looks like they have fixed it. I'll try and reach out to them.

Dieselboy

#18
December 09, 2018, 09:07:31 PM
They reached out to me first and I now have a bug ID. Well done you guys for figuring the root cause out from the issue description! Providing that the term "bundle" relates to a type of port channel ?

QuoteCEF entries pointing to incorrect bundle members for bundle-pos and bundle-ether
CSCvf99038

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf99038


PS. the guys who write the bug detail should do a better job imo.

Otanx

#19
December 10, 2018, 09:20:55 AM
Quote from: Dieselboy on December 09, 2018, 09:07:31 PM
PS. the guys who write the bug detail should do a better job imo.

Yep. The way I read that it is just a display issue with the show command.

-Otanx
Print
Go Up Pages 1 2
User actions