Cisco Webex Teams (Spark) Fully down 25/09/18 - 26 hours and counting

Dieselboy · September 25, 2018, 10:04:21 PM

Yesterday morning at just before 9am Australia Western Time 25/09/10, Cisco Webex Teams (Spark) and associated devices (hardware devices and meetings using the spark / teams platform) went fully down.

Reference: https://status.ciscospark.com/

At the time of writing (11am, 26/09/18), all services are still down while the hardware devices are kind of working, but not really. I was able to dial into my Webex PMR (personal meeting room) from the webex board but that was all I could do. Scheduled meetings have either been cancelled or moved to someones PMR or Google Hangout.

Our organisation is back to using Google Hangouts via our corporate google apps for messaging. since yesterday morning. Now into the second day of the outage, and I'm pretty concerned.

The length of this outage which is ongoing gives me a feeling that this is not a hardware fault. My feeling is that there has been some data security breach and Cisco has put a halt on the platform while they work on it to reduce further risk.

icecream-guy · September 26, 2018, 05:35:37 AM

Ahh, they probably followed others and dropped the service without telling anyone...

Dieselboy · September 26, 2018, 10:07:10 PM

Messaging came back after 31 hours but as we are over 50+ hours since the event, most things are still flaky. Messaging is mostly working but some users cannot message each other. If you create a room and add the user that you cannot message, then you can message in the room and they can see and reply.

deanwebb · October 19, 2018, 01:34:15 PM

Quote from: Dieselboy on September 26, 2018, 10:07:10 PM
Messaging came back after 31 hours but as we are over 50+ hours since the event, most things are still flaky. Messaging is mostly working but some users cannot message each other. If you create a room and add the user that you cannot message, then you can message in the room and they can see and reply.

Any explanation from Cisco regarding the outage?

mmcgurty · October 19, 2018, 10:04:21 PM

Automation nuked their own hosts!

https://www.theregister.co.uk/2018/10/03/cisco_webex_outage_script/

deanwebb · October 20, 2018, 06:20:56 PM

'This was a process issue, not a technical issue'

Dieselboy · October 28, 2018, 09:28:28 PM

ps. issue still not fully resolved. Some people can't message others in our org. And some group chats are not loading.

icecream-guy · October 30, 2018, 06:05:18 AM

'nother article on issue

https://www.networkworld.com/article/3310611/lan-wan/cisco-webex-outage-collaboration-service-finding-a-lumpy-recovery.html

Dieselboy · October 30, 2018, 10:12:02 PM

I'd be surprised if finance authorise our payment for this shocking service delivery to date.

I am now getting responses from TAC that our other random issues are due to webex back end maintenance... It's in the middle of our day. What is more frustrating is that these are clear service outages in the middle of the day. Furthermore, the debugs I am able to do from the client side are showing me indications that the cause could be back end maintenance and I have repeatedly asked "are you guys doing maintenance at the time of our issues, could this be the cause of our issues?" and they do not ever get back to me. I have so far had HTTP responses 500 as well as 429. TAC confirmed maintenance at my outage periods this week. Not happy, when I have been complaining of random service outages for 18 months.

I'm getting more complaints from users that they cannot message some other users. This is knock-on from the outage beginning 25/09.

deanwebb · November 05, 2018, 04:15:38 PM

Time to call Skype...