General Category => Forum Lobby => Topic started by: deanwebb on September 08, 2015, 10:09:38 AM
Title: Current frustration...
Post by: deanwebb on September 08, 2015, 10:09:38 AM
We had a VPN go down, rebuilt it. Now comes the user acceptance testing...
ME: Can you run your application? USER: No. ME: Can you show me a screen shot of the error? USER: Here. (Screen shot is a generic error, not much use.) ME: Can you ping the server? USER: I am unable to perform that task. ME: (pause) Do you mean that you cannot run a ping test, or that when you run a ping test, it fails? USER: Let me copy another user into this thread. He's more technical. MORE TECHNICAL USER: Hello. Are you able to provide directions to run this test?
:facepalm1:
Title: Re: Current frustration...
Post by: Nerm on September 08, 2015, 12:27:07 PM
:phone: :frustration: :facepalm3:
Title: Re: Current frustration...
Post by: deanwebb on September 08, 2015, 12:52:19 PM
Exactly.
Follow up:
MORE TECHNICAL USER: The ping didn't work. ME: What's the IP address of that machine? You can find it using "ipconfig" in the same command line. MORE TECHNICAL USER: It's 10-1-2-3. ME: (sees the dashes) :facepalm2:
ME: (after recovering) OK, the firewall doesn't permit that device to connect to the system. If you can find the device at 10.1.2.4, it should work. MORE TECHNICAL USER: OK, the ping worked from that device. ME: Awesome. You should be able to run the application. MORE TECHNICAL USER: Nope. Same error. ME: (checks firewall) Well, we allow PING and SSH from that IP. You need to check your documentation under "firewall permissions" to see what else is needed.
:notthefirewall:
Title: Re: Current frustration...
Post by: Ironman on September 09, 2015, 08:52:48 PM
Ohhhh, this is sooooooo perfect for how I feel every time someone calls me with a Network Issue!
That was me after the ping was successful, but they said their application didn't work... and then mentioned that their version of Windows was unlicensed.
Title: Re: Current frustration...
Post by: deanwebb on September 17, 2015, 07:07:12 PM
New frustration...
PM: Get me a report on everything that would be blocked if we turned NAC on full enforcement today.
Me: Holy crap, that would include all the phones, printers, wireless access points...
PM: Wait, don't we have those on MAC bypass lists?
Me: No, we're not doing 802.1X.
PM: Well, get them on a MAC bypass list!
Me: But we're not doing full enforcement right now. Most of our Windows devices would be blocked, as well.
PM: Don't they have the NAC client?
Me: Some of them do, from when we did the pilot. It's not officially part of the build.
... and on and on and on and on...
EVERY DAY I have this conversation when the PM asks for the "full enforcement report". The PM wants to show to higher-ups that we've "made progress" because we're blocking *something*.
PM: Just show me a list of all the devices that would be blocked. It's a simple request.
Me: We haven't even formally defined the criteria for blocking. Essentially, the list would be zero devices. I could make a variety of reports with different blocking criteria, and you could pick the one you liked the best.
PM: Now you're being a smartass.
What? Me? A smartass?
:steamtroll:
Title: Re: Current frustration...
Post by: hizzo3 on September 17, 2015, 09:06:41 PM
Ha ha chart-ology 101. Oh how I loathe it.
Title: Re: Current frustration...
Post by: icecream-guy on September 18, 2015, 07:19:34 AM
Me: We haven't even formally defined the criteria for blocking. Essentially, the list would be zero devices. I could make a variety of reports with different blocking criteria, and you could pick the one you liked the best.
Sounds like you need to spend the day building a flow chart in Visio of all the blocking criteria and the devices that could/would be blocked under the different scenarios. (something like today's XLCD comic strip)
Title: Re: Current frustration...
Post by: deanwebb on September 18, 2015, 09:56:05 AM
Actually, I think I will put together a bunch of rules for information gathering, so we can see all the layers of enforcement and how they'd interact. Should be fun.
Also, leaving work early today might also be fun. COMP TIME FTW!
Title: Re: Current frustration...
Post by: Dieselboy on September 25, 2015, 03:45:14 AM
I kept receiving strange webex meeting invites from one of the HR girls who works in another office. I mentioned it to my guy with rolled eyes, who told me he also received the invites. Then I noticed another guy looking over here, I asked him did he also receive it and answer was yes. So I messaged her on jabber asking her if she was aware she was sending odd meeting invites from webex, and her answer was no, it's not something she is doing.
So now, alarm bells are ringing, and she's not replying to my jabber messages either. So I call her, and we have the same conversation over the call and she says shes not using webex.
So I explain someone must have got into her account, she must change her password. I also say I'm logging in to block her account.
Then she says, actually no, she's in webex.
:developers:
It's always the same with this HR girl.
Another situation::
I'd set her up with a desk phone. The whole office only has 2 desk phones, the other one should be on a hot desk and has extension mobility enabled. I noticed one day, calls kept being placed from her from the hot desk phone. I logged in to CUCM and her phone that I'd set her up with is unregistered. I asked her to swap the phones over back to her desk and keep both phones plugged in because I'm doing upgrades (this was the other week). I explained this so many times I've lost count. So this week, I still see that her phone is unregistered and the hot desk phone is up and running on her desk still. So, since all the upgrades were complete I can just do some config work and swap the configs over between the phones. This would save her unplugging her phone, so this is what I did. A few hours later I get a message from her "how long is this phone thing going to take because it's showing a message on the screen and it's mostly blank". I log in and the phone I had now configured for her was unregistered. The hot desk phone was still pending some config and so it was not fully set up.
Why did she do this? Why did she come in, in the morning, and see the desk phone with her own name and extension number on the screen and think "I know what I'll do this morning!". It was also too much aggravation for her to reverse what she had done (plug / unplug two phones) so I undone the config I had done earlier that day.
Hot desk phone still unregistered.
:developers:
Title: Re: Current frustration...
Post by: SimonV on September 25, 2015, 07:48:34 AM
On all applications that have an ALG (inspection) :professorcat:
Title: Re: Current frustration...
Post by: deanwebb on September 25, 2015, 08:47:19 AM
... that moment when you discover that the clients in the CBD area are nothing at all like the clients in the production environment...
:wall:
CBD guys are all virtualized... and I think they either have the services I need for NAC turned off, or there's a VBlock firewall blocking traffic on the ports I need. Either way, that's not what we have in the real world, even though CBD is supposed to be just like our production environment...
Title: Re: Current frustration...
Post by: icecream-guy on September 28, 2015, 08:01:20 AM
Customer want's us to update firewall rules.....They don't want to deploy a host firewall
...to stop servers on the same subnet from communicating.
ok, we'll use a couple of port ACL's on the interfaces to make it work.
...come to find out the servers are virtual in the ESX environment, so once a server gets moved....
guess what? they're going to deploy host firewalls.
Title: Re: Current frustration...
Post by: Reggle on September 28, 2015, 08:31:23 AM
VLAN per application and full access in VLAN. I don't see the problem. Nor do I see a problem with host firewalls. Smells like someone wanted to be original and be noticed by his superiors for coming up with that.
Title: Re: Current frustration...
Post by: NetworkGroover on September 28, 2015, 10:55:36 AM
CBD guys are all virtualized... and I think they either have the services I need for NAC turned off, or there's a VBlock firewall blocking traffic on the ports I need. Either way, that's not what we have in the real world, even though CBD is supposed to be just like our production environment...
OHHHHHHH man don't get me started on that... I worked at a place where the ENTIRE point of the ENTIRE BUILDING was to test new solutions in a mock-up environment that was supposed to exactly match what was in production. It was FAR from that, and as someone who worked in engineering it was insanely frustrating trying to deal with that and then trying to deal with the pissed off (and rightly so) operations guys because yet again there was some big hang-up with trying to implement an engineered solution into production.
Good Lord.
Title: Re: Current frustration...
Post by: deanwebb on September 28, 2015, 01:26:35 PM
Got CBD to be working "good enough" and then tested it on my own box in production.
Ready for the corporate rollout.
:yeahright:
Title: Re: Current frustration...
Post by: wintermute000 on September 28, 2015, 04:50:36 PM
Are your vmware jockeys aware that a distributed vswitch can do ACLs
Title: Re: Current frustration...
Post by: deanwebb on September 28, 2015, 05:10:40 PM
Reading that makes me feel not so bad about the periodic java issues with ASDM... and makes me LOVE my Tufin even more.
Fun fact: Tufin was made by former Checkpoint guys who were tired of the way those firewalls update. It provisions code and ships it out to the firewalls very nicely. I recommend it strongly as a very delighted customer.
Title: Re: Current frustration...
Post by: SimonV on January 08, 2016, 02:05:40 PM
Bought myself a shiny new EX-2200C switch so I start adding all the basics, snmp strings, syslog, etc. Suddenly it crashes and reboots, then this comes up at boot
QuoteUNEXPECTED SOFT UPDATE INCONSISTENCY
CLEAR? yes
UNKNOWN FILE TYPE I=40704 UNEXPECTED SOFT UPDATE INCONSISTENCY
CLEAR? yes
UNKNOWN FILE TYPE I=40705 UNEXPECTED SOFT UPDATE INCONSISTENCY
Title: Re: Current frustration...
Post by: deanwebb on January 08, 2016, 02:52:25 PM
:haha1:
Looks like you got a problem there, son.
Title: Re: Current frustration...
Post by: Dieselboy on January 11, 2016, 01:14:03 AM
From a chat, earlier: Users laptop had his laptop name changed due to an incorrect PTR record.
Title: Re: Current frustration...
Post by: Dieselboy on January 11, 2016, 01:18:41 AM
From just now. Different user, same location.
Title: Re: Current frustration...
Post by: deanwebb on January 11, 2016, 07:16:34 AM
:facepalm1:
I believe that this is the most appropriate facepalm for you today, sir.
For me, it's this one...
:ckfacepalm:
... as I wait the 20 minutes it takes for my java-based user console to initialize.
Title: Re: Current frustration...
Post by: Dieselboy on January 11, 2016, 09:12:02 PM
Java based management applications :developers: :glitch:
Why do they have to make them in java? Java gets out of date a week after they release a version. At which point you can't run the management app. unless you update. When you update you find that the new version is not compatible with the management application. Java to put it simply is completely useless. I still cannot connect to one of my netapp controllers because I can't get oncommand system manager to run properly.
Java based applications that give me grief:
Effing all of them
All of them I say!
Title: Re: Current frustration...
Post by: deanwebb on January 11, 2016, 09:32:13 PM
How about the hardware that shipped a year ago, that you're just now deploying... you have to get an old version of the app with an old version of Java to connect to it so that you can upgrade the software on the device... and then, once you upgrade, you have to switch to the new app with a new version of Java...
But the worst part? You have to do this with five other boxes, so you have to keep a PC with one version of the client and Java and another PC with the other version(s)!
:rage:
Title: Re: Current frustration...
Post by: Dieselboy on January 11, 2016, 11:15:49 PM
This is why I made a windows 7 virtual machine that I can keep copying and spinning up. Windows is not even activated, don't care. I'll delete it once I'm done with completely ruining it with the different java versions.
Whats more, Chrome has dropped java support. I hope that this move is either a reflection that people are dropping java, or a push to get people to drop java. I've had Java ruin my days a number of times since 2007 when I started doing networking to earn money. Would be nice to on-charge those lost hours to Java or the company using java to make their management applications :)
Title: Re: Current frustration...
Post by: SimonV on September 06, 2016, 03:29:01 AM
So apparently Juniper is now no longer shipping rack mounts as standard on the EX2300 switches, what a rip-off :rolleyes:
Title: Re: Current frustration...
Post by: deanwebb on September 06, 2016, 07:43:12 AM
@SimonV: Ouch. Cheapskates.
I just stopped and restarted a service to demonstrate that stopping and restarting this service won't solve this particular problem.
Title: Re: Current frustration...
Post by: icecream-guy on September 06, 2016, 10:47:31 AM
you got alot of work to do, but can't really do it all yourself You need others action so you can move your task forward. other have their own priorities. and you sit on the back burner. waiting until they find the time.
Troubleshooting a N5K issue and need to do packet captures. First it's only fiber ports, so that out. then I find and 1G RJ45 SFP, next my laptops are old and don't do 1G connection, so that's out the server connects at 10G, so capturing packets looking for dropped packets on a 100Mb interface, there will be dropped packets for sure Next I had to wait for the cable guys to run cable next I'm waiting for port assignments on the gigamon..for like 6 days now... :drama: /VENT
Title: Re: Current frustration...
Post by: icecream-guy on September 07, 2016, 11:08:57 AM
Quote from: ristau5741 on September 06, 2016, 10:47:31 AM you got alot of work to do, but can't really do it all yourself You need others action so you can move your task forward. other have their own priorities. and you sit on the back burner. waiting until they find the time.
Troubleshooting a N5K issue and need to do packet captures. First it's only fiber ports, so that out. then I find and 1G RJ45 SFP, next my laptops are old and don't do 1G connection, so that's out the server connects at 10G, so capturing packets looking for dropped packets on a 100Mb interface, there will be dropped packets for sure Next I had to wait for the cable guys to run cable next I'm waiting for port assignments on the gigamon..for like 6 days now... :drama: /VENT
/unvent then the server they planned to send the data to, is out of ports, so they send it to another server. and I don't have an account on that server. so Now I will fill out paper work for a new account, submit, and wait for approval and then the linux server admin creates an account for me so I can login. /vent
Title: Re: Current frustration...
Post by: deanwebb on September 07, 2016, 11:24:03 AM
Massive re-org in the works... announced today that all associates will know their new reporting structure by the end of the year.
:ckfacepalm:
Ohhhhhhhhhhhhhhhhh kaaaaaaaaaaaaayyy...
Title: Re: Current frustration...
Post by: LynK on September 07, 2016, 12:54:50 PM
@ristau5741
What issues are you seeing?
Title: Re: Current frustration...
Post by: burnyd on September 07, 2016, 01:29:41 PM
I hate my macbook
Title: Re: Current frustration...
Post by: SofaKing on September 07, 2016, 01:35:39 PM
Just had a meeting about our re-org yesterday. We had a new VP come in about 6 months ago. Luckily he sat back and watched before making any changes. I am actually liking the changes being made. A couple I question but the one that affects me I am happy about.
Title: Re: Current frustration...
Post by: jericho on September 07, 2016, 01:45:56 PM
Underfloor heating, specifically the one I've just found in the area this months employer have earmarked as their new comms room.
Title: Re: Current frustration...
Post by: icecream-guy on September 08, 2016, 08:01:32 AM
Today's frustration: Email threads that fracture and go 5 different ways, all with the same subject subject and different recipients regarding things that I need to keep track of.
Title: Re: Current frustration...
Post by: deanwebb on September 08, 2016, 09:17:31 AM
I'm dealing with Shark Week on my IM... one guy found me about 15 minutes ago, then 3 managers, 2 project managers, and 4 co-workers sensed the blood in the water and hit me up on IM, one after the other BAMBAMBAMBAMBAMBAMBAMBAMBAMBAMBAMBAMBAMBAMBAM!
:frustration:
Title: Re: Current frustration...
Post by: wintermute000 on September 08, 2016, 11:07:04 PM
PSA: Palo Alto enforces eBGP split horizon when talking to multiple peers from the same ASN.
i.e. Switch (AS1) --> Palo (AS2) --> Switch (AS1) - Palo does not send the route on until you change one of the switches to a different AS.
Guess what happens when you run VRFs on any switch, any vendor. (hint: one BGP process....)
Support says log a feature request....
No, this is not RFC behaviour, there should not be any split horizon in eBGP YOU FIREWALL MUPPETS IF YOU'RE GOING TO DO A PROTOCOL FSCKING DO IT PROPERLY :flipdesk:
Title: Re: Current frustration...
Post by: deanwebb on September 09, 2016, 08:37:12 AM
Cisco does one better and simply has the ASA not participate in dynamic routing very much, if at all.
:challenge-denied:
And now I have a new frustration... the upgrade window we had scheduled has now been indefinitely postponed...
Title: Re: Current frustration...
Post by: NetworkGroover on September 09, 2016, 10:58:32 AM
i.e. Switch (AS1) --> Palo (AS2) --> Switch (AS1) - Palo does not send the route on until you change one of the switches to a different AS.
Guess what happens when you run VRFs on any switch, any vendor. (hint: one BGP process....)
Support says log a feature request....
No, this is not RFC behaviour, there should not be any split horizon in eBGP YOU FIREWALL MUPPETS IF YOU'RE GOING TO DO A PROTOCOL FSCKING DO IT PROPERLY :flipdesk:
If that wording is defined in an RFC, did you provide them the exact text?
Title: Re: Current frustration...
Post by: icecream-guy on September 09, 2016, 11:01:06 AM
Well, they can just kiss my rosy hind quarters. They taste like strawberries. Honest. Just try.
Meanwhile, I got a bigger frustration... spent the better part of a week rebuilding my topology in Tufin and I come to discover that I DID IT ALL WRONG AND HAVE TO DO IT OVER
:rage: :ivan: :frustration: :flipdesk: :no:
Title: Re: Current frustration...
Post by: wintermute000 on September 10, 2016, 06:12:20 AM
Why would you route on a firewall? use a router, let the firewall block/permit traffic.
Lots of reasons
- internet edge design (HSRP is not a substitute for routing) - requirement for intra-zone segmentation and there are multiple egress points from that zone (I count HSRP pairs as one... coz it is :) ) - customers don't want to buy a separate router when their firewall is supposed to work perfectly fine as one
The whole old school 'static routes only on FWs' is outdated and needs to go away, it completely shackles most routing designs including crowbaring a stupid L2 HSRP hop into an otherwise beautiful routed design. Ever had the fun of dealing with routed leaf/spine, MLAG and whoops there's a stupid L2 firewall that can't run a pair of /30s?
And BTW deanwebb, ASA's have run BGP since 9.x :) Though a little bird tells you not to send any graceful restart capabilities down a BGP peering to 9.1.x code, because the stupid ASA hexdumps the adjacency instead of just not negotiating that capability. *whistles*
Title: Re: Current frustration...
Post by: deanwebb on September 10, 2016, 08:53:09 AM
OK, so ASA runs BGP... but not very well. :mrgreen:
But if the firewall does routing, then who manages the firewall? Is it for the R&S team or for the security team? No, we better not have ACLs on the routers and no routing on the firewalls...
Title: Re: Current frustration...
Post by: wintermute000 on September 11, 2016, 06:32:47 AM
On that logic you might as well never deploy NSX or Openstack because there's too much networking in there. Or DMVPN - too much crypto for the networking team, surely...
you see my point....
I acknowledge that management demarcation may be a concern but where there's a will there's a way, and I've seen more than one (large) environment where the firewalls were driven by the networking team
Title: Re: Current frustration...
Post by: deanwebb on September 11, 2016, 10:22:38 AM
Clearly, we must strive for a more pluralistic networking society in which we do not see ourselves as "R&S people" or "Security people". I myself feel the sting of hypocrisy as I curse the developers for using hard-coded IP addresses instead of FQDNs in their code... and then build out ACLs based upon hard-coded IP addresses...
There is a huge difference between not wanting a security device to do routing because of security and not wanting it to do routing because you chose to support firewalls in order to get away from routing. And, you are bang to rights on the firewall == router situation for most SMB environments. That firewall isn't just the router, it's also the core switch. That's a lot to ask of an ASA 5505/5506, but it's the reality we face.
But, back to AspiringNetworker's comment... I really hate active/active. Because convergence.
Title: Re: Current frustration...
Post by: NetworkGroover on September 12, 2016, 11:09:49 AM
You say because convergence.. but convergence is exactly the reason why you go active/active... the more the two entities can act independently (within reason), the better. The more you have this, the more a failure "appears" to be transparent to a network. If a FW fails in a meshed routed design.. that's just one less path to take - rather than waiting for stuff to switch from passive to active, etc. Granted, I'm not a FW guy... so I could be making it sound easier than it is.
As for administration - I see it very simply as if you have separate teams to manage network and security/FW, the security/FW guy works with the network team to identify best practices for the routing piece, and the security guy maintains the other 95% of the FW's job. I don't see that simply the fact that you're routing on the FW meaning that you give the responsibility of the FWs to the network team.
Title: Re: Current frustration...
Post by: deanwebb on September 12, 2016, 11:24:49 AM
Not gonna argue... dealing with an end-of-life-do-not-resuscitate RADIUS server outage and moving all its WLCs over to the new system that will be at 100% capacity with this unscheduled move... managers be buying more gear as I type...
Title: Re: Current frustration...
Post by: NetworkGroover on September 12, 2016, 11:54:31 AM
Quote from: deanwebb on September 12, 2016, 11:24:49 AM Not gonna argue... dealing with an end-of-life-do-not-resuscitate RADIUS server outage and moving all its WLCs over to the new system that will be at 100% capacity with this unscheduled move... managers be buying more gear as I type...
Ok sorry.. I have a habit of just stating opinion, but it coming across as quasi-abrasive. Not trying to be argumentative.
Title: Re: Current frustration...
Post by: deanwebb on September 12, 2016, 12:00:48 PM
Not gonna argue... I always say that when I got my head totally into a support issue. Does not imply an argument. Just implies I have sacrificed all social skills in order to focus my brain on the issue... which, in this case, has technical, managerial, and budget concerns.
Also giving my lunch order to the guy heading off to a fast-food place concerns.
Title: Re: Current frustration...
Post by: burnyd on September 12, 2016, 02:48:54 PM
Quote from: deanwebb on September 11, 2016, 10:22:38 AM Clearly, we must strive for a more pluralistic networking society in which we do not see ourselves as "R&S people" or "Security people". I myself feel the sting of hypocrisy as I curse the developers for using hard-coded IP addresses instead of FQDNs in their code... and then build out ACLs based upon hard-coded IP addresses...
There is a huge difference between not wanting a security device to do routing because of security and not wanting it to do routing because you chose to support firewalls in order to get away from routing. And, you are bang to rights on the firewall == router situation for most SMB environments. That firewall isn't just the router, it's also the core switch. That's a lot to ask of an ASA 5505/5506, but it's the reality we face.
But, back to AspiringNetworker's comment... I really hate active/active. Because convergence.
Lol neither have enough networking. That is why OS has a bunch of 3rd part plugins that some what work.
NSX well yah...
Title: Re: Current frustration...
Post by: burnyd on September 12, 2016, 02:51:21 PM
Anyways, Active / Active is the way to go either in a clustered routed solution and share state in a routing protocol fashon. A popular one now a days is the mac redirection ie have the virtual address between FW's and the FW it self redirects the mac address session by session.
Title: Re: Current frustration...
Post by: icecream-guy on September 13, 2016, 05:42:37 AM
Also giving my lunch order to the guy heading off to a fast-food place concerns.
Whopperito?
man, I want one of those....
DO NOT WANT
He went to Panda Express. Yay orange chicken.
mmmm Orange Pandarito........
Title: Re: Current frustration...
Post by: icecream-guy on September 14, 2016, 07:20:57 AM
when you send out 2 week notice for an maintenance window. notifying users of possible outage and what will be affected the maintenance window is discussed at the CCB prior to the maintenance, and everyone is aware and on board. when the maintenance is done, customers still complain that something went down and want to know what happened. :angry: grrrr.
Title: Re: Current frustration...
Post by: deanwebb on September 14, 2016, 08:05:47 AM
Just tell them that the interruption only affected users that were torrenting furry pr0n. Then ask for the names of everyone impacted.
Title: Re: Current frustration...
Post by: Otanx on September 14, 2016, 10:36:08 AM
Quote from: ristau5741 on September 14, 2016, 07:20:57 AM when you send out 2 week notice for an maintenance window. notifying users of possible outage and what will be affected the maintenance window is discussed at the CCB prior to the maintenance, and everyone is aware and on board. when the maintenance is done, customers still complain that something went down and want to know what happened. :angry: grrrr.
I was this user recently. Saw the notice. Ignored it because it was pretty routine. They send the reminder notification the day of, and I ignored that because I was busy. They bounced my linux box, and I lost a few hours of work. I really wanted to rant at the guy who bounced my machine, but it was my fault. Also the hours of work lost actually was mostly me learning the correct command line options for curl and scp to do what I wanted so recreating it was much faster the second time.
My rant for today is property management. Got an email "Hey we can't find these devices do you know where they are?" The list is just model numbers, no serial numbers. "I have about 50 of those which ones can't you find?" Reply email has a list with location. Hmmm, nobody has touched that rack of gear for awhile. The network is up so nobody stole it. I am pretty sure it is there. They are going to go look again. I am 99% sure that they did not look at the back of the rack (you know where switches get mounted for ToR). Next week I will get the email they can't find any PDUs. This is a regular occurrence.
-Otanx
Title: Re: Current frustration...
Post by: SimonV on September 19, 2016, 02:56:50 AM
Going through a WLC at a remote location installed by a third party (Cisco partner). All APs set to power level 1, all fixed on the same channel.
:notbad:
Title: Re: Current frustration...
Post by: wintermute000 on September 19, 2016, 06:33:16 AM
are you sure it wasn't a meru partner? :p I just had a long argument with a junior colleague who had just done Meru training but no wireless knowledge in general and just could not grasp why single channel arch is a hack just to save on one piece of work (proper cell placement/channel design). Not even when I described it as 'so do you want a hub with vendor magic sauce to time the transmissions to minimise collisions, OR, a switch'. At the end I nearly threw the CWNA textbook at him.
I know the original motivation for SCA was roaming but let's face it, that's solved ('good enough') with various fast roaming techniques, rendering it a completely pointless exercise to defy the laws of physics
Title: Re: Current frustration...
Post by: icecream-guy on September 19, 2016, 10:51:56 AM
not sure if it's on my part or theirs
Them: I need 5 ips in zzz network
Me: you don't have enough available
-- time passes
Them: can we get a new network assigned?
Me: Yes
-- I do the research to find a nice little network, submit it to the keepers of the IP Address, have them create the new network in the system and have 5 ip's assigned.
-- time passes
Them: we can't reach out servers in the new network, nor ping the default gateway
-- troubleshoot (ok so I forgot to add the VLAN to the VPC trunk on the 5K's) they can ping wining their network
-- more time passes
Them: we can't access any of our resources on any other network
Me: Did you submit a firewall request?
Them: No
Me: well there are no firewall rules to allow off network traffic
- generously comb through the firewall ACLs and put in some rules that I think would work (knowing nothing about any of the hosts in the network
Me: Are you good now ?
Them: can we get a firewall audit to help us determine how the ACL rules need to be configured
Me: Ok
-- provide them with a firewall audit
-- more time passes
-- and they've already been hinting on a load balancer configuration for the new network
Title: Re: Current frustration...
Post by: deanwebb on September 19, 2016, 12:24:17 PM
Editing entries for 95 new WLCs in my RADIUS server.
:ivan:
Title: Re: Current frustration...
Post by: SimonV on September 19, 2016, 01:21:37 PM
Adding them as clients not sufficient? Are you doing anything client-specific in the policies?
Title: Re: Current frustration...
Post by: deanwebb on September 19, 2016, 02:11:01 PM
Each has its own little snowflake of a config, including hostname and RADIUS shared secret. The RADIUS server allows for an easy import of IP addresses, but not hostnames or shared secrets.
Just finished the WLCs, now it's time for those bastard autonomous APs...
:rage:
Title: Re: Current frustration...
Post by: SimonV on September 20, 2016, 02:31:22 PM
2016 and Frame Relay is still on the blueprint
:rage:
Title: Re: Current frustration...
Post by: deanwebb on September 20, 2016, 05:47:34 PM
-- I do the research to find a nice little network, submit it to the keepers of the IP Address, have them create the new network in the system and have 5 ip's assigned.
-- time passes
Them: we can't reach out servers in the new network, nor ping the default gateway
-- troubleshoot (ok so I forgot to add the VLAN to the VPC trunk on the 5K's) they can ping wining their network
-- more time passes
Them: we can't access any of our resources on any other network
Me: Did you submit a firewall request?
Them: No
Me: well there are no firewall rules to allow off network traffic
- generously comb through the firewall ACLs and put in some rules that I think would work (knowing nothing about any of the hosts in the network
Me: Are you good now ?
Them: can we get a firewall audit to help us determine how the ACL rules need to be configured
Me: Ok
-- provide them with a firewall audit
-- more time passes
-- and they've already been hinting on a load balancer configuration for the new network
mmm.. now they want remote access to all the new servers for all the admins.
Haven't they ever heard of a jump box?
Title: Re: Current frustration...
Post by: deanwebb on September 27, 2016, 08:25:18 AM
A jump box... isn't that like a bouncy castle at kids' parties?
Title: Re: Current frustration...
Post by: icecream-guy on September 28, 2016, 08:29:46 AM
no it's a security thing, I'm surprised you don't know about them. or you are just pulling my chain....
:umad:
Title: Re: Current frustration...
Post by: SimonV on October 20, 2016, 09:04:27 AM
Configured a JunOS bandwidth policer last night to prevent link saturation on a DSL line, forgot to add the permit-any term at the end. One guy even went back home
:jackie-chan:
But at least the link didn't get saturated...
:awesome:
Title: Re: Current frustration...
Post by: deanwebb on October 20, 2016, 09:39:24 AM
In training, have a design document due Friday, need to do a mandatory anti-bribery training that is overdue by 2 weeks and I have to submit my end-of-year appraisal that is overdue by 1 week.
Scheduling can be a nightmare...
:ivan:
Title: Re: Current frustration...
Post by: deanwebb on October 20, 2016, 10:54:51 AM
Oh great, I have to upgrade RAM on 21 virtual appliances ASAP or we'll sustain a systemwide outage in the next few hours/days.
Title: Re: Current frustration...
Post by: icecream-guy on October 20, 2016, 10:57:42 AM
Quote from: deanwebb on October 20, 2016, 09:39:24 AM In training, have a design document due Friday, need to do a mandatory anti-bribery training that is overdue by 2 weeks and I have to submit my end-of-year appraisal that is overdue by 1 week.
Scheduling can be a nightmare...
I'll paypal you 10 bucks if you don't do the training... :lol:
Title: Re: Current frustration...
Post by: deanwebb on October 20, 2016, 12:31:00 PM
I'm hardcore. Participating in class discussions while my manager submits emergency change requests, since I don't have authority to do so.
Title: Re: Current frustration...
Post by: Nerm on November 03, 2016, 07:55:00 AM
While discussing plans for a PoC with an SD-WAN provider they tell you that currently the only way they support exchanging routes with your on-premise is via static or RIP.
:phone:
Title: Re: Current frustration...
Post by: deanwebb on November 03, 2016, 08:05:48 AM
Quote from: Nerm on November 03, 2016, 07:55:00 AM While discussing plans for a PoC with an SD-WAN provider they tell you that currently the only way they support exchanging routes with your on-premise is via static or RIP.
:haha3:
Title: Re: Current frustration...
Post by: wintermute000 on November 03, 2016, 02:29:55 PM
Quote from: Nerm on November 03, 2016, 07:55:00 AM While discussing plans for a PoC with an SD-WAN provider they tell you that currently the only way they support exchanging routes with your on-premise is via static or RIP.
:phone:
Riverbed?
Title: Re: Current frustration...
Post by: Nerm on November 10, 2016, 08:51:39 AM
No, it was Aryaka.
*btw, sorry for the late response. Been busy lately. :)
Title: Re: Current frustration...
Post by: deanwebb on November 10, 2016, 11:11:48 AM
Current frustration: taking over network IT functions from another vendor for 80% of Multinational Megacorporation's global footprint. Said other vendor can only apply an approximate view of the network, not a complete one. Massive research effort...
Title: Re: Current frustration...
Post by: icecream-guy on January 27, 2017, 05:43:34 AM
Argh, why don't users understand the concept of stateful firewalls,
user request. source desktop <-> Server, bi directional?
I call user, Asks: do you have a need to ssh to your desktop from the server? he says he doesn't understand what I am asking. I ask user if these running a ssh server on his pc? he says not. I tell hum I'm cancelling the bidirectional flow since it's not need (i can see his puzzled look over the phone)
---
another customer asks to open port 8443 to his server hisserver.mynetwork.com I respond back request is completed. he responds back, it's not working and includes the URL to the server http://hisserver.mynetwork.com:9000/web/app
:facepalm2:
Title: Re: Current frustration...
Post by: deanwebb on January 27, 2017, 09:18:55 AM
Yes, please can you do the needful and open port 8443 so I can communicate on port 9000?
:rofl:
Title: Re: Current frustration...
Post by: SimonV on January 30, 2017, 07:18:32 AM
Working on a warehouse wifi refresh, found this on 4 of the 10 standalone access points:
XXXAP03#sh ip int br Interface IP-Address OK? Method Status Protocol BVI1 xxx.xxx.xxx.xxx YES NVRAM up up Dot11Radio0 unassigned YES NVRAM administratively down down Dot11Radio0.1 unassigned YES unset administratively down down Dot11Radio0.10 unassigned YES unset administratively down down Dot11Radio0.33 unassigned YES unset administratively down down Dot11Radio0.34 unassigned YES unset administratively down down Dot11Radio1 unassigned YES NVRAM administratively down down FastEthernet0 unassigned YES NVRAM up up FastEthernet0.1 unassigned YES unset up up FastEthernet0.10 unassigned YES unset up up FastEthernet0.33 unassigned YES unset up up FastEthernet0.34 unassigned YES unset up up
:yuno:
Title: Re: Current frustration...
Post by: EOS on January 30, 2017, 08:14:07 AM
LOL
Title: Re: Current frustration...
Post by: deanwebb on January 30, 2017, 08:16:39 AM
:wtf:
and
:zomgwtfbbq:
Title: Re: Current frustration...
Post by: SofaKing on January 30, 2017, 12:56:25 PM
Walk ups... m-fing walk ups :developers: :developers: :developers:
On the three it didn't work for, it totally hosed up our guest wireless environment.
:wall: :wall: :wall: :wall: :wall: :wall:
Fixing that now and making a note to never issue broad, all-inclusive instructions again...
now you can understand why we only do 1 task on 1 device per maintenance window. short windows, lots of verification, time to roll back, and reverify everything. makes sure nothing else impacts the maintenance, and if something does, it can only be 1 thing.
Title: Re: Current frustration...
Post by: NetworkGroover on March 02, 2017, 04:28:44 PM
On the three it didn't work for, it totally hosed up our guest wireless environment.
:wall: :wall: :wall: :wall: :wall: :wall:
Fixing that now and making a note to never issue broad, all-inclusive instructions again...
now you can understand why we only do 1 task on 1 device per maintenance window. short windows, lots of verification, time to roll back, and reverify everything. makes sure nothing else impacts the maintenance, and if something does, it can only be 1 thing.
Think you can probably find a better way to handle things.. that won't a hundred change control windows. Of course, easy for me to say.
Why did the same process break three devices while wasn't an issue for the other forty?
Title: Re: Current frustration...
Post by: deanwebb on March 02, 2017, 06:51:37 PM
40 of the devices needed an *internal* web server cert. 3 of the devices needed an *external* cert, but since I didn't leave them out of the instruction to "get an internal web cert on all these boxes!", they got an internal cert that invalidated the external cert when they had to have their private key regenerated for the internal cert...
Title: Re: Current frustration...
Post by: icecream-guy on March 03, 2017, 05:56:48 AM
On the three it didn't work for, it totally hosed up our guest wireless environment.
:wall: :wall: :wall: :wall: :wall: :wall:
Fixing that now and making a note to never issue broad, all-inclusive instructions again...
now you can understand why we only do 1 task on 1 device per maintenance window. short windows, lots of verification, time to roll back, and reverify everything. makes sure nothing else impacts the maintenance, and if something does, it can only be 1 thing.
Think you can probably find a better way to handle things.. that won't a hundred change control windows. Of course, easy for me to say.
As large and complicated that our network is, and the lack of knowledge about who connected what where, and as loud as the customers yell when their stuff goes down, even when we notify them. Best to play it safe.
Title: Re: Current frustration...
Post by: SimonV on March 08, 2017, 08:26:01 AM
Every time we've tried to script it, we failed. Heck, even ASDM can't do it on an ASA, had to go to the CLI to get the job done.
What is your secret, man? WE MUST KNOW!!!
Title: Re: Current frustration...
Post by: SofaKing on March 08, 2017, 02:32:37 PM
Performance reviews - I hate writing these up. Usually multiple objectives are the same as the previous but you have to write a small book for each objective so it becomes very redundant.
Title: Re: Current frustration...
Post by: wintermute000 on March 08, 2017, 02:41:46 PM
python using raw commands. Definitely python as you can manually invoke pause. Of course you'd need at least telnet
Title: Re: Current frustration...
Post by: dlots on March 08, 2017, 02:50:12 PM
If you are trying to do telnet you might look at telnetlib
I did a super crapy program to telnet into devices and try and find some info.
Every time we've tried to script it, we failed. Heck, even ASDM can't do it on an ASA, had to go to the CLI to get the job done.
What is your secret, man? WE MUST KNOW!!!
cheat mode ON as using netmiko module, but if I had time I could work it out with a manual library as I've done plenty of expect style pain before netmiko was a thing
ssh://ansible@192.168.145.129:22/usr/bin/python -u /home/ansible/generate-ssh-key/generate-ssh-key.py config term Enter configuration commands, one per line. End with CNTL/Z. WAN1(config)#crypto key generate rsa modulus 2048 % You already have RSA keys defined named WAN1.cisco.com. % They will be replaced.
% The key modulus size is 2048 bits % Generating 2048 bit RSA keys, keys will be non-exportable...end WAN1# config term Enter configuration commands, one per line. End with CNTL/Z. WAN2(config)#crypto key generate rsa modulus 2048 % You already have RSA keys defined named WAN2.cisco.com. % They will be replaced.
% The key modulus size is 2048 bits % Generating 2048 bit RSA keys, keys will be non-exportable... [OK] (elapsed time was 2 seconds)
WAN2(config)#end WAN2#
If this was serious you could easily whip up something to convert a CSV into a list of dictionary for the variables e.g. 1 per row
Title: Re: Current frustration...
Post by: icecream-guy on March 09, 2017, 06:10:09 AM
Quote from: SofaKing on March 08, 2017, 02:32:37 PM Performance reviews - I hate writing these up. Usually multiple objectives are the same as the previous but you have to write a small book for each objective so it becomes very redundant.
LOL I haven't had a performance review in like 5 years. no perfomance review = no planned career growth..no meager raises. no self recommendations on how I can improve myself, no jabs at my performance, etc. is it worth it? :steamtroll:
Title: Re: Current frustration...
Post by: deanwebb on March 09, 2017, 08:45:35 AM
I have to enter my objectives for 2017 today. That means copying and pasting the objectives our upper management said we should have. I'm not a manager, yet I have "Retain employees and reduce turnover" as a goal. I guess if I stay here through the next review cycle, I'll be 100% in that area.
:facepalm3:
Title: Re: Current frustration...
Post by: Otanx on March 09, 2017, 09:25:00 AM
To add to Wintermute's suggestions you can also create SSH keys using SNMP. On Cisco it is OID - 1.3.6.1.4.1.9.9.339.1.1.2.1 so as long as you have a SNMP user setup you can script this without relying on SSH working before generating the key.
Cisco SNMP Object Navigator for that OID: http://snmp.cloudapps.cisco.com/Support/SNMP/do/BrowseOID.do?local=en&translate=Translate&objectInput=1.3.6.1.4.1.9.9.339.1.1.2.1#oidContent
-Otanx
Title: Re: Current frustration...
Post by: wintermute000 on March 09, 2017, 02:47:29 PM
My script didn't need ssh working. It uses telnet The ssh in the output is me running the python remotely LOL
Title: Re: Current frustration...
Post by: wintermute000 on March 09, 2017, 08:43:50 PM
version 2. put your devices in a csv with the following headers. and yes, it uses telnet (observe device_type: cisco_ios_telnet)
***Executing script on device: {'username': 'cisco', 'ip': '172.17.1.151', 'secret': 'cisco', 'host': 'WAN1', 'device_type': 'cisco_ios_telnet', 'password': 'cisco'} ***Script Output: config term Enter configuration commands, one per line. End with CNTL/Z. WAN1(config)#crypto key generate rsa modulus 2048 % You already have RSA keys defined named WAN1.cisco.com. % They will be replaced.
% The key modulus size is 2048 bits % Generating 2048 bit RSA keys, keys will be non-exportable... [OK] (elapsed time was 1 seconds)
WAN1(config)#end WAN1# ***device end*** ***Executing script on device: {'username': 'cisco', 'ip': '172.17.1.152', 'secret': 'cisco', 'host': 'WAN2', 'device_type': 'cisco_ios_telnet', 'password': 'cisco'} ***Script Output: config term Enter configuration commands, one per line. End with CNTL/Z. WAN2(config)#crypto key generate rsa modulus 2048 % You already have RSA keys defined named WAN2.cisco.com. % They will be replaced.
% The key modulus size is 2048 bits % Generating 2048 bit RSA keys, keys will be non-exportable... [OK] (elapsed time was 2 seconds)
WAN2(config)#end WAN2# ***device end***
Title: Re: Current frustration...
Post by: icecream-guy on March 10, 2017, 06:06:35 AM
think were getting off track here, the netmiko/telnet thing should probably be broken out into a different thread, as it's turning into more of a discussion than a frustration.
Title: Re: Current frustration...
Post by: dlots on March 13, 2017, 12:15:40 PM
I was asked to write a program to take an export from one server, and make it so we could import it into another server... but both servers were ACS, so at least it was quite easy.
Title: Re: Current frustration...
Post by: deanwebb on March 14, 2017, 10:57:22 AM
Waiting for the TAC callback on a case I had to reopen...
... and if they tell me the same resolution, this will be me:
:phone:
Title: Re: Current frustration...
Post by: SimonV on March 15, 2017, 03:29:03 AM
Too much projects at once, and they all want priority
:frustration:
Title: Re: Current frustration...
Post by: deanwebb on March 15, 2017, 08:25:51 AM
LICENSING
:jackie-chan: :rage: :no:
Title: Re: Current frustration...
Post by: deanwebb on March 21, 2017, 12:26:07 PM
MARKETING SUPPORTS EXTERNAL DNS. ALL OF IT.
:jackie-chan: :rage: :no:
Title: Re: Current frustration...
Post by: deanwebb on March 21, 2017, 12:27:01 PM
Feels much better, actually, to see that in 72pt Impact with associated graphics, especially Tracy Morgan.
Title: Re: Current frustration...
Post by: Nerm on March 21, 2017, 02:08:07 PM
Apps development team tries to manage the setup of a VPN between one of our locations and a cloud hosted solution provider. Didn't even mention anything to anyone on the network team until "go live" day.
:phone: :developers:
Title: Re: Current frustration...
Post by: Otanx on March 21, 2017, 07:28:55 PM
I just commented on your comment in the other thread about marketing running DNS. Glad it isn't considered normal.
To stay on topic, SNMP! Not simple. I knew that, but man can we please standardize on one convention within the same OID from the same vendor? I just want the interface name, there is an OID for interface name. IOS returns GigabitEthernet0/0/0 which is exactly what I would expect. IOS-XE returns GigabitEthernet0/0/0 woohoo. ASA returns "Adaptive Security Appliance 'Outside'". /sigh. You want the actual interface name? That is a Cisco OID and returns GigabitEthernet1.
The pains of automating stuff that was never built or designed to be automated.
-Otanx
Title: Re: Current frustration...
Post by: icecream-guy on March 22, 2017, 06:02:52 AM
Quote from: Nerm on March 21, 2017, 02:08:07 PM Apps development team tries to manage the setup of a VPN between one of our locations and a cloud hosted solution provider. Didn't even mention anything to anyone on the network team until "go live" day.
Dont' worry nothing will get to the cloud without network involvement to route traffic there. They will have to make a change request, and run it through CCB, then....
Title: Re: Current frustration...
Post by: wintermute000 on March 22, 2017, 06:50:16 AM
To stay on topic, SNMP! Not simple. I knew that, but man can we please standardize on one convention within the same OID from the same vendor? I just want the interface name, there is an OID for interface name. IOS returns GigabitEthernet0/0/0 which is exactly what I would expect. IOS-XE returns GigabitEthernet0/0/0 woohoo. ASA returns "Adaptive Security Appliance 'Outside'". /sigh. You want the actual interface name? That is a Cisco OID and returns GigabitEthernet1.
The pains of automating stuff that was never built or designed to be automated.
-Otanx
remember to enable SNMP ifIndex persistence otherwise your rage will increase exponentially upon next reload.
SNMP is retarded, I guess it wasn't designed for an automated era but god damn a bit more foresight and standards would make it a heck of a lot better RESTAPI and a browseable tree or GTFO
Title: Re: Current frustration...
Post by: deanwebb on March 22, 2017, 07:35:56 AM
To stay on topic, SNMP! Not simple. I knew that, but man can we please standardize on one convention within the same OID from the same vendor? I just want the interface name, there is an OID for interface name. IOS returns GigabitEthernet0/0/0 which is exactly what I would expect. IOS-XE returns GigabitEthernet0/0/0 woohoo. ASA returns "Adaptive Security Appliance 'Outside'". /sigh. You want the actual interface name? That is a Cisco OID and returns GigabitEthernet1.
The pains of automating stuff that was never built or designed to be automated.
-Otanx
The guys in charge of ASA reply, "But... that *is* the interface name!"
:problem?:
Title: Re: Current frustration...
Post by: Otanx on March 22, 2017, 10:44:45 AM
SNMP is retarded, I guess it wasn't designed for an automated era but god damn a bit more foresight and standards would make it a heck of a lot better RESTAPI and a browseable tree or GTFO
ifIndex persist is set in our build templates. That is another WTF. Who thought it was a good idea to renumber interfaces on reboot as a default?
-Otanx
Title: Re: Current frustration...
Post by: icecream-guy on March 22, 2017, 10:51:09 AM
SNMP is retarded, I guess it wasn't designed for an automated era but god damn a bit more foresight and standards would make it a heck of a lot better RESTAPI and a browseable tree or GTFO
ifIndex persist is set in our build templates. That is another WTF. Who thought it was a good idea to renumber interfaces on reboot as a default?
-Otanx
well if you are inserting an Ethernet module into slot 4 on that 6500, with 7, 8, & 9 full, if might be just a good idea to have your interfaces in sequential order. yea, I know hot swap able but point is made.
Title: Re: Current frustration...
Post by: Otanx on March 22, 2017, 11:07:01 AM
Quote from: ristau5741 on March 22, 2017, 10:51:09 AM well if you are inserting an Ethernet module into slot 4 on that 6500, with 7, 8, & 9 full, if might be just a good idea to have your interfaces in sequential order. yea, I know hot swap able but point is made.
Sure, but if you insert a module then you can run a command ifIndex renumber if you need them renumbered. The default should be not to change something. Somewhere else on the forums AspiringNetworker (I think) said if you have to always configure something then why isn't it a default? Even when SNMP was new everyone using SNMP was using ifindex persist command. However, SNMP was yesterdays rant.
My new frustration - In a load balancing configuration what would you think src-ip-hash option would do? Would you change your mind if I told you there was also an option called src-ip-only-hash? Just like taking a multiple choice test. Read all the answers and pick the most correct one.
-Otanx
Title: Re: Current frustration...
Post by: Ctrl Z on March 22, 2017, 12:19:33 PM
Quote from: Otanx on March 22, 2017, 11:07:01 AM My new frustration - In a load balancing configuration what would you think src-ip-hash option would do? Would you change your mind if I told you there was also an option called src-ip-only-hash? Just like taking a multiple choice test. Read all the answers and pick the most correct one.
I can see how that might cause a lot of people to use the wrong one.
The problem with the multiple choice test is if you don't know all the available options, someone might think they were making up fake options to throw them off.
Title: Re: Current frustration...
Post by: deanwebb on March 22, 2017, 12:25:46 PM
Quote from: Otanx on March 22, 2017, 11:07:01 AM My new frustration - In a load balancing configuration what would you think src-ip-hash option would do? Would you change your mind if I told you there was also an option called src-ip-only-hash? Just like taking a multiple choice test. Read all the answers and pick the most correct one.
TIL the difference between those two things: https://files.a10networks.com/vadc/forums/topic/destination-ip-hash-vs-destination-ip-only-ihash/
First one includes the port number in the hash. Second does not. First one should be named src-ip_port-hash
Title: Re: Current frustration...
Post by: Otanx on March 22, 2017, 02:06:12 PM
Mr Webb has it. Sorry, was going to put the answer at the bottom, and got distracted with work. Problem was the backend replicates sessions between the hosts, but not everything in the session. So if you are doing basic stuff if works. All the back ends know your session ID, and connecting to different hosts was OK. However, if you schedule a job the status of the job is not replicated between backends it is only stored on the server running the job. Once the job finishes the results are replicated. So depending on luck you may be able to see the status of the job. Looks like this fixed a few other weird things that were happening as well.
-Otanx
Title: Re: Current frustration...
Post by: Nerm on March 24, 2017, 09:03:56 AM
Employer makes a new acquisition and you find out that at one of the acquired sites the network is just one big /21. Wired/wireless/servers/everything all on the same broadcast domain with no segmentation of any kind. Oh and the /21 is full so the onsite "tech's" were in the process of adding a new network for wireless (192.168.1.0/24).
:facepalm3:
Title: Re: Current frustration...
Post by: deanwebb on March 24, 2017, 10:22:01 AM
Quote from: Nerm on March 24, 2017, 09:03:56 AM Employer makes a new acquisition and you find out that at one of the acquired sites the network is just one big /21. Wired/wireless/servers/everything all on the same broadcast domain with no segmentation of any kind. Oh and the /21 is full so the onsite "tech's" were in the process of adding a new network for wireless (192.168.1.0/24).
:facepalm3:
:ivan: Not just it's all a /21, but that it is FULL. Makes the 192.168.1.0/24 for wireless seem tame by comparison.
Meanwhile, on my side of things, I had to reopen a case about logfiles filling up on my NAC servers. Seems like there's additional things we need to learn about deleting. The ones tech support told us to delete are fine, we just found more things that are filling up the partitions.
This is supposed to be fixed in the latest SP, but we can't deploy it until we finish testing in the validation environment. Of course, we could test in the val environment if we weren't busy trying to keep the servers going by deleting logfiles that keep filling up the partition...
Title: Re: Current frustration...
Post by: icecream-guy on March 25, 2017, 07:33:36 AM
Working in a data center that is scheduled to close by the end of 2017. No improvements, no big network changes, no desire by anyone, lots of "Don't give a S" attitude, since the data center will be closing. no action, skills deteriorating, desire waning.
Title: Re: Current frustration...
Post by: deanwebb on March 25, 2017, 09:32:54 AM
Quote from: ristau5741 on March 25, 2017, 07:33:36 AM Working in a data center that is scheduled to close by the end of 2017. No improvements, no big network changes, no desire by anyone, lots of "Don't give a S" attitude, since the data center will be closing. no action, skills deteriorating, desire waning.
I had a friend once that was promoted to CIO when his firm went into bankruptcy. He had a bigger salary, true, but his job was to count all the computers and gear in the firm and keep an eye on it until it sold at auction. So, he brought a lot of reading material with him each day. There were zero other people in the entire office.
He was so glad to finally get out of there and back into a firm that was alive.
Title: Re: Current frustration...
Post by: icecream-guy on March 26, 2017, 07:08:32 AM
Quote from: ristau5741 on March 25, 2017, 07:33:36 AM Working in a data center that is scheduled to close by the end of 2017. No improvements, no big network changes, no desire by anyone, lots of "Don't give a S" attitude, since the data center will be closing. no action, skills deteriorating, desire waning.
I had a friend once that was promoted to CIO when his firm went into bankruptcy. He had a bigger salary, true, but his job was to count all the computers and gear in the firm and keep an eye on it until it sold at auction. So, he brought a lot of reading material with him each day. There were zero other people in the entire office.
He was so glad to finally get out of there and back into a firm that was alive.
I've been working on learning python, so far, not frustrating at all. Pretty easy in fact, well so far. using free Think Python book I found online.
Title: Re: Current frustration...
Post by: Otanx on March 28, 2017, 10:08:49 AM
More SNMP rants. Configuring SNMPv3 users. I need to define both the auth and priv passwords. This is all done by script so longer is better. Look at documentation and IOS allows 128 characters for both. Perfect. Write up the script, and run it. Script SSHs to the box, and runs the command. However, the user is not working. After much troubleshooting I find out that IOS has a command line length limit of 255 characters. It does not throw an error it just only accepts 255 characters of what you typed. So in my case the priv pass gets cut short.
-Otanx
Title: Re: Current frustration...
Post by: Netwörkheäd on March 28, 2017, 12:01:26 PM
Oh man, that sucks. When something is broken, we want an errmsg!
Over here, my Internet is out due to construction, so it's me and 3G for a while.
Sent from my SM-N900P using Tapatalk
Title: Re: Current frustration...
Post by: dlots on March 28, 2017, 12:27:07 PM
If you are interested I used codecadmey (although it uses 2.7 instead of 3.#) to get started.
If you ever want code to look at I'll be happy to give you some of mine (just having a program that SSHed to a device would have saved me weeks)
Title: Re: Current frustration...
Post by: deanwebb on April 06, 2017, 04:17:57 PM
Visio.
:rage:
Title: Re: Current frustration...
Post by: deanwebb on April 06, 2017, 04:24:07 PM
Also, doing an access audit on a system that does not map nicely to the levels of access defined in the audit.
:no:
Also, that system doesn't have any groups. Only 4 pre-defined roles that have to be assigned to individual users, all 197 of them (and counting).
:printer:
Title: Re: Current frustration...
Post by: icecream-guy on April 11, 2017, 07:47:21 AM
I don't know if it's frustrating or not, nor in which way.
Customer want us to SWAG a hardware proposal to build out a network at co-location, they estimate 10 racks. that's it. no bandwidth requirements, no services, oh, wait some video services, no amount of servers, no connectivity requirements, no power allowances, no idea of type or speed of hand off.
Title: Re: Current frustration...
Post by: deanwebb on April 11, 2017, 08:18:39 AM
Customer want us to SWAG a hardware proposal to build out a network at co-location, they estimate 10 racks. that's it. no bandwidth requirements, no services, oh, wait some video services, no amount of servers, no connectivity requirements, no power allowances, no idea of type or speed of hand off.
Provide a quote for 10 empty racks. Oh wait, 10 racks with a camera.
Title: Re: Current frustration...
Post by: icecream-guy on April 14, 2017, 11:57:26 AM
got my work computer refresh, win 10 laptop :rage: no admin rights.
Title: Re: Current frustration...
Post by: deanwebb on April 14, 2017, 12:20:18 PM
ME: Sounds like I'll need to get the firewalls sending all their logging to Splunk.
BOSS: Good luck. Nobody knows anything about our Splunk server except the guy that set it up.
ME: Well, I actually read up on it a while ago, so I can get started on it.
BOSS: AWESOME! YOU ARE NOW THE SPLUNK GUY FOR THE AMERICAS REGION!
ME: :rage:
Title: Re: Current frustration...
Post by: icecream-guy on April 16, 2017, 06:40:42 AM
BOSS: Good luck. Nobody knows anything about our Splunk server except the guy that set it up.
ME: Well, I actually read up on it a while ago, so I can get started on it.
BOSS: AWESOME! YOU ARE NOW THE SPLUNK GUY FOR THE AMERICAS REGION!
ME: :rage:
took us like 8 months to fill a splunk position.
Title: Re: Current frustration...
Post by: deanwebb on April 17, 2017, 08:13:32 AM
I actually look forward to using Splunk, but not with my current workload. Then again, isn't loading up massively how one gets promoted/bonused/otherwise rewarded?
Since they cut all the conference attendance out of the budget, I'm thinking *this* is how they want me to get introduced to new technologies... :steamtroll:
Title: Re: Current frustration...
Post by: mlan on April 18, 2017, 04:49:53 PM
Sounds like you will also need to increase your Splunk licensing. ;)
After talking with the Splunk guys, i'll also have to help actually build out the servers and collectors.
don't skimp on the hardware, splunk is a very cpu/memory intensive application. junk hardware will end up in slow searches and underwhelming performance take the specs from the website and then bump those up a few notches.
Title: Re: Current frustration...
Post by: deanwebb on April 19, 2017, 02:36:19 PM
Noted on that recommendation. It pretty much applies for any vendor's VM recommendations, as well.
Right now, the current frustration du jour is getting yelled at for dropping a task I was working 2 months ago, even though the same guy yelled at me to drop that task to get to work on other stuff.
Gotta have a thick skin in this biz, I tell ya...
Title: Re: Current frustration...
Post by: icecream-guy on April 20, 2017, 08:34:13 AM
Right now, the current frustration du jour is getting yelled at for dropping a task I was working 2 months ago, even though the same guy yelled at me to drop that task to get to work on other stuff.
I got an email system for that, using categories in outlook, categorize things I need to work on via email responses, either hot (red), medium (yellow) or low(green) placed in a sub-solder, and rotate through on a first come first serve basis, so when all my hot stuff gets to a point where I can no longer work on stuff, e.g. waiting on others for something, I start on the medium stuff, then move to the low, I refill the queues first thing, and then again at lunch, after lunch I'll start working on the hot stuff again. don't hit the low queue too often, but then that's the point, it's low priority anyway.
Title: Re: Current frustration...
Post by: wintermute000 on April 20, 2017, 04:27:05 PM
I miss Google desktop. When that was around, I didn't file anything, and literally googled my own pc (emails/office docs/text).
Title: Re: Current frustration...
Post by: deanwebb on May 01, 2017, 01:14:32 PM
Today at 8AM when I logged on to the VPN:
WINDOWS UPDATE WARNING! WE GONNA REBOOT YOU LIKE A MILLION TIMES! YOU BEST BE READY, FOOL!
So I click on the "update now" button and resign myself to a morning full of reboots.
Five hours later, it's still on "Downloading (0% Complete)"
:ivan:
Title: Re: Current frustration...
Post by: icecream-guy on May 02, 2017, 06:09:50 AM
WINDOWS UPDATE WARNING! WE GONNA REBOOT YOU LIKE A MILLION TIMES! YOU BEST BE READY, FOOL!
So I click on the "update now" button and resign myself to a morning full of reboots.
Five hours later, it's still on "Downloading (0% Complete)"
:ivan:
I've seen that a few times, sits at 0% for hours on end, then eventually installs, I've stopped them after a few hours only to have the downloads restart. so it probably is downloading, just not reflecting the actual percentage number
Title: Re: Current frustration...
Post by: deanwebb on May 02, 2017, 09:19:19 AM
It actually started running at 5PM.
And then failed at 5:21 PM.
Title: Re: Current frustration...
Post by: SimonV on May 02, 2017, 10:36:08 AM
Windows Update itself sucks. In enterprise you can't get around it but on other PCs you're much better off using a third-party updater...
Title: Re: Current frustration...
Post by: deanwebb on May 04, 2017, 03:20:21 PM
:wall: :wall: :wall: :wall: :wall:
Just got two BRAND NEW devices set up in the datacenter. Consoled in to them to put an IP address on them. They can't ping out, I can't ping in to them...
Datacenter guy says it's likely a bad interface.
On one, I could believe. But on two different devices? Brand new? How about maybe the cabling guys got the wrong port connected.
Further developments as we receive them...
:facepalm1:
Title: Re: Current frustration...
Post by: deanwebb on May 04, 2017, 06:20:54 PM
... this just in!
They had the cables plugged into the MGMT interface, not the E1 interface as I specified.
:facepalm4:
Title: Re: Current frustration...
Post by: Ctrl Z on May 04, 2017, 07:35:33 PM
Quote from: ristau5741 on April 20, 2017, 08:34:13 AM got an email system for that, using categories in outlook, categorize things I need to work on via email responses, either hot (red), medium (yellow) or low(green) placed in a sub-solder, and rotate through on a first come first serve basis, so when all my hot stuff gets to a point where I can no longer work on stuff, e.g. waiting on others for something, I start on the medium stuff, then move to the low, I refill the queues first thing, and then again at lunch, after lunch I'll start working on the hot stuff again. don't hit the low queue too often, but then that's the point, it's low priority anyway.
Beautiful. I'm going to start doing this!
Title: Re: Current frustration...
Post by: Nerm on May 17, 2017, 10:50:02 AM
CHINA!!!
Title: Re: Current frustration...
Post by: deanwebb on May 17, 2017, 11:16:43 AM
Not the country so much as the infrastructure we have there.
Title: Re: Current frustration...
Post by: weasleman on May 18, 2017, 12:43:34 PM
huawei!!! need i say more ??
Title: Re: Current frustration...
Post by: Otanx on May 18, 2017, 01:51:58 PM
Frustrated with myself, linux, and one of our customers all at once.
The host command on linux does not check the host file. Host is a DNS command not a name resolution command. Found this out after deploying a script I wrote to 4 customers. The 5th customer's DNS is not reliable, and is missing entries. We gave up fighting to get it fixed a long time ago, and just use local host files for our stuff. Script was failing, and after much troubleshooting I figured out it was the host command in my script that was causing the issue. Talked to the customer, and got a bunch of push back on adding five A records. Rewrote my script to call getent instead, and redeployed it to the other four customers as well.
So frustrated at linux for a command called host that does not look at a host file. Frustrated with the customer for failed DNS. Frustrated with myself for not using getent to begin with (I didn't know the getent command existed).
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on May 22, 2017, 04:01:13 PM
Frustrated with how, if we had made all the changes to the switches manually, we'd be done by now and they'd all be working fine with NAC.
But nooooooooooooooooooooooooo we had to use Priiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiime...
:butno:
( ^ New GIF because we needed this one... )
Months later, we're still dealing with "Well, that switch isn't in Prime..." "We didn't change that template..." "There were some errors in the last push..." and so on and so on and so on...
Title: Re: Current frustration...
Post by: wintermute000 on May 30, 2017, 06:04:12 AM
security 'architects' who still think its 2007 and you want firewalls everywhere. Esp as the default gateway on a DC fabric, obviously that's where you would put ALL TEH FIREWALLs. And Esp if you are also deploying micro-seg, why more firewalls never got in the way of anyone, just keep buying more firewalls.
bonus points if they can't route or switch their way out of a paper bag, but consider themselves fit to solution your DC fabric design for you.
That a multi-billion org can be held hostage by pinheads who would get absolutely demolished in any real engineering workshop out of their sheltered little enterprise enclave where they call all the shots and vendors are too scared to call them out on their BS.... mind boggling
Title: Re: Current frustration...
Post by: SimonV on May 30, 2017, 06:38:06 AM
Hey wintermute, having some firewall issues? :mrgreen:
on topic: Just had a 45 minute internet outage on our central breakout. Manual failover on the PA, only the fail part was successful :whistling:
Title: Re: Current frustration...
Post by: wintermute000 on May 30, 2017, 07:26:37 AM
not so much firewall issues but rather security pricks (who can't even subnet) trying to dictate DC fabric architecture by planting their flag in the ground everywhere. Its pure politics and its disgusting
Title: Re: Current frustration...
Post by: deanwebb on May 30, 2017, 09:23:29 AM
Quote from: wintermute000 on May 30, 2017, 07:26:37 AM not so much firewall issues but rather security pricks (who can't even subnet) trying to dictate DC fabric architecture by planting their flag in the ground everywhere. Its pure politics and its disgusting
Just to be clear, this is a case of:
:oracle:
And not a case of:
:notthefirewall:
I agree, if there is a DC firewall, it's there with an IPS to secure north-south traffic. East-west security is handled with products that ride in the VM space, stuff like that. Stuff that secures the chassis from VMs or containers trying to break out or break in. Segmentation is for sensitive devices. Access control should NOT be handled via the firewall, I can guarantee you that!
Title: Re: Current frustration...
Post by: deanwebb on May 30, 2017, 03:14:12 PM
ME: I have a non-technical problem.
BOSS: You should call HR. Here is the number.
HR: I can't help with that problem. Let me give you the phone number to call for someone who could help with that issue. gives me the same phone number my boss gave me
:disappoint:
Title: Re: Current frustration...
Post by: icecream-guy on June 01, 2017, 12:39:46 PM
service is down, alarms are ringing, name resolves in DNS can't ping, not sure where the service lives, not sure who owns the service, not sure who to contact about the service, they guy who did know this just left the team, left not very detailed instructions on the service, just sent out a bunch of emails to team leads, with no responses. oh, I can ping it, but alarms are still going off. seems to be limping along, somehow somewhere. UGH!
Title: Re: Current frustration...
Post by: deanwebb on June 01, 2017, 01:12:35 PM
Quote from: ristau5741 on June 01, 2017, 12:39:46 PM service is down, alarms are ringing, name resolves in DNS can't ping, not sure where the service lives, not sure who owns the service, not sure who to contact about the service, they guy who did know this just left the team, left not very detailed instructions on the service, just sent out a bunch of emails to team leads, with no responses. oh, I can ping it, but alarms are still going off. seems to be limping along, somehow somewhere. UGH!
:itcrowd:
:problem?:
Title: Re: Current frustration...
Post by: icecream-guy on June 02, 2017, 07:48:45 AM
Quote from: ristau5741 on June 01, 2017, 12:39:46 PM service is down, alarms are ringing, name resolves in DNS can't ping, not sure where the service lives, not sure who owns the service, not sure who to contact about the service, they guy who did know this just left the team, left not very detailed instructions on the service, just sent out a bunch of emails to team leads, with no responses. oh, I can ping it, but alarms are still going off. seems to be limping along, somehow somewhere. UGH!
:itcrowd:
:problem?:
don't know where it lives in the DC.
I could have used the IP to find the MAC address, manually traced the MAC down to a switch port, gotten the cable number off the cable, logged into the cable database, where 50% of the cables are not documented. I didn't bother.
Title: Re: Current frustration...
Post by: deanwebb on June 02, 2017, 02:50:54 PM
Just reboot everything then.
Even the Internet.
Title: Re: Current frustration...
Post by: deanwebb on June 19, 2017, 08:06:34 AM
THREE YEARS AGO...
Me: We want A and B. Project manager: Well, we can only get B for now. A will come later.
TWO YEARS AGO:
Me: Boy, can't wait to got A. Project manager: B is good for now, but, yes, getting A is going to be sweet.
ONE YEAR AGO:
Me: Are we ready to roll out A? Project manager: There are some budget issues, probably not this year, but next year we can likely roll out A.
TODAY...
Me: Looks like we're ready to roll out A. Project manager: A? I thought we were cool with just having B. Now I have to rewrite the project charter...
:disappoint:
Title: Re: Current frustration...
Post by: Nerm on June 28, 2017, 02:22:20 PM
:notthefirewall:
Title: Re: Current frustration...
Post by: deanwebb on June 28, 2017, 11:43:44 PM
Well, it's *not* the firewall, of that you can be sure.
Title: Re: Current frustration...
Post by: SimonV on June 30, 2017, 02:53:27 AM
Proxy server down, which team gets the tickets?
:notthefirewall:
Title: Re: Current frustration...
Post by: deanwebb on June 30, 2017, 09:06:11 AM
Heard about some Petya-related action at one of the big companies it hit... very very ugly situation there. My sympathy goes out to anyone hit by it, as it apparently wasn't just a WannaCry latecomer, but had some clever twists in it.
Title: Re: Current frustration...
Post by: wintermute000 on June 30, 2017, 05:31:24 PM
When I was in large MSP, at least 2 days a week the entire security tower would be running around cleaning up after some crypto attack on a big account. That was a few years ago as well it's likely worse now, not to mention it was a Cisco wonderland and you all know how good ASAs are at sand box/IPS etc LOL
Title: Re: Current frustration...
Post by: deanwebb on June 30, 2017, 06:26:17 PM
Quote from: wintermute000 on June 30, 2017, 05:31:24 PM When I was in large MSP, at least 2 days a week the entire security tower would be running around cleaning up after some crypto attack on a big account. That was a few years ago as well it's likely worse now, not to mention it was a Cisco wonderland and you all know how good ASAs are at sand box/IPS etc LOL
Checking on the Cisco ASA after a crypto attack:
:shock:
Title: Re: Current frustration...
Post by: deanwebb on July 05, 2017, 05:39:29 PM
OK, one site in Latin America... Skype works just fine, but all web traffic does not. Outlook and Oracle don't work. AD works just fine. RADIUS, SNMP, and doing a telnet to port 80 or 443 on internal IP addresses work fine, but telnet to 80/443 for external addresses do not work. Proxy can be pinged and we can telnet to the proxy port, but we can't get the proxy script to run on these boxes... other sites just fine...
:zomgwtfbbq:
Update: HTTPS to WLCs works just fine, all around the globe.
:phone:
You heard right. HTTPS to anything but a WLC fails.
Title: Re: Current frustration...
Post by: deanwebb on July 05, 2017, 09:57:31 PM
Made progress: the crazy stuff may be linked to internal sites having external web certificates and the proxy not working properly for that site.
:notthefirewall:
Feelin' good it's not the firewall. Also not the NAC.
Title: Re: Current frustration...
Post by: deanwebb on July 05, 2017, 11:06:18 PM
Confirmed it was the proxy by switching the users to a legacy proxy that's about to be decommissioned. That proxy then was able to proxy the traffic to the proxy in production. Now I need to hand this off to the proxy team.
:yeahright:
Title: Re: Current frustration...
Post by: deanwebb on July 06, 2017, 11:23:03 AM
Turns out, it was NOT a proxy issue... now I have to start a thread about what fixed it so I can learn more stuff better.
Title: Re: Current frustration...
Post by: Otanx on July 07, 2017, 10:32:59 AM
You can only have one priority one. No you can't say priority 1a and 1b. That is priority 1 and 2. No I can't work on both at the same time. They each take 4 people, and I have 6. Grow a backbone and tell one customer they have to wait. It isn't my fault both customers waited till the last minute to ask for help, but apparently it is my problem. Also every time you bug me about this puts me further behind because you wanted a 30 minute meeting about 1, and then another 30 minute meeting about 2, and you want to schedule these with a 30 minute opening between them in case the first one goes over. This means I am getting nothing done for an hour and a half.
Also if these two are so important stop interrupting my guys to do something else that "just takes 5 minutes". By the time they save what they were doing, login to do your "5 minute" task, then go back to the original task, remember where they were, and get back to that priority 1 it has been 15 - 30 minutes.
The best part was getting about 50% done with priority 1 (which had 100% to be done by COB Friday or heads would roll), and asking the customer for some information we need that wasn't in the request. The answer came back that they will get that info to us on Monday. We made it clear we need the info to finish the tasks. Yep, we just are going to push the deadline to end of next week.
I feel better. Yesterday kind of sucked. Now that one is on hold the other "priority 1" should be done on time. It will be close, but as long as the data center doesn't catch on fire we should be good. (hmmm, why is my temp sensor reading 300 degrees?)
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on July 07, 2017, 10:58:11 AM
"If you keep talking, it's gonna take longer..."
Title: Re: Current frustration...
Post by: dlots on July 10, 2017, 03:21:03 PM
Me working with a client who "Passed" the CCIE written on a packet cap. He is obsessed with TCP "Zero Window" packets. If you are not familiar with them that's when one device tells the other that it is busy and can't take any more data.
We started the capture, pinged the device, then ran the test.
Client: look at 192.168.2.205 (You can see laptop trying to connect to something that wouldn't reply) Me: the pings are at packet number 11519, this is long before we started the test. [....] Client: so here is my concern any hiccup between client and server will give the program an error. We need to look at how we can stop instances where the communication is lost in order to stop the timeout error. Me: I don't see where the communication is lost though, can you point me to something?? Client: look at packet 199 Me: that happened WAY before the test actually started, it's the wrong server, and it happened after the TCP session was closed
Notably the issue was also a Zero Window which isn't a network issue.
Title: Re: Current frustration...
Post by: deanwebb on July 10, 2017, 04:26:39 PM
So, he's CCIE Written, But Expire?
Title: Re: Current frustration...
Post by: wintermute000 on July 10, 2017, 04:48:47 PM
I love it when people tell me they passed the written. I just straight up ask them if they've passed the lab and then when they're planning on attempting it
Title: Re: Current frustration...
Post by: deanwebb on July 10, 2017, 05:40:21 PM
We had a guy from India apply to us who had CCIE-Written. We asked and he said it took him a year and a half to prepare for it and then he went to the lab and failed it. He then said he didn't want to attempt the lab again and got into other vendors instead.
He had no hesitation as we asked him questions about said other vendors' stuff and he really knew the products well on a technical level. Guy got thumbs up from us.
Title: Re: Current frustration...
Post by: icecream-guy on August 09, 2017, 06:09:04 AM
Troubleshooting a IPv6 routing issue for 4 hours and then finding out it's a...a..... Cisco bug..... :twitch:
Title: Re: Current frustration...
Post by: deanwebb on August 09, 2017, 08:53:03 AM
OUCH
Man, that's gotta hurt.
Title: Re: Current frustration...
Post by: icecream-guy on August 09, 2017, 12:01:59 PM
default route ::0/0 wasn't propagating, next hop wasn't available on the active, but it was on the standby. Fail over fixed it temporarily.
Title: Re: Current frustration...
Post by: Nerm on September 25, 2017, 11:13:20 PM
So I am in China helping the locals migrate off of a flat /21 to a properly designed network. While here I am taken to visit another location of ours and find that the core switch is an unmanaged netgear. lol.....this trip is very eye opening and sometimes sad all at once.
Title: Re: Current frustration...
Post by: deanwebb on September 26, 2017, 10:20:38 AM
Quote from: Nerm on September 25, 2017, 11:13:20 PM So I am in China helping the locals migrate off of a flat /21 to a properly designed network. While here I am taken to visit another location of ours and find that the core switch is an unmanaged netgear. lol.....this trip is very eye opening and sometimes sad all at once.
:jackie-chan: WHY DIDN'T THEY USE A BELKIN?
Title: Re: Current frustration...
Post by: icecream-guy on September 26, 2017, 11:07:16 AM
FPR4100-EVAL1-A# scope fabric-interconnect a FPR4100-EVAL1-A /fabric-interconnect* # show version Fabric Interconnect A: Running-Kern-Vers: 5.0(3)N2(4.21.64) Running-Sys-Vers: 5.0(3)N2(4.21.64) Package-Vers: Startup-Kern-Vers: Startup-Sys-Vers: Act-Kern-Status: Ready Act-Sys-Status: Ready Bootloader-Vers:
FPR4100-EVAL1-A /fabric-interconnect* # activate firmware system 5.0(3)N2(4.21.64) kernel-version 5.0(3)N2(4.21.64) Warning: When committed this command will reset the end-point
FPR4100-EVAL1-A /fabric-interconnect* # commit-buffer Error: Update failed: [Unable to find switch kernel image for Vendor Cisco Systems, Inc., Model FPR-4140-SUP and Version 5.0(3)N2(4.21.64)]
FPR4100-EVAL1-A /fabric-interconnect* # show version Fabric Interconnect A: Running-Kern-Vers: 5.0(3)N2(4.21.64) Running-Sys-Vers: 5.0(3)N2(4.21.64) Package-Vers: Startup-Kern-Vers: Startup-Sys-Vers: Act-Kern-Status: Ready Act-Sys-Status: Ready Bootloader-Vers:
-- Any ideas? (other than don't reboot)
Title: Re: Current frustration...
Post by: deanwebb on September 26, 2017, 01:23:17 PM
Check the integrity of the image, possibly restart process if there are issues with image integrity? https://www.cisco.com/c/en/us/td/docs/security/firepower/fxos/fxos221/cli-guide/b_CLI_ConfigGuide_FXOS_221/image_management.html#topic_903C3296477343699E65B97FB6EB5FE8
Title: Re: Current frustration...
Post by: icecream-guy on September 27, 2017, 05:47:21 AM
Quote from: deanwebb on September 26, 2017, 01:23:17 PM Check the integrity of the image, possibly restart process if there are issues with image integrity? https://www.cisco.com/c/en/us/td/docs/security/firepower/fxos/fxos221/cli-guide/b_CLI_ConfigGuide_FXOS_221/image_management.html#topic_903C3296477343699E65B97FB6EB5FE8
yeah, I found that link yesterday, and was going through it , unfortunately I don' t have access to download the image.....somebody forgot to add SmartNet to the PO for our new 4110's*. so I've got to go around through the back door and get entitlement.
* oh, that's another good frustration
Title: Re: Current frustration...
Post by: deanwebb on September 27, 2017, 10:15:13 AM
Title: Re: Current frustration...
Post by: icecream-guy on September 29, 2017, 01:15:09 PM
developers that don't know about hosts.allow on the unix severs they need to ssh to; and they swear up and down that the firewall is blocking access.
Title: Re: Current frustration...
Post by: deanwebb on September 29, 2017, 02:38:27 PM
All day long, I've been trying to get ESXi on my lab server without using a DVD. Finally got it on by juggling boot order for the HDDs and getting it on a bootable flash drive.
Dell iDRAC could have saved me, but for incompatible Java versions.
Java... Java makes me say...
:rage:
Title: Re: Current frustration...
Post by: deanwebb on October 26, 2017, 05:28:16 PM
Customer is frustrated with internal processes, then I get frustrated, too.
We share the sad.
:boohoo:
Then the PM gets engaged and finds out we're behind schedule and we face, together:
:disappoint:
Title: Re: Current frustration...
Post by: SimonV on October 27, 2017, 04:26:40 AM
Microsoft Azure Check Point firewalls These two combined.
Title: Re: Current frustration...
Post by: deanwebb on October 27, 2017, 06:00:53 AM
This one is most likely the one that matches that condition.
Title: Re: Current frustration...
Post by: icecream-guy on October 27, 2017, 06:09:04 AM
my work laptop crashed earlier in the week, 1/2 recovered with some old backups but lost alot of data. The desktop tech was able to access the hard drive and scrape off some important files I needed. I've been living off a loaner for a few days living in a hazy cloud waiting for my laptop to be reimaged, then I've got to spend half a day reinstalling all the custom configurations I had been running.
See Today I learned thread for more....
Title: Re: Current frustration...
Post by: deanwebb on October 27, 2017, 08:35:55 AM
I hate laptop crashes. This is why I copy my data to several locations, I can't rely on just one backup.
Title: Re: Current frustration...
Post by: SimonV on October 27, 2017, 09:20:07 AM
I'm switching between three or four PCs all the time, so I've moved most of my stuff to OneDrive Business, not keeping anything important on the computers locally anymore.
I kept off for a long time (for security) but it does make life a lot easier in the end. it also keeps KeePass and Superputty in sync across machines.
Title: Re: Current frustration...
Post by: deanwebb on December 13, 2017, 01:10:22 PM
Project started Monday... today is Wednesday... client asked for ALL project documentation by CoB today.
:haha1:
HLD, LLD, AH, Runbook... all by CoB today.
:haha2:
OK, how about just copies of stuff you've given to other customers?
:haha4:
Dude, we have to totally clean out all the NDA stuff, that's gonna take a while and a few reviews to make sure we got it all.
:challenge-denied:
How about I talk with my people and get back with you all on when we think we can have that stuff ready... :smug:
Title: Re: Current frustration...
Post by: icecream-guy on December 13, 2017, 01:16:53 PM
cover page, with project name title page with "this page intentionally left blank" written on it, closing page, with credits. (preferrably someone whom you don't want to keep their job)
done it for ya.!! :P
Title: Re: Current frustration...
Post by: SimonV on April 25, 2018, 09:56:58 AM
QuoteH.323 Keep Alive packets sent (sent every two minutes) from the localhost to the remote host are being intercepted by the Security Gateway and are not being forwarded to the remote host.
Instead of forwarding these H.323 Keep Alive packets, the Security Gateway is incorrectly responding to the local host, as if it were the remote host
On the flip side, I did learn a lot about our nGenious One, neat product!
Title: Re: Current frustration...
Post by: deanwebb on April 25, 2018, 08:01:27 PM
So... the security gateway is making the remote host seem like it's still up?
Smoooooooooooooooooooooooooooooth.
:yeahright:
Title: Re: Current frustration...
Post by: SimonV on April 26, 2018, 01:30:01 AM
Client sends keepalives. Firewall responds to clients' keepalives but somehow forgets to forward them to the remote VCS gateway and that one closes the session after one hour.
Worst thing is that all these silly Check Point bugs always require some sort of update or hotfix, which is not a simple task in 24/7 datacenters.
Title: Re: Current frustration...
Post by: icecream-guy on April 26, 2018, 07:21:58 AM
Quote from: SimonV on April 26, 2018, 01:30:01 AM Client sends keepalives. Firewall responds to clients' keepalives but somehow forgets to forward them to the remote VCS gateway and that one closes the session after one hour.
Worst thing is that all these silly Check Point bugs always require some sort of update or hotfix, which is not a simple task in 24/7 datacenters.
Had something like that at the airline where I worked, setup a VOIP environment, and every morning during the morning meeting between the two sites, the concall would drop, call lasted an hour, and after some variable time the call would drop after 30 minutes, never before. but not always.
After much troubleshooting we determined that the remote site put the call on mute and if no one from remote office spoke, the VOIP system would think the line was dead after 30 minutes and the system would close the connection. The variability came into play where if someone spoke after 10 minutes, then put the phone back on mute, 30 minutes after that call would drop, and if they spoke a few times, without a 30 minute lapse being on mute, that call would not drop. <insert head scratch emoticon here>
Title: Re: Current frustration...
Post by: Otanx on January 31, 2019, 10:15:52 AM
We just installed a line for a new fax machine. Not my job anymore, but still just drives me crazy when I walk past it. Oh, and the requirement is an internal one. This isn't a regulations say, or customer requires. Nope, a group here decided that a fax machine was a solution, and implemented it "just in case email is broken" except that email is already the backup to the web form they use to receive requests. Not my problem, but it makes me irrationally angry.
-Otanx
Title: Re: Current frustration...
Post by: icecream-guy on January 31, 2019, 11:39:58 AM
Quote from: Otanx on January 31, 2019, 10:15:52 AM We just installed a line for a new fax machine. Not my job anymore, but still just drives me crazy when I walk past it. Oh, and the requirement is an internal one. This isn't a regulations say, or customer requires. Nope, a group here decided that a fax machine was a solution, and implemented it "just in case email is broken" except that email is already the backup to the web form they use to receive requests. Not my problem, but it makes me irrationally angry.
-Otanx
Give 'em a few months of fax spam, and they'll be ripping it out
Title: Re: Current frustration...
Post by: deanwebb on January 31, 2019, 12:55:59 PM
PC LOAD LETTER!?!?!?! WHAT THE [expletive deleted] DOES PC LOAD LETTER MEAN???
:printer:
Title: Re: Current frustration...
Post by: SimonV on March 21, 2019, 04:33:03 PM
Working on a merger for some factories in France. Someone decided it was a good idea to assign prefixes based on the area code, conflict gallore.
:facepalm1:
Title: Re: Current frustration...
Post by: deanwebb on March 22, 2019, 07:11:40 AM
I've got a customer that has about 30K endpoints, all on a flat 10.0.0.0/8 network. Because that's how they have their Meraki set up.
:facepalm4:
Title: Re: Current frustration...
Post by: Otanx on March 22, 2019, 09:08:25 AM
That means there are two of these out there. We have a customer that does this, and they don't have Meraki so it can't be the same one. Just to add to the insanity they expanded their network to remote sites. Those remote sites overlap IP space of course. So they do a weird double nat thing with proxy-arp and "reserve" the remote site space in their /8 then the firewall NATs it to itself somehow to make routing work.
It is ugly and I am glad I only am responsible for the external POP there.
-Otanx
Title: Re: Current frustration...
Post by: Nerm on March 22, 2019, 10:28:57 PM
:facepalm1:
Title: Re: Current frustration...
Post by: deanwebb on March 23, 2019, 09:30:36 AM
That means there are two of these out there. We have a customer that does this, and they don't have Meraki so it can't be the same one. Just to add to the insanity they expanded their network to remote sites. Those remote sites overlap IP space of course. So they do a weird double nat thing with proxy-arp and "reserve" the remote site space in their /8 then the firewall NATs it to itself somehow to make routing work.
It is ugly and I am glad I only am responsible for the external POP there.
-Otanx
Wait, which address is real and which one is NAT? Someone gave me an IP to check out, but I have no idea where to start!
:morty:
Title: Re: Current frustration...
Post by: Otanx on May 08, 2019, 05:47:08 PM
Frustration solved post. Had to look at this thread to see if I had ranted about this already, but didn't find anything. About 6 months ago there were a ton of changes here. We had one network guy leave for a better job, I got promoted to a new slot that was created on the architect team, and the company did a reorganization that split the network team into two separate groups. The guys picked for the other team moved to another office, and are no longer part of the on-call roster. Basically we went from a team of 6 to a network team of 3, and one of those 3 is my old slot that isn't filled yet.
We just found out today that they added another slot to the team. So we now have two openings. Once we can fill them both we might actually be able to do more than keep the network from crashing, and won't have anyone rage quiting over the frequency of on-call.
Now to find someone that doesn't lie on the resume.
-Otanx
Title: Re: Current frustration...
Post by: Otanx on October 03, 2019, 11:20:15 AM
Trying to do temperature monitoring. Dig into SNMP and find Cisco exposes several temperature sensors, and it is easy to get the temps. Also exposed is the "shutdown" temp for each sensor that if hit the switch will shutdown to protect itself. What is not exposed? A warning value. If you login to the switch, and do a show environment temp there is a warning value shown. They just don't expose it in SNMP. All the other vendors I poked at have current, warning, critical exposed for each sensor. Did a walk on the entire tree on Cisco looking for the temperature shown in the CLI. So now I have to play guessing games. Is 70% of critical a good value? 80%? Who knows.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on October 03, 2019, 03:22:09 PM
Wow, no warning value... :naughty: Bad Cisco! Sounds like you need to submit a feature request.
Title: Re: Current frustration...
Post by: Otanx on October 03, 2019, 04:45:56 PM
I am considering it. I know I will just get told about this great thing called DNA Center, or whatever the new cool Cisco monitoring tools are called. Last time I worked on it was called CiscoWorks.
Side note. We found a room with a failing AC. Inlet temps on a pair of switches were showing 40C. Critical to Cisco is 57C so we have a little time. Also you get a small scare when you finish with Cisco and start working on Arista, and see a temp of 347C. Then realize it is measured in 10ths of C, and is really 34.7.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on October 09, 2019, 05:24:52 AM
347 C, I think, is the melting point of some soft steels... :)
Title: Re: Current frustration...
Post by: config t on October 21, 2019, 08:18:36 AM
We had a contract turnover and lost our network engineer last week because the incoming company severely under-bid. They offered him a 30% pay cut, no PTO, no HOLA/COLA and a 7-day work week.
While I don't blame him for turning down the opportunity of a lifetime, as the only other network guy here my phone now rings a lot more for an extra $0/hour.
The replacement was promoted from within and is in hiding down south at the TCF. This individual doesn't even have a CCNA and "forgot" to come and do a turnover. Coooolio...
Title: Re: Current frustration...
Post by: Nerm on October 21, 2019, 09:23:16 PM
I'm sorry but I am having trouble understanding the 30% pay cut. I mean who the hell would think the guy would even consider it.
Title: Re: Current frustration...
Post by: config t on October 21, 2019, 11:59:18 PM
Nobody who has an actual skill set would take a 30% pay cut. But I think that was the point.
I found out this morning that the outgoing engineer's tier 3 billet is now a "Tier 1 Network Engineer Support" billet. What exactly Network Engineer Support is, I have not a clue.
I think it means the individual can continue hiding down south at the TCF on night shift and run cable or something.
Title: Re: Current frustration...
Post by: deanwebb on October 22, 2019, 08:56:28 AM
Title: Re: Current frustration...
Post by: Dieselboy on November 21, 2019, 03:03:17 AM
Some thoughts I had: Two ways to perform changes during business hours and without change control:
1. To upgrade the code of a network device but don't initiate a reboot. Wait for power cut or crash or some other issue to initiate a reboot. Device loads new code next time :mrgreen:
2. Apply config change but don't commit to "copy run start" until a later date. That way, if anyone complains you can say "that damned device, I'll reboot it". Proceed to reboot your device which will fix the issue and management will pat you on the back :mrgreen:
Title: Re: Current frustration...
Post by: config t on November 21, 2019, 06:46:46 AM
Respek. I too like to pull the occasional sneaky :whistling: >:D
Title: Re: Current frustration...
Post by: deanwebb on November 23, 2019, 07:53:15 AM
Sometimes, one does have to have an aggressive interpretation of "standard change" in order to get things done...
Title: Re: Current frustration...
Post by: icecream-guy on November 23, 2019, 03:12:58 PM
how about people that don't plan their projects, spent 4 hours today to troubleshoot firewall issue only to find out the that the network wasn't trunked to the F5, no wonder there were connectivity issues.... :blank:
and it was break fix, and they couldn't roll back, something about some MS servers going EOL EOY :squint:
Title: Re: Current frustration...
Post by: icecream-guy on November 23, 2019, 03:13:52 PM
Quote from: ristau5741 on November 23, 2019, 03:12:58 PM how about people that don't plan their projects, spent 4 hours today to troubleshoot firewall issue only to find out the that the network wasn't trunked to the F5, no wonder there were connectivity issues.... :blank:
and it was break fix, and they couldn't roll back, something about some MS servers going EOL EOY :squint:
oh that was a good frustration.....
Title: Re: Current frustration...
Post by: Otanx on November 25, 2019, 10:47:08 AM
1. To upgrade the code of a network device but don't initiate a reboot. Wait for power cut or crash or some other issue to initiate a reboot. Device loads new code next time :mrgreen:
2. Apply config change but don't commit to "copy run start" until a later date. That way, if anyone complains you can say "that damned device, I'll reboot it". Proceed to reboot your device which will fix the issue and management will pat you on the back :mrgreen:
At one place I worked we had ASIs or "Authorized Service Interruptions" aka maintenance window. To get one approved you had to get approval from about 5 people including the manager of the group that would be impacted. One of those groups refused to approve any ASI. They were a 9 - 5 office, but would refuse any after hours work because they couldn't be down in the morning when they came in if something went wrong. Asking for someone to stay late so we could test was also a no go. We had ASI or Alex Service Interruptions(Bosses name was Alex). We would reload the box, and record it as an unknown power hit to their building. Our boss knew what was up, and was OK looking the other way as well as his boss. The group that was impacted never complained, or even commented on how often they had power issues in their building. They also never made the connection that those "power issues" always happened on nights that we had an ASI for every other building.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on November 25, 2019, 11:05:43 AM
1. To upgrade the code of a network device but don't initiate a reboot. Wait for power cut or crash or some other issue to initiate a reboot. Device loads new code next time :mrgreen:
2. Apply config change but don't commit to "copy run start" until a later date. That way, if anyone complains you can say "that damned device, I'll reboot it". Proceed to reboot your device which will fix the issue and management will pat you on the back :mrgreen:
At one place I worked we had ASIs or "Authorized Service Interruptions" aka maintenance window. To get one approved you had to get approval from about 5 people including the manager of the group that would be impacted. One of those groups refused to approve any ASI. They were a 9 - 5 office, but would refuse any after hours work because they couldn't be down in the morning when they came in if something went wrong. Asking for someone to stay late so we could test was also a no go. We had ASI or Alex Service Interruptions(Bosses name was Alex). We would reload the box, and record it as an unknown power hit to their building. Our boss knew what was up, and was OK looking the other way as well as his boss. The group that was impacted never complained, or even commented on how often they had power issues in their building. They also never made the connection that those "power issues" always happened on nights that we had an ASI for every other building.
-Otanx
Immediate vulnerability patches also have a nice "code red" feel about them, you can get lots of stuff upgraded and rebooted during a security emergency.
Title: Re: Current frustration...
Post by: Otanx on December 19, 2019, 04:07:13 PM
Anti-frustration post...
One of our developers was told his application was not working right. The application is basically a relay. Data comes in the front end, it gets some processing done, and the data is forwarded to another system. The report was that data of a specific type was not being delivered on the back side. Typical developer he "knew" it wasn't his application, but unlike a typical developer he wanted to prove it wasn't. He asked me if there was a way to prove if the missing type of data was being delivered to his system. I mentioned tcpdump, and he went away. That was yesterday afternoon. I just got a email that he used tcpdump, and confirmed the data was getting to his box, and is now looking at why his application is dropping the traffic.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on December 20, 2019, 01:18:58 PM
One of our developers was told his application was not working right. The application is basically a relay. Data comes in the front end, it gets some processing done, and the data is forwarded to another system. The report was that data of a specific type was not being delivered on the back side. Typical developer he "knew" it wasn't his application, but unlike a typical developer he wanted to prove it wasn't. He asked me if there was a way to prove if the missing type of data was being delivered to his system. I mentioned tcpdump, and he went away. That was yesterday afternoon. I just got a email that he used tcpdump, and confirmed the data was getting to his box, and is now looking at why his application is dropping the traffic.
-Otanx
Wow.
:applause:
For you AND that developer!
Title: Re: Current frustration...
Post by: Nerm on January 16, 2020, 09:31:38 AM
Meraki....
:notthefirewall:
...unless the firewall is Meraki
Title: Re: Current frustration...
Post by: Otanx on January 16, 2020, 10:28:58 AM
I will have no bosses come Feb 1. We are in contract transition. We got told yesterday that all three levels of leadership above me will not be transitioning to the new contract. There is a lot of other stuff going on as well, but nothing I would share on an open forum. The next two weeks are going to be fun.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on January 16, 2020, 01:26:48 PM
Title: Re: Current frustration...
Post by: deanwebb on January 16, 2020, 01:27:38 PM
Quote from: Otanx on January 16, 2020, 10:28:58 AM I will have no bosses come Feb 1. We are in contract transition. We got told yesterday that all three levels of leadership above me will not be transitioning to the new contract. There is a lot of other stuff going on as well, but nothing I would share on an open forum. The next two weeks are going to be fun.
-Otanx
:explosion2:
Title: Re: Current frustration...
Post by: config t on January 17, 2020, 06:17:24 AM
Sounds about right during a transition. At least you get the right to first refusal.
Good luck brudder. Hopefully the new company is respectable. I would be interested to know which company you were working with and who the new one is. It's a small world.
Title: Re: Current frustration...
Post by: config t on January 17, 2020, 06:19:23 AM
We have Fortinet at our boundaries and I have to say I like it. Easy to admin, troubleshoot and understand.
Title: Re: Current frustration...
Post by: Otanx on January 27, 2020, 09:22:40 AM
Quote from: Otanx on January 16, 2020, 10:28:58 AM I will have no bosses come Feb 1. We are in contract transition. We got told yesterday that all three levels of leadership above me will not be transitioning to the new contract. There is a lot of other stuff going on as well, but nothing I would share on an open forum. The next two weeks are going to be fun.
-Otanx
Everything is canceled. I don't know what happened, but Friday at 6PM we got told the government canceled the new contract, and extended our current contract for one more year. It is crazy now. People had offers from the new company +20K/yr, people blew PTO instead of taking the payout so now they have none. One person turned down a job offer at another company because the new company offered her one of the leadership slots that was opening. Now that isn't happening. It is a wild ride. My boss said he will be staying on for a while, but still plans to move to his other opportunity. Just now he does not have to do it on the first.
Personally I am fine with it, but it is going to be wild. Government contacting at its finest.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on January 27, 2020, 11:01:20 AM
When you know you have uncertainty, the wrenching emotion is balanced by an ability to scramble and get a more sure future lined up.
Title: Re: Current frustration...
Post by: config t on January 30, 2020, 01:39:37 AM
"Do you know anything about printers?"
Title: Re: Current frustration...
Post by: config t on January 30, 2020, 03:48:43 AM
Everything is canceled. I don't know what happened, but Friday at 6PM we got told the government canceled the new contract, and extended our current contract for one more year. It is crazy now. People had offers from the new company +20K/yr, people blew PTO instead of taking the payout so now they have none. One person turned down a job offer at another company because the new company offered her one of the leadership slots that was opening. Now that isn't happening. It is a wild ride. My boss said he will be staying on for a while, but still plans to move to his other opportunity. Just now he does not have to do it on the first.
Personally I am fine with it, but it is going to be wild. Government contacting at its finest.
-Otanx
Maybe the win was contested.
I quit my job last year to take a gig at Ft. Huachuca and they lost the contract one week to the day after I arrived. They moved me to another team and lost that contract four months later. Then they offered me a job in Virgina. Now I'm in Bahrain and thankfully on a stable contract. Transition periods are definitely eventful.
Title: Re: Current frustration...
Post by: icecream-guy on January 30, 2020, 06:04:11 AM
I quit my job last year to take a gig at Ft. Huachuca and they lost the contract one week to the day after I arrived. They moved me to another team and lost that contract four months later. Then they offered me a job in Virgina. Now I'm in Bahrain and thankfully on a stable contract. Transition periods are definitely eventful.
you telling me, last contract turnover the new win thought they could do the job of 5 with 2.. they only put out 2 offers, now in our second year, we are up to 7.
My PM said that the group that bids contracts are not bean counters, their job is to win the contract, and it's up to finance to figure out how to make a profit.
Title: Re: Current frustration...
Post by: deanwebb on January 30, 2020, 08:14:13 AM
Yeah... sales always seems to think it can get away with underselling services. Then the customer has a higher risk of not renewing because the company failed to deliver on what it promised.
Title: Re: Current frustration...
Post by: Otanx on January 30, 2020, 11:41:00 AM
Last few days have been fun. Leadership has been out from DC. They are trying to take care of some of the folks who were impacted. I have heard three different rumors on why the contract was revoked last minute. All sound plausible. Now back to work for another year. Then we get to do this all again.
-Otanx
Title: Re: Current frustration...
Post by: config t on January 31, 2020, 12:31:36 AM
Quote from: Otanx on January 30, 2020, 11:41:00 AM Last few days have been fun. Leadership has been out from DC. They are trying to take care of some of the folks who were impacted. I have heard three different rumors on why the contract was revoked last minute. All sound plausible. Now back to work for another year. Then we get to do this all again.
you telling me, last contract turnover the new win thought they could do the job of 5 with 2.. they only put out 2 offers, now in our second year, we are up to 7.
My PM said that the group that bids contracts are not bean counters, their job is to win the contract, and it's up to finance to figure out how to make a profit.
That's precisely why I'm having to keep my customer's network on life support while I tend to my other duties. Sometimes I am busier than a two-pecker billygoat but I will be working with these guys for the next 2-3 years so I don't mind building up the goodwill now.
In this case they still had the slot but they bid the contract so low that they couldn't afford to fulfill the requirement. They BS'd about hiring a network guy several times over the last two months but I think they actually did this time. They alternate between calling it a Network Engineer, Network Architect, and Network Engineer Support.
I don't care what they call it as long as he knows how to config t and doesn't just sit around. There is a ton of work to do.
Title: Re: Current frustration...
Post by: icecream-guy on January 31, 2020, 05:48:13 AM
LOL, that better than, "feeling like an octopus and getting pulled in 8 different directions at once".
They got me goin' nine ways to Sunday!
Title: Re: Current frustration...
Post by: config t on February 05, 2020, 11:46:52 PM
Another contractor who knows without a doubt that his devices aren't causing the problem and won't even log in to look at log files or configurations. This time it's a VTC bridge.
Why do people behave this way? Is it to conceal incompetence? Laziness? Arrogance? I need answers :XD:
Title: Re: Current frustration...
Post by: icecream-guy on February 06, 2020, 05:55:19 AM
Quote from: config t on February 05, 2020, 11:46:52 PM Another contractor who knows without a doubt that his devices aren't causing the problem and won't even log in to look at log files or configurations. This time it's a VTC bridge.
Why do people behave this way? Is it to conceal incompetence? Laziness? Arrogance? I need answers :XD:
that ain't right, we _always_ have to look into everything, to prove it is not us, otherwise the finger gets pointed directly at us. so rule everything else out, and then go back to the contractor and point that finger.
Title: Re: Current frustration...
Post by: deanwebb on February 06, 2020, 10:12:57 AM
Could very well be incompetence. The less he looks at stuff, the less chance he has of being shown up as a paper tiger.
Title: Re: Current frustration...
Post by: Otanx on May 06, 2020, 05:07:30 PM
Minor frustration, but if you use the management interface on Cisco IOS you can't use the default "tacacs+" group for authentication. The management port is in a VRF, and so you have to define a new aaa group and then tell that group to use that VRF. You can not change the default "tacacs+" group to use a VRF. 99.9% of my gear is in-band, and it just works. For our lab we are linking it to auth and couldn't get it to work. We use the management port so we can access the lab from prod, but still keep it separate. Now either my lab has a one off authentication configuration, or I need to update production to use aaa groups.
-Otanx
Title: Re: Current frustration...
Post by: icecream-guy on May 07, 2020, 08:13:08 AM
If I remember correctly from my MPLS studies, the preference would be to have the management in the global routing table and all the customer data in VRF
Title: Re: Current frustration...
Post by: Otanx on May 07, 2020, 08:55:08 AM
Quote from: ristau5741 on May 07, 2020, 08:13:08 AM If I remember correctly from my MPLS studies, the preference would be to have the management in the global routing table and all the customer data in VRF
That is how I would normally do it. However, Cisco doesn't let you take the management interface out of the "Mgmt-vrf". So if you use the physical management port on the device you are stuck using that VRF for management along with all the other things that go with it. I also found out about "vrf-also" on the VTY ACL command.
-Otanx
Title: Re: Current frustration...
Post by: config t on May 22, 2020, 12:42:49 AM
Most of my frustration about having stuff piled on me when the engineer left is gone. It happened again this week when the operations chief left. At this rate I'm going to be the last man standing soon. That's pretty much how deployed environments work though. In a year or two when everyone has rotated out and it's all new folks I will be the old hand.
We finally got a new guy about 2 months ago. FNG isn't green but he's still timid so I like to throw operations stuff at him and give him opportunities to take the lead on things. I don't let him flounder though, if I see him struggling I jump in and help out.
I think it came down to recognizing the opportunity. Instead of being pigeon-holed into one role I get to do everything. WAN, LAN, FW, tactical networks and project work. By the time I move on I will be a much more well-rounded network professional.
I still get paid the same though.
Title: Re: Current frustration...
Post by: deanwebb on May 22, 2020, 10:26:43 AM
Meanwhile, I'm dealing with a customer canceling meetings next week, so I'm dealing with major bench time. Sigh.
Title: Re: Current frustration...
Post by: Otanx on May 28, 2020, 01:02:26 PM
If a subnet is setup for DHCP, and you are told it is using DHCP, and are given directions on how to use DHCP, and given directions on creating reservations in DHCP even if your system can't do DHCP, and you just set a static IP on a system without putting it in DHCP because DHCP is too much work. I will stomp on your system when I try to deploy a new system. I will not be sorry. I will send out an IT wide email explaining why setting DHCP reservations for static IPs when that IP is inside a DHCP scope is important, and use your system as an example.
That is all. I feel better now.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on May 28, 2020, 02:21:42 PM
Quote from: Otanx on May 28, 2020, 01:02:26 PM If a subnet is setup for DHCP, and you are told it is using DHCP, and are given directions on how to use DHCP, and given directions on creating reservations in DHCP even if your system can't do DHCP, and you just set a static IP on a system without putting it in DHCP because DHCP is too much work. I will stomp on your system when I try to deploy a new system. I will not be sorry. I will send out an IT wide email explaining why setting DHCP reservations for static IPs when that IP is inside a DHCP scope is important, and use your system as an example.
That is all. I feel better now.
-Otanx
But I use DHCP or whatever you're talking about! The instructions said I'd get an IP address from the router!
Title: Re: Current frustration...
Post by: Otanx on May 28, 2020, 04:25:37 PM
If you use DHCP then I am OK. I won't even notice you didn't create a reservation. My problem are the guys that are setting static IPs on their servers that fall in the DHCP pool range. DHCP will try to identify if the IP is in use, but it can't always do that. All of a sudden I have duplicate IP problems when I am deploying a new box. I found out after writing my rant that one guy uses DHCP to get the initial address then sets whatever he gets as a static because "the DHCP server might fail"
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on May 29, 2020, 08:39:49 AM
Quote from: Otanx on May 28, 2020, 04:25:37 PM If you use DHCP then I am OK. I won't even notice you didn't create a reservation. My problem are the guys that are setting static IPs on their servers that fall in the DHCP pool range. DHCP will try to identify if the IP is in use, but it can't always do that. All of a sudden I have duplicate IP problems when I am deploying a new box. I found out after writing my rant that one guy uses DHCP to get the initial address then sets whatever he gets as a static because "the DHCP server might fail"
-Otanx
Exactly. We have to give ourselves an IP address in case the DHCP server fails.
I also run iptables and BGP on every one of my servers, just in case the router fails. :problem?:
Title: Re: Current frustration...
Post by: Otanx on May 29, 2020, 09:01:00 AM
You joke, but we are looking at doing this. There isn't a good way to dual home systems at layer2. Most of the solutions are active/passive, or require custom vendor magic to do multi-chasis LAGs. Route to the host, create and advertise a loopback, and bind all your services to it. I can now get redundancy, load balance, etc. We might do OSPF instead of BGP, but I feel BGP is better for this. However, we would need licensing, and our guys are more familiar with OSPF.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on May 29, 2020, 09:52:22 AM
I can say that a number of vendors focus on their core product without looking at basic operational considerations like NIC teaming.
Title: Re: Current frustration...
Post by: Otanx on May 29, 2020, 12:07:57 PM
Oh, lets kick off another rant. If you are building enterprise hardware appliances dual power supplies are a must. Especially on the lower end models where customers may not be able to afford buying two appliances.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on May 29, 2020, 12:16:30 PM
Heh. That's why they're "lower end models." All kinds of limitations on those bad boys.
Title: Re: Current frustration...
Post by: config t on June 02, 2020, 02:16:31 AM
I miss the old dual power supply Brocade ICX.. sounded like a jet engine.
Title: Re: Current frustration...
Post by: wintermute000 on June 02, 2020, 07:02:36 AM
1st gen Nexus 5K FTW I nearly had a heart attack the first time I turned one on.
---
Why on earth are all official MS learn Azure tutorials either GUI, powershell or Az CLI? BUT BUT BUT in real devops land, you deploy via code via pipeline always, GUI/CLI drivers get a smack on the head (or are customers' former wintel engineers rebranded as cloud engineers) Why not educate people the IaC way up front? I get it, play around with clicky clicky/typy typy first but if you're training people officially why not get them trained on the 'correct' way up front as well? Instead you can pass expert level certs being a total ARM / terraform scrub which is just incomprehensible IMOThe crying shame is that in every other aspect MS Learn is amazing, and the tutorials are amazing BUT they're teaching you the 'wrong' way to deploy - at least teach the automated way AFTER the clicky walkthrough
Title: Re: Current frustration...
Post by: deanwebb on June 03, 2020, 09:58:31 AM
Today, I get to hold a hand and write a step-by-step runbook on how to upgrade.
:caine:
Title: Re: Current frustration...
Post by: Otanx on June 03, 2020, 10:36:55 AM
Why on earth are all official MS learn Azure tutorials either GUI, powershell or Az CLI? BUT BUT BUT in real devops land, you deploy via code via pipeline always, GUI/CLI drivers get a smack on the head (or are customers' former wintel engineers rebranded as cloud engineers) Why not educate people the IaC way up front? I get it, play around with clicky clicky/typy typy first but if you're training people officially why not get them trained on the 'correct' way up front as well? Instead you can pass expert level certs being a total ARM / terraform scrub which is just incomprehensible IMOThe crying shame is that in every other aspect MS Learn is amazing, and the tutorials are amazing BUT they're teaching you the 'wrong' way to deploy - at least teach the automated way AFTER the clicky walkthrough
The problem with that is what automated ways do you want them to teach? Raw python using requests to make the API calls? Ansible with the official modules? Something else? There are a ton of them. Maybe some simple stand alone courses that cover the different popular ones so students can take the one that applies to them.
Non-technical rant today. Traffic sucked this morning. Nevada opened more on June first so the commuters are back. I was getting used to the empty roads. I have been looking at getting a Tesla for awhile, and had convinced myself I didn't need to spend that much money on a new car. After this morning I am ready to hit the button just for all the autopilot features.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on June 03, 2020, 02:16:02 PM
I know more than a few Tesla drivers, they're not ever going back.
Title: Re: Current frustration...
Post by: Otanx on June 03, 2020, 03:07:45 PM
Yep, my brother owns a Model 3 that I get to "not drive" on the weekends. I love it. He finally got the update that tracks stop lights, and stop signs. While I don't think real self driving is anywhere near ready this is an awesome start. I just have a hard time justifying a car payment higher than my mortgage. Even if I buy used to get what I want is over 64K. Plus we have a reservation on the Cyber Truck so maybe if I just wait... or maybe not.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on June 03, 2020, 04:39:49 PM
Just buy a DeLorean and a bunch of batteries. How hard could it be?
:haha4:
Title: Re: Current frustration...
Post by: wintermute000 on June 04, 2020, 02:57:21 AM
Why on earth are all official MS learn Azure tutorials either GUI, powershell or Az CLI? BUT BUT BUT in real devops land, you deploy via code via pipeline always, GUI/CLI drivers get a smack on the head (or are customers' former wintel engineers rebranded as cloud engineers) Why not educate people the IaC way up front? I get it, play around with clicky clicky/typy typy first but if you're training people officially why not get them trained on the 'correct' way up front as well? Instead you can pass expert level certs being a total ARM / terraform scrub which is just incomprehensible IMOThe crying shame is that in every other aspect MS Learn is amazing, and the tutorials are amazing BUT they're teaching you the 'wrong' way to deploy - at least teach the automated way AFTER the clicky walkthrough
The problem with that is what automated ways do you want them to teach? Raw python using requests to make the API calls? Ansible with the official modules? Something else? There are a ton of them. Maybe some simple stand alone courses that cover the different popular ones so students can take the one that applies to them.
Simple, ARM template deployments via Azure Devops with Github triggers. Or Terraform. Nobody serious uses Ansible for cloud, its got no state, its imperative, and soforth. And nobody is going to seriously suggest having to go naked python unless/until you have a very advanced use-case.For infra the 'mainstream' way is coalescing already:
Azure - ARM or terraform
AWS - Cloudformation or terraform
VMware - v<product> or terraform (not sure how mature it is but its starting get a ton of mentions and apparently under the hood a lot of vRa 7.x is rebadged terraform)
Network boxes - ansible
Linux boxes - ansible / chef / puppet / salt
Windows boxes - yech
TLDR learn ansible and one of terraform/CF or terraform/ARM depending on whether you want to go team orange or team win Unless you go down dlot's path you are never going to be more than scrub tier python hacker so unless you change your career focus to pure automation/programming just do python basics for now. Any 'real' dev will run all over you - that's been our experience here, its just not worth trying to convert a net-eng to a part time python dev (I stress the part time bit - ppl like dlots have gone full time, different story).Basically just learn enough so that you can actually write a class, then after that focus on tooling, that's all the python you'll effectively need for now. And oh learn bash/powershell.And azure CLI / AWS CLI.My head hurts
Title: Re: Current frustration...
Post by: Otanx on June 04, 2020, 04:00:37 PM
That shows my lack of experience with cloud. If Terraform has taken over for cloud automation then 100% teach GUI/CLI to get basics then move the training to Terraform. Teach how you would do it in production.
My goal with python is like you said. Be good enough I can get stuff working, and not embarrass myself too bad when others look at the code. Most of the stuff I have written are just middle ware stuff to get two different products to work together that don't already have something. So quick module to say get data from Netbox, and then a module to feed that data into Nessus Security Center, and kick off a scan. I have no plans to be good enough to consider myself a developer.
There is a guy here in town with an electric DeLorean and an electric Mustang. Both were self conversions. He shows them off at the local car shows.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on June 04, 2020, 06:05:55 PM
^ Noice. :smug:
Title: Re: Current frustration...
Post by: icecream-guy on June 05, 2020, 01:53:55 PM
Hey,
Q. Why did John Delorean's cars sell so well?
A. There was a 1/2 oz of coke in the glove.
hahahahaha......Old joke.....
Title: Re: Current frustration...
Post by: deanwebb on June 05, 2020, 02:51:58 PM
:haha2:
Title: Re: Current frustration...
Post by: config t on June 08, 2020, 11:21:31 PM
We have a network team disto list that I encourage end customer to use whenever they communicate instead of unicasting one of the network team directly. This allows everyone including HQ leadership to have SA on whatever is going on. It's especially useful since we work in different locations.
Last week they had an idea to create a separate distro list for the O&M team's internal shift change reporting. Great idea and I encourage that kind of behavior. Except it has the word Network and Admin in it, same as our "everyone" distro. So of course end customer gets confused and uses it, and now there is mass $&*%ing confusion any time there is a network issue.
I encouraged them to change the name but not include the word Network. So they added an extra acronym but kept everything else. I give up.
Title: Re: Current frustration...
Post by: deanwebb on June 09, 2020, 03:13:27 PM
Wait, so do I contact the Network Administration Staff list, or the Staff Network Administration list?
Title: Re: Current frustration...
Post by: Otanx on June 09, 2020, 03:42:01 PM
That depends. Do you need an administrator for the staff network, or do you need to talk to the network administration staff?
-Otanx
No, I need to address the network of staff administrators.
:curly:
Title: Re: Current frustration...
Post by: deanwebb on June 20, 2020, 09:18:19 AM
Guys that say everything is broken when two clients that aren't part of the domain are having problems accessing a server that requires domain membership in order to reach... but we only get the "everything is broken" part and have to deal with a major incident which, as a vendor, is always done in the shadow of an approaching bus.
:rage:
Title: Re: Current frustration...
Post by: deanwebb on June 29, 2020, 01:30:21 PM
8am Monday morning meeting getting moved to 9am...
... meeting change went out Sunday night...
... LATE Sunday night...
... and I have another meeting already set up with another customer at that time...
Customer wanted to do all the scheduling and seems to have no clue how I actually work with *other* customers...
:oracle:
Title: Re: Current frustration...
Post by: Otanx on June 30, 2020, 09:54:10 AM
Visibility into ASA IPSec sucks. I just want to know what my crypto engine utilization is at. We have started pushing more traffic over our tunnels, and I just want to know how much head room I have before the ASA is my bottle neck. Lets see... "show crypto accelerator statistics" hmmm, nope. A bunch of packet counts, but nothing I can link to utilization. Oh, "show crypto accelerator load-balance detail ipsec" Ah, percentages. This looks promising. Hmm, nope, never mind. This just shows the percentage of current traffic being processed by each engine. Nothing about maximum capacity. Lets google this... Oh, here is a command. oh, never mind. Firepower only.
I guess I will just use the IPSec throughput numbers on the data sheet as my max. Hope those are at least close to real life.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on June 30, 2020, 10:08:56 AM
Quote from: Otanx on June 30, 2020, 09:54:10 AM Visibility into ASA IPSec sucks. I just want to know what my crypto engine utilization is at. We have started pushing more traffic over our tunnels, and I just want to know how much head room I have before the ASA is my bottle neck. Lets see... "show crypto accelerator statistics" hmmm, nope. A bunch of packet counts, but nothing I can link to utilization. Oh, "show crypto accelerator load-balance detail ipsec" Ah, percentages. This looks promising. Hmm, nope, never mind. This just shows the percentage of current traffic being processed by each engine. Nothing about maximum capacity. Lets google this... Oh, here is a command. oh, never mind. Firepower only.
I guess I will just use the IPSec throughput numbers on the data sheet as my max. Hope those are at least close to real life.
-Otanx
You don't want a bottleneck because it leads to the dreaded calls of, "It's slow! Fix the slow!"
Title: Re: Current frustration...
Post by: Otanx on June 30, 2020, 03:54:27 PM
For this application I don't dread the "it's slow calls". The system that uses this part of the network timestamps everything when it is generated on the far side, and then again when it is processed in our DC. So compare the two timestamps. If there is a large delta then something is slow. Maybe not the network, but something. Also, and I know this is crazy, but we monitor those timestamps, and alert if they start getting larger deltas so we can fix it before it becomes a problem.
Now users on VPNs are another story. It's slow. My music had to buffer for 2 seconds yesterday. I got disconnected 4 times this morning. Those I hate.
-Otanx
Title: Re: Current frustration...
Post by: config t on July 01, 2020, 04:22:23 AM
I'm not familiar with ASA. Does it show up when you do the ol' "show proc cpu | include"
Quote from: deanwebb on June 20, 2020, 09:18:19 AM Guys that say everything is broken when two clients that aren't part of the domain are having problems accessing a server that requires domain membership in order to reach... but we only get the "everything is broken" part and have to deal with a major incident which, as a vendor, is always done in the shadow of an approaching bus.
:rage:
"The internet is down" - I hate it most when it comes from a lazy network person.
Title: Re: Current frustration...
Post by: Otanx on July 01, 2020, 08:51:57 AM
Nope, I have show cpu commands, but they don't include the crypto hardware modules. I will be reaching out to TAC to see if there is something maybe undocumented. I also realized last night that I have some smaller ASA5515s I may be able to setup a test, and overload the crypto to see what happens. They are only rated for 250Mb/s so I can see what happens as I get close to that. Maybe figure out some canaries I can use if I can't graph the utilization directly.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on July 01, 2020, 10:45:43 AM
Today's frustration is the PM in the kickoff meeting that's demanding to know where the design documentation is and then doesn't understand that we haven't had the necessary meetings yet to produce that documentation because, you know, this is the *kickoff* meeting...
Title: Re: Current frustration...
Post by: config t on July 01, 2020, 12:41:45 PM
Quote from: Otanx on July 01, 2020, 08:51:57 AM Nope, I have show cpu commands, but they don't include the crypto hardware modules. I will be reaching out to TAC to see if there is something maybe undocumented. I also realized last night that I have some smaller ASA5515s I may be able to setup a test, and overload the crypto to see what happens. They are only rated for 250Mb/s so I can see what happens as I get close to that. Maybe figure out some canaries I can use if I can't graph the utilization directly.
-Otanx
I'm curious to see how this shakes out. Even though DoD is moving to PA there will still be a $*#&-ton of ASA's around for ages.
Title: Re: Current frustration...
Post by: config t on July 01, 2020, 01:02:31 PM
I have hit the one year mark in my current role and had a meeting today to discuss the fact that I am billeted as a Net Admin II yet I am clearly performing network engineering functions. Integration, configuration baseline, network discovery and documentation, SME support, etc.
"Come back in budget time before the option year in March".
Not really mad. I love my job because I touch everything even if it's all old gear. I rocked out hard on key integrations. My customer barely had to bat an eye while I engineered new hardware specific SATCOM network packages, troubleshot, baselined and produced documentation and logical diagrams for the same. All while jamming out to 80's retro synthwave on my noise canceling headphones.
Should I even be complaining? Do job titles even matter? I think it comes down to the fact that if they could squeeze out another 20k I would stay until the last day of the last option year March of 2024. Despite all the high priority work we have coming down the pipline.
Title: Re: Current frustration...
Post by: Otanx on July 01, 2020, 01:54:10 PM
Oh, ASAs are probably going to be in prod longer than I will be working. If you need basic firewalling they are solid boxes. I predict you will see them running 10 years from now in environments that don't change very often. I will update here when I either solve it or give up.
To me the job title is a means to an end. I really don't care what you call me as long as it is close to what I do. Unfortunately the job title may impact other stuff like pay caps so that can be a concern. So in your case if they can give you the extra 20K but still call you a NE2 does it matter? I know one of my old companies they had pay bands, and a NE2 couldn't be paid more than X. If you needed more than that you had to be moved to a different title. Because of that I was officially a "Cyber Analyst 4" for a while because I wanted more than their max for the network titles. Stupid HR games, but they finally added some more numbers to the NE titles, and got me moved back over.
There is a case to be made about future jobs, and if they call you a windows admin, but you are really a network engineer that could be a concern. But Net Admin 2 vs a Net Admin 3? As someone who interviews I don't care about the number. I pay more attention to what you described as your duties, and how well you can talk to those.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on July 01, 2020, 02:15:06 PM
Job title matters, for sure. And you can't lie about it, because that's info a former employer can divulge in a request.
Title: Re: Current frustration...
Post by: config t on July 02, 2020, 10:59:23 AM
I think you're right Otanx. Like you said, anyone technical who potentially speaks with me in an interview would know what I'm about
It looks like the pay bump would have to be tied to job title with the way this contract works. Since I am already here doing the job, I doubt the customer will be willing to pay more for something they are already receiving. I did get an email yesterday that my HOLA/COLA went up substantially, so I guess that's a pseudo raise.
It's all good. For the first time in a while I am actually happy where I am at so I plan to take advantage of this overseas gravy train with biscuit wheels for at least another couple years.
I guess putting Rocket Surgeon on my resume is out :XD:
Title: Re: Current frustration...
Post by: Otanx on July 13, 2020, 11:34:29 AM
I have spent the last week dealing with logrotate in Linux. For those that don't know logrotate is used in Linux to prevent log files from just growing in size and filling the disk. You tell it what files to watch, and can rotate on size or time. You can compress the log files when you rotate, and tell it to keep the last X files. So pretty easy.
Except that the application I am working with creates a new directory for each day, and puts the days logs there. I just need to compress and then delete anything old. Log rotate can't do this natively. You can give it a wild card so I could tell it /var/log/*.log and it would rotate any files ending in .log. What I couldn't do was /var/log/application/*/*.log. I had to match every directory under the application and then the log files in those directories.
Long story short I created a file in /var/log/application/ called fake.log. I told logrotate to rotate that file every day. Then told it to run a script after it is finished rotating the file. The script handles compressing the previous days log, and then deleting anything older than a week. I could have just written the script and added it to cron, but I like having it in logrotate because that is where people in the future will look for log rotation stuff.
The last part of that is log rotate, and the script ran great when I did it manually. However, it wasn't working when log rotate ran from cron. I learned that the path environment variable in cron is different than a users path. So while I could run "find /var/log/application/*/*.log" from the command line when cron runs it can't find find. I have to use /usr/bin/find ... to get it to work.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on July 13, 2020, 06:16:25 PM
Ah yes, there's always one app that has to be *different*...
:oracle:
Title: Re: Current frustration...
Post by: config t on July 14, 2020, 04:20:50 AM
"They have IPs! DHCP is working! But they can't get to any websites!" yes DHCP is working, but where is the default gateway?
"DHCP is really working now! But I can't get to any websites! All of my routes are advertised in EIGRP!" yes, all of your routes are advertised in EIGRP, but why is your gateway of last resort not set?
Title: Re: Current frustration...
Post by: wintermute000 on July 14, 2020, 05:45:48 AM
We're in the middle of a new campus build and the geniuses that are in charge of setting up the AD have somehow disabled the DHCP scope for the AV VLAN / subnet 3 times in the last fortnight.
Naturally, the leases are set to 30 minutes because we're not in prod, so when an entire building's worth of AV (in build, not prod) goes offline, and they run around screaming "the whole network is down". The same people in charge of the DHCP scope they accidentally disabled. 3 times in a fortnight.
I hate wintel guys so much
Title: Re: Current frustration...
Post by: config t on July 14, 2020, 07:01:27 AM
That's classic.
Title: Re: Current frustration...
Post by: Otanx on July 14, 2020, 08:19:01 AM
This I can get behind. Even staying on topic with DHCP. We are moving a bunch of our stuff to DHCP, and part of that is creating reservations for the existing static gear before we change it to DHCP. I gave them a list of about 300 systems to add. I found out after three days they were doing this by hand in the GUI, and had done about 50. Pointed out that they could use powershell to do this much faster, and was told they don't know powershell. A quick Google later, and I gave them the powershell commands. I don't think they used them.
Our Linux DHCP servers? Oh, go update the dhcp git repo, submit a pull request. Once approved click the rocket ship in Ansible. If you just wait till tomorrow Ansible will update it overnight.
Lets not mention the Windows NPS servers. The ones with a scheduled task to reboot every night because they can't figure out why they stop responding to RADIUS after a few days. Nope not going to mention those.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on July 14, 2020, 11:02:27 AM
If you're a sysadmin, you MUST Powershell. You MUST.
IPAM solutions are because you don't want to trust your DHCP to the sysadmins that can't do Powershell...
:mssql:
Title: Re: Current frustration...
Post by: config t on July 15, 2020, 07:21:05 AM
According to our night shift sysadmin Option 150 and Gateway was unecessary fluff.
I'm also dealing with a guy who it turns out completely crumbles under pressure. During the last ASI I needed him to identify an interface connected to a VTC and set it as an access port. Instead of doing it while he was physically consoled into the switch he went into panic mode and ran around between the comm closet and the office for about 30 minutes trying to set up SSH access because he needed to be at his desk where it's warm and quiet. I fixed the vty line for him and left him there for the rest of the night so he could clicky clack on his keyboard and stay out of my hair (what's left of it :XD:)
Title: Re: Current frustration...
Post by: deanwebb on July 15, 2020, 01:13:50 PM
I'm dealing with a guy who can't count past 2.
But he's also in charge of scoping out how many appliances we need to set up for the solution, and we need more than 2, of more than 2 varieties of boxes, in more than 2 locations.
:rage:
Title: Re: Current frustration...
Post by: config t on July 16, 2020, 12:16:52 AM
But he's also in charge of scoping out how many appliances we need to set up for the solution, and we need more than 2, of more than 2 varieties of boxes, in more than 2 locations.
:rage:
:morty:
Title: Re: Current frustration...
Post by: deanwebb on July 16, 2020, 12:48:43 PM
Yep, exactly what's going on. It may soon look like this:
:shock2:
The concept of all professional services/consultants as Ricks and all customers as Mortys is an interesting thought to ponder... as is the concept of the Rickest Ricks, Mortiest Ricks, Rickest Mortys, and Mortiest Morties...
I consider myself to be a fairly strong Rickish Rick, and this customer is one of the Mortiest of Mortys out there...
Title: Re: Current frustration...
Post by: Otanx on July 16, 2020, 04:47:29 PM
Morty isn't a customer. He is the new guy that left enterprise to become a PS/consultant. Rick is his mentor. Morty only knows the one network from his old job he had for 15 years. Now he is being faced with all those other networks...
Specifically that photo is them at a customer site, and the Rick sees the look in the customers eyes. He shoves Morty in front of him to take the next question. A question about if their product supports IPX because they have a legacy Novell network that is mission critical be included. Rick avoids the pain, and Morty learns about the real world.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on July 16, 2020, 05:07:25 PM
Quote from: Otanx on July 16, 2020, 04:47:29 PM Morty isn't a customer. He is the new guy that left enterprise to become a PS/consultant. Rick is his mentor. Morty only knows the one network from his old job he had for 15 years. Now he is being faced with all those other networks...
Specifically that photo is them at a customer site, and the Rick sees the look in the customers eyes. He shoves Morty in front of him to take the next question. A question about if their product supports IPX because they have a legacy Novell network that is mission critical be included. Rick avoids the pain, and Morty learns about the real world.
-Otanx
This actually happen to/near you?
Title: Re: Current frustration...
Post by: Otanx on July 16, 2020, 07:33:48 PM
Oh, thank god no. I just have my old Novell CNE certification wallet card sitting on my desk so that is where I went. I have seen horrible stuff... but not that.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on July 16, 2020, 08:03:05 PM
Quote from: Otanx on July 16, 2020, 07:33:48 PM Oh, thank god no. I just have my old Novell CNE certification wallet card sitting on my desk so that is where I went. I have seen horrible stuff... but not that.
-Otanx
I almost went for my CNE... then NT 4.0 released...Went MCSE instead.
Title: Re: Current frustration...
Post by: Otanx on July 16, 2020, 10:10:02 PM
Quote from: Otanx on July 16, 2020, 07:33:48 PM Oh, thank god no. I just have my old Novell CNE certification wallet card sitting on my desk so that is where I went. I have seen horrible stuff... but not that.
-Otanx
I almost went for my CNE... then NT 4.0 released...Went MCSE instead.
The only reason I got my CNE instead of my MCSE was the local community college had the MCSE classes during the weekdays and CNA/CNE classes at night. I worked so CNE it was. did 3.12 then upgraded to 4/Intranetware, then 5. Never did 6.
Best part was I passed my CNA test, and talked to job placement at the school. Couldn't figure out why they kept referring me to medical/nursing jobs.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on July 17, 2020, 10:11:21 AM
:rofl:
Title: Re: Current frustration...
Post by: icecream-guy on July 17, 2020, 07:40:21 PM
man, back in the day if you could configure IP to run over IPX on a novell network, you were the cats pajamas.. really cool.
Title: Re: Current frustration...
Post by: deanwebb on July 17, 2020, 09:49:59 PM
... that's when home LANs were all NetBEUI because it was simple and there was no Internet.
ahh the days of running NetBEUI over thin net and using BNC connectors connecting your devices. and tracking down a bad 50 ohm terminator
Title: Re: Current frustration...
Post by: config t on July 19, 2020, 03:32:34 AM
Quote from: deanwebb on July 16, 2020, 12:48:43 PM The concept of all professional services/consultants as Ricks and all customers as Mortys is an interesting thought to ponder... as is the concept of the Rickest Ricks, Mortiest Ricks, Rickest Mortys, and Mortiest Morties...
I consider myself to be a fairly strong Rickish Rick, and this customer is one of the Mortiest of Mortys out there...
Aww geez. I've had brief a brief stint doing PS/consulting and I've roamed the world dabbling in various networks. Does that mean I'm a Rickish Morty? Or a Mortyish Rick? It's a little rickdiculous to think I haven't reached at least some level of Rickdom by now.
Title: Re: Current frustration...
Post by: deanwebb on July 19, 2020, 05:02:00 PM
I think I'd rather be a Rickish Morty than a Mortyish Rick...
Title: Re: Current frustration...
Post by: wintermute000 on July 20, 2020, 11:41:49 PM
Wasting my life on a DCNM product session and the entire first 2 hours is SAN shit. Just show me the VXLAN stuff kthxbye
Title: Re: Current frustration...
Post by: config t on August 06, 2020, 04:59:07 PM
"The new IOS's and switch IOS make Layer 2 automatic and there is nothing to worry about"
Sure but what about your root bridge, spanning-tree version, VTP and..
"No the new IOS's make it transparent.."
Ok well let me know WHEN you have problems..
"My networks won't connect.."
Title: Re: Current frustration...
Post by: deanwebb on August 10, 2020, 11:32:31 AM
Summer can be a slow time in Vendorland... so my *biggest* frustration right now happens when I get off work and have to deal with all the damn Voidwoken monsters in Divinity Original Sin 2. :smug:
Title: Re: Current frustration...
Post by: wintermute000 on August 10, 2020, 08:53:13 PM
I could never get into the Divinities (only tried 1) or Pillars of Eternity. Which is strange, as I've played through all the old school iso RPGs - Baldurs Gates/Neverwinter Nights/Icewind Dales, the Dragon Ages, Pathfinder Kingmaker, the old school Fallouts, Wasteland 2 etc.
Title: Re: Current frustration...
Post by: config t on August 11, 2020, 03:45:28 AM
Quote from: wintermute000 on August 10, 2020, 08:53:13 PM I could never get into the Divinities (only tried 1) or Pillars of Eternity. Which is strange, as I've played through all the old school iso RPGs - Baldurs Gates/Neverwinter Nights/Icewind Dales, the Dragon Ages, Pathfinder Kingmaker, the old school Fallouts, Wasteland 2 etc.
I got like halfway through Pillars. Love me some old school rpg's too. I've always been a huge Total War fan also and playing the crap out of Warhammer 2.
Title: Re: Current frustration...
Post by: wintermute000 on August 11, 2020, 07:47:52 PM
I have played TW to death (spent the longest actually in the Third Age mod for Medieval 2 - LOTR in total war engine, heaven), from Shogun 1 onwards. Rome, Medieval, M2, Empire, Rome 2.... Gave Warhammer a spin on the weekend, couldn't get into it, I know objectively its a great game but subjectively it simply felt like I'd played it before... man I'm getting old lol
Title: Re: Current frustration...
Post by: deanwebb on August 11, 2020, 08:22:20 PM
Just finished my first Divinity 2 run, had a great time. Now I've loaded up on the mods to have a trigger-happy OP run for the lulz. :problem?:
Title: Re: Current frustration...
Post by: icecream-guy on August 12, 2020, 04:07:08 PM
video game rant in the frustrations thread. can we split it out? into another thread? C:-)
That and UPS's that don't have enough power to drive the device power supply fans, went through 3 RMA's before I realized that if I power them directly into outlet, power supply is not bad. Unfortunately to due to cord length limitations both power supplies are plugged in same UPS circuit. better than power supply being "down" i guess, at least the SNMP monitors are off my back.
Title: Re: Current frustration...
Post by: icecream-guy on August 12, 2020, 09:34:59 PM
Cisco FX-OS Bugs. :o vASA Firewall fails to load with checksum error after FX-OS upgrade.
Title: Re: Current frustration...
Post by: deanwebb on August 14, 2020, 12:16:57 PM
New version of Juniper OS that the customer upgraded to before checking compatibility with $VENDOR...
Title: Re: Current frustration...
Post by: wintermute000 on August 15, 2020, 08:05:19 AM
vendor bugs to the left, vendor bugs to the right. In the middle of 2 projects with multiple implementation breaking bugs. Totally different technologies and vendors as well (both very large, very well known and not at all 'niche' or startupy).
One of them has necessitated multiple re-designs (and reduced functionality) to work around.
The other is waiting for hail mary vendor software upgrade, release on target next week (assurances from regional exec level), if that goes to hell then yep we're re-designing (and again forced to accept reduced functionality, egg on face and loss of money as we re-implement to design...)
My last 2-3 weeks have been a procession of failed changes, TAC calls, vendor escalation calls, emergency re-design followed by emergency political deathmatch where I scream 'alternative design' and they scream 'temp workaround' and are forced to explain to non-technical cast of dozens (thanks large Enterprise projects and the baggage train of moochers they always attract), followed by desperate field testing of workarounds leading to repeat change attempts. Then we go back and amend the design docs (repeatedly). All of which is being done staring at the same 4 walls of my home study thanks to COVID isolation (beats getting infected I suppose).
Title: Re: Current frustration...
Post by: deanwebb on August 15, 2020, 09:09:43 AM
Worst part about the baggage train is when you hit "reply all" and put in a technical response and then one of the non-technical people throws a fit and responds with a "why am *I* getting *THESE* emails???" rants, with even more directors and managers and execs CC'd on it.
:rage:
Title: Re: Current frustration...
Post by: Otanx on August 17, 2020, 08:46:31 AM
Then you make an email chain just for the tech guys. An email from the tech group gets forwarded to management, and they complain they are not being kept in the loop because there are project emails that are not getting forwarded to them.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on August 17, 2020, 09:29:29 AM
Quote from: Otanx on August 17, 2020, 08:46:31 AM Then you make an email chain just for the tech guys. An email from the tech group gets forwarded to management, and they complain they are not being kept in the loop because there are project emails that are not getting forwarded to them.
-Otanx
^ THIS
:facepalm1:
Title: Re: Current frustration...
Post by: icecream-guy on August 18, 2020, 04:31:26 PM
latest, self diagnosed with mouse finger, that or arthritis in my right pointer finger knuckle joint, got a weird painful lump on the side of my finger, frustration is trying to learn left handed mousing, and not using right index finger for nothing, not even typing. have realized how important index fingers are, from pointing, writing, typing, opening car doors, cans of beer,....
Title: Re: Current frustration...
Post by: deanwebb on August 19, 2020, 09:52:12 AM
Owowowowowowowowowowowowowowow
I can get some major stiffness/pain in my right hand that I compensate for with a hand/wrist brace and upper arm sleeves and generally taking things easy when it flares up. I live in dread of losing function in my hand.
Title: Re: Current frustration...
Post by: Otanx on August 19, 2020, 10:18:51 AM
Man that sucks. Sorry to hear you are having issues. I hope it gets better. I am with Dean. I fear loss of my hands, and eyes. I had some wrist pain years ago, and moved to a ergo keyboard for most of my work, and have not had any more problems.
-Otanx
Title: Re: Current frustration...
Post by: icecream-guy on August 19, 2020, 03:48:50 PM
listening to music on the computer and using voice command input does not work so well.. :smug:
Title: Re: Current frustration...
Post by: deanwebb on August 20, 2020, 10:08:56 AM
Title: Re: Current frustration...
Post by: config t on August 25, 2020, 06:44:28 AM
KlasOS and trying to find command references and configuration guides for the red enclave on Voyager kits. Frustrating.
Title: Re: Current frustration...
Post by: deanwebb on August 25, 2020, 09:02:12 AM
Questions of billabillity and unbillabillity.
Title: Re: Current frustration...
Post by: config t on August 26, 2020, 09:25:28 AM
KlasOS is a "cisco-like" layer 3 switch CLI
What they fail to describe is that it is buggy as hell. Like if you delete major configs they will no longer show up in the "show run" output but they will still be there running in code until you reboot. Sometimes, they will still be there anyway after you reboot and the only option is to wipe it and completely reload the config.
Several hours of head scratching that I will never get back.
Title: Re: Current frustration...
Post by: deanwebb on August 26, 2020, 12:14:40 PM
What they fail to describe is that it is buggy as hell. Like if you delete major configs they will no longer show up in the "show run" output but they will still be there running in code until you reboot. Sometimes, they will still be there anyway after you reboot and the only option is to wipe it and completely reload the config.
Several hours of head scratching that I will never get back.
Is there a "really delete" switch you have to use with that?
Title: Re: Current frustration...
Post by: config t on August 27, 2020, 01:13:49 AM
What they fail to describe is that it is buggy as hell. Like if you delete major configs they will no longer show up in the "show run" output but they will still be there running in code until you reboot. Sometimes, they will still be there anyway after you reboot and the only option is to wipe it and completely reload the config.
Several hours of head scratching that I will never get back.
Is there a "really delete" switch you have to use with that?
Router>en Router#config t Router(config)#really delete it ^ % Invalid input detected at '^' marker.
Router(config)#
No dice! :XD:
Title: Re: Current frustration...
Post by: deanwebb on August 27, 2020, 10:10:35 AM
^ Put in the FR with your account team!
Title: Re: Current frustration...
Post by: config t on August 31, 2020, 11:43:34 AM
Trying to convince a unit in the field that a disadvantaged (SATCOM) network sitting behind a TACLANE is definitely not going to achieve VoIP and web access capabilities with wild swings of 10% - 70% packet loss on the transport.
This has been going on for three days. They don't want to believe me 'cause SATCOM guy say "but the signal is good"
Weeks like this are when I perform the "is the money still worth it" calculation.
Title: Re: Current frustration...
Post by: deanwebb on August 31, 2020, 02:16:42 PM
Just got finished with a scoping call when someone asks the question, "Are those APs independent or running off the WLCs?"
Because if running off WLCs, our scoping estimate still stands. We good.
If they are *independent* and we have to *manage* communications to *6000* APs...
:rage:
Title: Re: Current frustration...
Post by: deanwebb on September 01, 2020, 08:38:20 PM
Meeting with a different region was supposed to start 7 mins ago... if it was canceled, that would have been nice to know before I rearranged my evening around the call...
Title: Re: Current frustration...
Post by: Otanx on September 02, 2020, 05:04:18 PM
I try really hard to only accept meetings that were scheduled at least a full work day in advanced. Obviously exceptions for critical stuff, but I need to be able to plan my day, and pop-up meetings kill my productivity. Also pop-up meetings usually are not planned very well, and nothing really comes out of it except a lot of "I don't know I will have to find out and get back to you"
My rant for today? Comparing dates. I just can't get my head around it. Is yesterday less than today? or greater than? I just wrote the code to do this five minutes ago, but I already can't remember if my compare is > or <. It just does not make sense to me.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on September 02, 2020, 07:42:31 PM
In vendorland, no one can hear you scream when you're on mute. But you gotta take the meetings when the customers call for them unless it's 100% not doable. When I'm dealing with good customers who need help quickly, I am flexible. We also tend to have an agenda like "finish the stuff we were working on yesterday", which helps to keep things on track.
Yesterday is less than today if you use yyyymmdd format. 20200901 is yesterday and that is less than today, 20200902. I use yyyymmdd format in the titles of all my time-sensitive documents.
Title: Re: Current frustration...
Post by: icecream-guy on September 03, 2020, 08:14:25 AM
Quote from: deanwebb on September 02, 2020, 07:42:31 PM In vendorland, no one can hear you scream when you're on mute. But you gotta take the meetings when the customers call for them unless it's 100% not doable. When I'm dealing with good customers who need help quickly, I am flexible. We also tend to have an agenda like "finish the stuff we were working on yesterday", which helps to keep things on track.
Yesterday is less than today if you use yyyymmdd format. 20200901 is yesterday and that is less than today, 20200902. I use yyyymmdd format in the titles of all my time-sensitive documents.
now if we all could only agree on using 3 digit IPV4 addressing scheme 192.168.001.010 so the IP's can sort properly in a spreadsheet.
Title: Re: Current frustration...
Post by: deanwebb on September 03, 2020, 10:04:13 AM
now if we all could only agree on using 3 digit IPV4 addressing scheme 192.168.001.010 so the IP's can sort properly in a spreadsheet.
:applause:
Title: Re: Current frustration...
Post by: icecream-guy on September 03, 2020, 04:21:08 PM
todays frustration is a list of problem that I have no solution to, e.g.
customer says its a network or routing issue, since they are unable to access database after 2 or 3 successful connections swear it is not a application issue, since no issues in another data center for same application
one of my firewalls, management interface went unreachable in middle of night, port flaps, and device cannot be managed except oob why? i think its optics, but 60 mile drive to replace some unknown optic and TAC engineer took 24 hours to get back to me. I am trying to determine a way in fx-os to determine installed optic type. so I know what I need to bring and not waste a trip.
I can go on.....
Title: Re: Current frustration...
Post by: Otanx on September 03, 2020, 06:16:14 PM
Thanks Dean. That actually makes sense. The funny thing is I do the same thing with my files, but never correlated that to ><. One of my coworkers also mentioned Unix Epoch. Newer time has more seconds than older time so it is greater. The best part was I went to show my coworker what I did, and when I ran the code it bombed out. They patched the system Tuesday night and changed how the date is reported. It was a text string as YYYY-MM-DD, and now it is returned as a HEX String. 30 minutes of converting Hex to Int to String, and a quick if statement to figure out if I have to do hex conversion, and I made it work again.
Maybe not your problem ristau, but we had an issue with an application and database. The application would open a session with the database server. It would use that connection for all look ups. However, if nobody was using the application there was no traffic. The firewall would timeout the TCP session after four hours of 0 bytes. Usually around midnight. Then in the morning the application would bomb, and they had to restart it. Nobody ever coded in a recovery for a failed database connection. We ended up having to change the idle TCP timeout on the firewall to something like 96 hours so it wouldn't die over long weekends. I tried to get them to just setup a database check to do tests every 15 minutes, but they wouldn't go for it.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on September 08, 2020, 10:19:54 AM
Wow. Actually having a customer argue with the DC guys because we're asking for a 1G copper interface and they want us to use a 10G fiber interface. We can do that no problem, but the customer then has to order 10G cabling if they go that route.
:facepalm3:
Title: Re: Current frustration...
Post by: wintermute000 on September 09, 2020, 07:50:26 PM
whats cabling, i thought that we cloud all the things
Title: Re: Current frustration...
Post by: config t on September 10, 2020, 05:22:16 AM
Layer 1? It's like Layer 2, right? It just magically works.
Title: Re: Current frustration...
Post by: Otanx on September 10, 2020, 09:23:58 AM
Layer 2? Is that like Layer 3? It just magically works. Wait what do we pay you network people for? Everything just works.... Only funny because it's true.
-Otanx
Title: Re: Current frustration...
Post by: config t on September 10, 2020, 02:13:38 PM
Till that dual hub DMPVN single cloud overlay on top of a SATCOM MRT mesh network faults during the failover test. >:D
I was a champion today. Not gonna lie though.. I totally forgot that the timers on RIP are just updates and not route flaps :)
Days like today are when I'm reminded that I love this business.
Title: Re: Current frustration...
Post by: deanwebb on September 11, 2020, 02:27:44 PM
I think I'll just start telling everyone to not worry about cables, just cloud it. :smug:
Title: Re: Current frustration...
Post by: config t on September 15, 2020, 11:14:31 AM
For government contractors it has become a pattern for me to see situations where they pay 4 guys the same and 1-2 will actually have enough knowledge to do the work while the rest freeload with zero intent to learn but have big mouths when it comes to sounding off about how long they have been professionals.
In our case it became apparent today that only one guy has the knowledge while 3 others freeload. I heard him on the phone today describing the difference between exec, priv exec, and global configuration modes, as well as point out that the "shiny thing plugged into one of the holes on the right side of the switch" was an SFP, and attempt to get the guy to describe the shape of the fiber interface since he had no idea what SC, LC and FC even means.
Don't get me wrong. I understand that we all start somewhere. But these guys have been there 1 year plus. One of them has been there for over 4 years and couldn't describe the WAN if he tried. The minimum requirement for the job is allegedly 5 years experience.
Title: Re: Current frustration...
Post by: deanwebb on September 15, 2020, 11:27:28 AM
I have a customer that has trouble with numbers bigger than 2.
No lie, as soon as we hit 3 or 4 of something, the arguments start and the confusion sets in.
Title: Re: Current frustration...
Post by: icecream-guy on September 15, 2020, 03:43:20 PM
customer: we need a new VPN to Azure
Requester: (Friday) Dave, we need you to work on this project.
Me: what are the requirements.
Requester: we need to do this next Saturday.
Me: No, I need dest networks IP's, transit networks IP's, security policy, etc.
Requester: No issue, I requested you build a new firewall context, should be a clean slate, just create a blank CR and you can work the details later
Requester: I need this on Monday so we can push to tech review asap.
Me: I can't build 2 entire firewall contexts configurations in a day. especially with no security policy direction
,,..so usually it's a 3-4 week process, engineering review board should have been last week. tech review board yesterday. approval tomorrow, for it to go off next Saturday. nobody has reviewed CR
Routing engineer: Can you push CR to Technical Review?
Me: NO CR is incomplete, it will be rejected, finish up CM with missing information and I will push to tech review
PM: Dave, can you finish up configs today (Tue afternoon)
Me: Did this morning.
i dont know how this will fit into the normal process without going emergency, then with end of fiscal year....
never pushed CR .
Arrrrrgh...... customer wants it implemented Saturday must have
Title: Re: Current frustration...
Post by: icecream-guy on September 15, 2020, 07:10:16 PM
windows 1903 update forced on me after lockup and hard reboot at least I am up and running again
Title: Re: Current frustration...
Post by: deanwebb on September 16, 2020, 09:13:39 AM
Mideast time zones... their early is too late and their late can often be too early...
Title: Re: Current frustration...
Post by: icecream-guy on September 16, 2020, 03:34:47 PM
haha people at work call me during their working hours 9-5 Im 6-2, so I call them back at 6AM when I can. some are not understanding but most oblige since we are 6AM - 6PM primary support
AND don't get me started on Cisco TAC. and their weird hours around the world, and trying to get someone to work with at a reasonable hour. without having to requeue.
Title: Re: Current frustration...
Post by: deanwebb on September 17, 2020, 10:25:29 AM
Back to back to back to back meetings where I have to carve out bio-breaks.
Title: Re: Current frustration...
Post by: Otanx on September 23, 2020, 01:54:40 PM
Current frustration: The phrase "Faliure is not an option". What? Is the world going to end if my project schedule slips two weeks? I feel like we might be on the nightly news if that was the case. I am pretty sure failure is an option. You just don't want to deal with it. I guess we will just CYA, and deal with it when it happens.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on September 23, 2020, 02:37:00 PM
Failure isn't an option, true. It's an end-state condition resulting from poor selection of options in the planning, allocation, scheduling, and execution phases of the project. :smug:
Two teams go into a game, both having decided that failure is not an option. For at least one of those teams, the end-state is going to be at variance with their preferences.
Title: Re: Current frustration...
Post by: Otanx on September 23, 2020, 03:31:58 PM
Quote from: deanwebb on September 23, 2020, 02:37:00 PM Failure isn't an option, true. It's an end-state condition resulting from poor selection of options in the planning, allocation, scheduling, and execution phases of the project. :smug:
I may have a new signature line for my work emai. :))
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on September 23, 2020, 04:41:50 PM
Quote from: deanwebb on September 23, 2020, 02:37:00 PM Failure isn't an option, true. It's an end-state condition resulting from poor selection of options in the planning, allocation, scheduling, and execution phases of the project. :smug:
I may have a new signature line for my work emai. :))
-Otanx
Enjoy!
:yeahright:
Title: Re: Current frustration...
Post by: config t on September 24, 2020, 01:26:37 PM
Classic.
Title: Re: Current frustration...
Post by: Otanx on October 21, 2020, 12:35:11 PM
We started "COVID Schedules" back in March. Basically we work half our hours at home, and half in the office. While I have always suspected that most people are slacking off when working from home I just got told by several people that they just check email every couple hours, and that is it. They said this in front of their boss who seems OK with that. The reason it came up is I wanted them to actually do work on their "home" days. They wanted to assign it to the "office" team for that day.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on October 21, 2020, 12:54:07 PM
Quote from: Otanx on October 21, 2020, 12:35:11 PM We started "COVID Schedules" back in March. Basically we work half our hours at home, and half in the office. While I have always suspected that most people are slacking off when working from home I just got told by several people that they just check email every couple hours, and that is it. They said this in front of their boss who seems OK with that. The reason it came up is I wanted them to actually do work on their "home" days. They wanted to assign it to the "office" team for that day.
-Otanx
Wow. Meanwhile, the people that are 100% at home pretty much get stuff done in order to alleviate the boredom of being 100% at home. :smug:
Title: Re: Current frustration...
Post by: config t on October 22, 2020, 02:04:25 AM
We are due for semi-annual performance evals and I'm told they will all be "non-observed" since the entire team has been working from home. I'm assuming they mean CONUS, Europe and Japan, because here in Bahrain-land we never left the office. If anything I have been busier this year than I was last year.
Quote from: Otanx on October 21, 2020, 12:35:11 PM We started "COVID Schedules" back in March. Basically we work half our hours at home, and half in the office. While I have always suspected that most people are slacking off when working from home I just got told by several people that they just check email every couple hours, and that is it. They said this in front of their boss who seems OK with that. The reason it came up is I wanted them to actually do work on their "home" days. They wanted to assign it to the "office" team for that day.
-Otanx
So much for work ethic. This would super p*** me off.
Title: Re: Current frustration...
Post by: icecream-guy on October 22, 2020, 08:09:01 AM
Quote from: config t on October 22, 2020, 02:04:25 AM We are due for semi-annual performance evals and I'm told they will all be "non-observed" since the entire team has been working from home. I'm assuming they mean CONUS, Europe and Japan, because here in Bahrain-land we never left the office. If anything I have been busier this year than I was last year.
Quote from: Otanx on October 21, 2020, 12:35:11 PM We started "COVID Schedules" back in March. Basically we work half our hours at home, and half in the office. While I have always suspected that most people are slacking off when working from home I just got told by several people that they just check email every couple hours, and that is it. They said this in front of their boss who seems OK with that. The reason it came up is I wanted them to actually do work on their "home" days. They wanted to assign it to the "office" team for that day.
-Otanx
So much for work ethic. This would super p*** me off.
We've been troubleshooting AnyConnect "Internal System Error" pop-ups, users are complaining. so after much troubleshooting, DART bundles, and ISE logs, Cisco thinks thye have the cause.... when computers come out of hibernation, from a particular state. the user gets the message...
now... I wonder why these users are letting their laptops go into hibernation mode if they are working all day long. seems to me that it shouldn't be happening.
personally, I have to account for every activity and time spent on each activity, for every day, and report daily.
Title: Re: Current frustration...
Post by: deanwebb on October 22, 2020, 09:43:10 AM
^ 99% of all client problems are Windows problems, and 99% of all Windows problems are Windows user problems.
Maybe by percentages are off a bit, but not by much. :smug:
Title: Re: Current frustration...
Post by: Otanx on October 22, 2020, 02:24:03 PM
I kind of feel a lot of Windows issues are poor admins, not really user issues. Microsoft has made it so easy to admin their systems (which is a good thing) that the admins don't have to learn the underlying technology. So when something goes wrong if a reboot doesn't fix it then many are at a loss. There are what I consider good admins that learn how everything works under the hood, but many don't bother.
-Otanx
Title: Re: Current frustration...
Post by: icecream-guy on October 23, 2020, 08:12:18 AM
we always joked about "L" class users... Lusers wayyy over their heads.
Title: Re: Current frustration...
Post by: config t on October 24, 2020, 06:05:51 AM
Layer 0 problems
Title: Re: Current frustration...
Post by: deanwebb on October 24, 2020, 09:03:33 AM
More than once, I've said that we can dramatically improve throughput, performance, stability, and security if we don't allow anyone to access the network.
Title: Re: Current frustration...
Post by: Otanx on October 26, 2020, 01:33:20 PM
I always heard it the other way. Layer 8: Users Layer 9: Politics
New frustration. Customers canceling scheduled work less than 24 hours prior. Important enough customer that I can't even bill them for not enough notice. Just get to suck it up. Oh, well. More time for me to plan and test.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on October 27, 2020, 12:30:56 PM
Between a canceled meeting and a meeting that drags on without the right participants to make a decision - effectively meaning we have to have the meeting again - I prefer the canceled meeting. Still a frustration, but at least we don't have to endure it all twice.
Title: Re: Current frustration...
Post by: icecream-guy on October 30, 2020, 08:21:24 AM
heard a good one yesterday.... you put the F U in fun.
Title: Re: Current frustration...
Post by: deanwebb on October 30, 2020, 09:55:54 AM
I had never heard it put that way until yesterday.
It usually is that way. And if that 20% leaves and gets hired with 80% material, then it's a recipe for 100% trainwreck IT within a few weeks.
Title: Re: Current frustration...
Post by: deanwebb on November 16, 2020, 07:03:53 PM
I am 40 minutes into a 3-hour call for upgrade support.
We are at 30% file copy of a 3GB file.
Suddenly, both customers drop off the call. I try to email them, no response. I try to email their manager, no response.
That was 20 minutes ago.
And as I type this, the customer *just* reconnected. :rofl:
Title: Re: Current frustration...
Post by: deanwebb on November 16, 2020, 07:12:34 PM
And he wandered off again just a few minutes after reconnecting...
:rage:
Title: Re: Current frustration...
Post by: deanwebb on November 16, 2020, 07:46:56 PM
He's back and now we're finding that some networking ace decided it would be a good idea to block access from our main appliance to about half the child appliances. Whether it was a routing table mistake or a firewall mistake, I don't know yet.
UPDATE: It's a sad day for security guys everywhere.
It *was* the firewall.
:disappoint:
Title: Re: Current frustration...
Post by: config t on November 22, 2020, 05:08:17 AM
The same guys who had to request password resets on our IDS/IPS three times because nobody could seem to write down a password sent new devices to replace our old hardware. In the email they said the password is
Title: Re: Current frustration...
Post by: Otanx on November 23, 2020, 08:11:40 AM
did you try "example* OR example!" without the quotes?
-Otanx
Title: Re: Current frustration...
Post by: config t on November 23, 2020, 11:16:53 PM
I knew someone was going to say that..
After a few emails they asked me to recover and reconfigure the device.
Title: Re: Current frustration...
Post by: deanwebb on November 24, 2020, 09:39:53 AM
Today I will give thanks for console recovery KB articles. :rofl:
Title: Re: Current frustration...
Post by: Otanx on January 06, 2021, 11:41:53 AM
Anti-frustration post: I got an email from my boss. Next time I am in the office I have to clear out my desk because I was given an office. I think the first time in my career I have my own office. I even have a window. It is frosted so I can't see anything, but I have one. Going in tomorrow to take a look, and see what I need to do.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on January 06, 2021, 01:57:39 PM
Nice problem to have, and office! :D
Title: Re: Current frustration...
Post by: config t on January 07, 2021, 04:33:45 AM
Nice. I often wish I had a door to close instead of needing to put on noise-canceling headphones.
A couple years ago I had a co-worker who was known to be pretty loud. I guess enough people complained because they moved us out of the way to a corner in cubicle-ville. We ended up with a big window and a view of the mountains :)
Title: Re: Current frustration...
Post by: deanwebb on January 07, 2021, 09:43:34 AM
Big question is whether or not you get your own server rack and lab gear in that new office. :smug:
Title: Re: Current frustration...
Post by: Otanx on January 07, 2021, 12:33:27 PM
No rack in my office. I wouldn't want all the noise. I do have a rack in our data center that is for my lab. 2x50A208v power, and all the networking I could want. It is nice being the guy who designed and built out the data center. Not a bad office. All the offices here are "Fish bowl" style with glass walls. Except mine is one of two that are frosted so nobody can see in. I heard it was the HR managers office for the last company that was in this space.
-Otanx
Title: Re: Current frustration...
Post by: icecream-guy on January 07, 2021, 04:06:50 PM
Time to see if there is budget for a secretary. then you can have a nice rack in your office....har...har..... and with frosted glass, nobody can see the "noise" ....har...har.....
Title: Re: Current frustration...
Post by: Otanx on January 07, 2021, 09:38:03 PM
Even if I was inclined to do that my wife sits four rows away from my new office. I probably wouldn't last long if I tried that. Rumor is she my be moving desks in the second wave, and end up in the row right outside my office.
Now that I have moved into the office I can say the whole moving people thing is stupid. They are moving people to put the teams together, but with COVID only half the teams are allowed to be in the office at one time anyway so it doesn't matter that much. What they should do it take the desks from those that are now 100% remote, add them as rooms in Exchange, and let people reserve them if they need to be in the office. People who come in more than 50% of the time can have a desk. Everyone else hot desks.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on January 08, 2021, 07:47:17 AM
My wife works very close to my office, as well. And she's making chili cheese fries for lunch, too! :D
Title: Re: Current frustration...
Post by: icecream-guy on January 10, 2021, 04:42:21 PM
Title: Re: Current frustration...
Post by: deanwebb on January 11, 2021, 07:24:24 AM
We're going more for "Texas Poutine", nice and easy, something that won't interrupt my 2PM meetings. :smug:
Yeah, I'm lookin' at YOU, Chipotle!
Title: Re: Current frustration...
Post by: Otanx on April 20, 2021, 11:20:02 AM
FIPS mode on Palo Altos. To enable, boot into maintenance mode. Tell it to enable FIPS. This factory resets the box. So far so good. Except FIPS mode also disables all input from the console after boot. You can use the console to do a factory reset, and disable FIPS, but you can't login, and say change the default management IP to something useful. Glad I figured this out at our local data center.
-Otanx
Title: Re: Current frustration...
Post by: deanwebb on April 21, 2021, 08:35:55 AM
Quote from: Otanx on April 20, 2021, 11:20:02 AM FIPS mode on Palo Altos. To enable, boot into maintenance mode. Tell it to enable FIPS. This factory resets the box. So far so good. Except FIPS mode also disables all input from the console after boot. You can use the console to do a factory reset, and disable FIPS, but you can't login, and say change the default management IP to something useful. Glad I figured this out at our local data center.
-Otanx
Ouch
Title: Re: Current frustration...
Post by: heath on December 16, 2021, 11:18:03 PM
My current frustration is my Chief Technology Officer. See attached example.
Title: Re: Current frustration...
Post by: deanwebb on December 17, 2021, 07:59:07 AM
Oof.
Title: Re: Current frustration...
Post by: config t on June 22, 2022, 08:56:25 PM
Troubleshooting RADIUS voice fail open.. PCs are also failing open but I didn't configure that.. customer at the end of the day tells me his old RADIUS configs are still in the test switch.
Title: Re: Current frustration...
Post by: deanwebb on June 23, 2022, 10:55:52 PM
Quote from: config t on June 22, 2022, 08:56:25 PM Troubleshooting RADIUS voice fail open.. PCs are also failing open but I didn't configure that.. customer at the end of the day tells me his old RADIUS configs are still in the test switch.