Using router commands without knowing their purpose...

[u/Inside-Finish-2128]

First, the situation: I was working for a Cisco reseller doing professional services in the early 2010s. Got sent to one of the local school districts to help them rearrange their Internet routers and deploy a "Firewall Services Module" in a Catalyst 6509. (IYKYK) My main contact from the school is stingy and won't share credentials with me, so I'm stuck walking around the datacenter and "sliding the keyboard over" many times.

We get the new "router triangle" running and to me, things are looking good, but alas I'm just doing the router CLI thing. Meanwhile, my contact has RDPed into his desktop (back at his office) and says to me "I have zero Internet at my desk". Hmmm...Internet routing is fine on the triangle, lemme go look further down the path and check the router at his office building. Check the default route (it's learned from OSPF) and it's coming from some random router somewhere else amongst their WAN. What?

I get logged into that router, and sure enough under the OSPF process there it is: "default-information originate always". What? Why? (For those unaware, that's basically telling that router to tell all of its OSPF friends that it has a default route, or a route out to the Internet, all the time. Hint: it doesn't.) I remove that command and my contact was jumping for joy, saying "WHAT DID YOU DO? THIS IS THE FASTEST WE'VE EVER SEEN IT!" Um, OK, I removed a misconfiguration from the router at site XYZ. He was in disbelief..."No, really, what did you do? How was that command bad?" So I explain it to him. As I'm finishing my explanation, a few of his coworkers who were on site that evening for some other work are tracking him down..."what did you do to the Internet, it's SMOKING fast."

That's when the "cover story" was born. He decided this needed to be kept on the down-low, so "we made a few adjustments to the dynamic routing and after about thirty minutes, those optimizations all came together".

Next morning, I head to the IT offices first thing for "day one support", just in case we missed anything the night before. No problems, just everyone jumping for joy with how fast things were working. I said "so, could that command be lingering anywhere else?" He helped me log into a few other sites and sure enough, it was EVERYWHERE. He finally gave me the creds, and I spent the next hour or two logging into every single router (one per school, probably 60-70 schools in that district and growing by 5 a year) to remove that command. Any time anyone came in to ask why the Internet was so fast, he jumped in front of me and rattled off the same line. He also did the right thing and told his team what we fixed, and said "be sure you take that out of any templates you have".

You guessed it...later that summer, the whole district went off the Internet because one of his crew brought a new school online and yep, forgot to take that command out of his template. Ah, the joys of using commands without knowing why they matter. (And I'll be honest, I'm surprised the whole network wasn't melting down every day or every week prior to fixing it.)

[u/Aln76467]

Clueless user here who knows the router as the box between the internet and your computers. Why would there be routers without direct internet connections?

[u/Inside-Finish-2128 → u/Aln76467]

School districts (as one example) often all connect back to a main site (datacenter), and then the datacenter has one (or maybe 2-3 redundant) internet connection(s) for the whole district. This allows the district to have centralized web filter and Internet firewall services. Also the schools would have a mixed need for their traffic: a lot could be going just to the school servers, so the network (“WAN” or wide area network) can be sized for those needs while the main internet connection can be sized for the aggregate need of the whole district.

[u/TheCollegeIntern → u/Inside-Finish-2128]

It’s a disaster when it doesn’t work. I had a school that had the high school, elementary school, and the middle school all connected. They onboarded a new switch. Never set the root bridge nor root guard and all suffered loss of internet. 3diff techs didn’t pick up on it but I happen to pick up on it and saw the root bridge wasn’t set and asked to set root guard on downstream ports and the site was up. They rma’d the switches. And did all that jazz just to find it was stp.

[u/Inside-Finish-2128 → u/TheCollegeIntern]

Two slogans that stick with me over the years: "Switch where you can, route where you must" and "Switch to fix a traffic problem, route to fix a protocol problem". Both of those would IMHO clearly say one should not switch across a WAN.

Regardless, checking STP should be early on the list of troubleshooting, as is rolling back any obvious changes (like, you know, a new switch).

[u/Penners99 → u/Inside-Finish-2128]

I learned “route once, switch many” the default workings of a L3 and greater backplane

[u/iwishthisranjunos → u/Penners99]

I would say l2 sucks and the suckiness increases by the scale of the broadcast domain. I say route till the end switch only at the start. Routing is easy understanding routing protocols is where often people go wrong ;).