Random Acts of (dis)Connectivity
Summer time is busy time. July and August, when all the staff and students are out enjoying their vacation, the IT department for a public school division is hard at work. It’s one of the rare opportunities I’m allowed to unplug, replace, upgrade, reboot, and/or generally break our infrastructure, because nobody is using it. So with a full plate of projects to complete before the end of August – when random things break “on their own” it sometimes causes some confusion.
Last week I had two wireless access points stop communicating to the controller at one of my sites. Now, had there been people in the building I could have initially chalked this up to someone playing around in the wiring closet (we have switches in the weirdest, least secure places) or some kids throwing basketballs at them (yes, we have AP’s in the gyms), but I knew for a fact that this building was empty. Our Facilities guys generally let us know if there is going to be some work done that is going to involve power going down, and this definitely wasn’t the case because the switch these 2 AP’s were on was up and running.
I checked the switch and the ports that the AP’s were on showed up/up. I could even ping their IP addresses. What was strange was I could not ping the controller from that switch, nor could I ping the site gateway. While the switch appeared up and functional for all intents and purposes, it appeared as though it was simply refusing to forward any traffic. Then I checked the logs:
Aug 9 13:11:48: %ILET-1-AUTHENTICATION_FAIL: This Switch may not have been manufactured by Cisco or with Cisco’s authorization. This product may contain software that was copied in violation of Cisco’s license terms. If your use of this product is the cause of a support issue, Cisco may deny operation of the product, support under your warranty or under a Cisco technical support program such as Smartnet. Please contact Cisco’s Technical Assistance Center for more information.
Dammit! I knew that guy selling Cisco gear off the back of his unmarked white truck in the back alley was too good to be true! His prices were so reasonable, even compared to our educational discount. You know it’s so hard to work within an ever-decreasing hardware budget these days.
Alright in reality this batch of switches was bought from a legitimate Cisco Partner, one of Canada’s largest, and I hadn’t touched them since rolling them out a couple of summers ago. They had the out of the box IOS 12.2(22)SE that they had shipped with, and I had about 30 of them around. These were a batch of 2960S switches that were replacing some aging 2950 units that had calcified and grown long grey beards.
I started checking several others and quickly realized this was an isolated issue, limited to this one switch.
Licensing, Contracts, Smartnet, Oh My!
As part of our ongoing efforts to reduce expenses and save what little budget we have, we don’t get Smartnet on our access switches. Ultimately all of the current 2xxx and 3xxx switches (sometimes referred to as the DSBU or Desktop Switching Business Unit switches) come with Cisco’s Enhanced Limited Lifetime Warranty which includes free IOS software updates. The lifetime warranty is arguably Cisco’s response to pressure from HP’s long-standing lifetime warranty on their Procurve switches, and free IOS updates are always nice:
Software UpdateQ. Can I obtain a “no additional cost” Cisco IOS® Software update for the Cisco Catalyst 2960 Series?A. Yes. Cisco offers ongoing Cisco IOS Software updates for certain fixed-configuration and stackable Cisco Catalyst switches at no additional cost. For the life of the product, updates within the Cisco IOS Software package purchased (LAN Lite and LAN Base) will be made available.Note that upgrades are different from updates. For example, an upgrade from the IP Base package to IP Services package provides significant new function; therefore, this upgrade requires the purchase of a software license upgrade. Updates are incremental software features and bug fixes that are released within a licensed Cisco IOS Software package.This statement supersedes any previous warranty or software statement and is subject to change without notice.Q. How do I get a “no additional cost” Cisco IOS Software update for the Cisco Catalyst 2960 Series?A. Visit http://www.cisco.com, click “Downloads,” and select “Switch Software.” Downloading software requires a Cisco.com username and password. If you do not have a Cisco.com username, you can obtain one by clicking “Register” at the top of any Cisco.com Webpage.
Since we also keep a couple of spare switches around we don’t need overnight or next-business-day hardware replacement, Smartnet is an expense we can live without. This sometimes causes issues when trying to RMA a product…
Someone Tell TAC
I found a few posts in the Cisco Support Community from folks who had run into this same issue. It seemed a general consensus that there was no real solution. It appears a small batch of 2xxx and 3xxx series switches were having this issue as far back as March of 2010. There had been a Bug ID for the issue but the workaround had been to update the IOS to 12.2(44)SE or later, with no other workaround. Since I was already past that update, I decided it would be easier to just call TAC.
Apparently nobody tells TAC about lifetime warranties or free software. It always seems to turn into an argument with the front line support agents because as soon as they see there is no support contract, they won’t open a case. It’s pretty binary in their world…no contract equals no support.
So how do you get support for a lifetime warranty product? Just tell them you want to RMA the device. Cisco is more than happy to simply drop ship a new switch than to waste time troubleshooting and that’s fine by me. I’m not a CCIE so I don’t get bumped up to a level 2 engineer right away, and I’d rather not waste time on the phone or via email with a level 1 engineer. It’s in the best interest of everyone involved to just send me a replacement.
I’d have to say all of my experiences with Cisco’s RMA process have been excellent. Even without 8×5 or NBD service, they are quick and painless to deal with. A box arrives with a pre-paid UPS waybill and I happily ship the dead product back to them in the same box the replacement came in. They even have a Canadian location I can ship to so I don’t have to fill out reams of Customs paperwork (I’m looking at you, Ruckus Wireless) or pay brokerage fees.
Thanks for reading. As always feedback or comments are welcome. I have to run, Tony is here with some deeply discounted UCS servers and Rolex watches I need to take a look at.