Cisco Catalyst Wifi, Take Two

Posted on December 10, 2018 by NetworkCanuck

On November 13th, Cisco announced their next-generation wireless platform with the release of the Catalyst 9800 Series Wireless Controller.

You read that right, the next WLC platform from Cisco is running on Catalyst and expands Cisco’s DNA-Center architecture into the wireless space.

The Catalyst 9800 controllers come in a variety of form factors. The option for a standalone hardware controller is still here with the 9800-40 and 9800-80, or the 9800 series can be run as a VM in a private or public cloud. A third option is now to run embedded wireless on the Catalyst 9k series switches.

Embedded wireless controllers on Catalyst switches…that sounds familiar, doesn’t it?

Cisco made a similar move a few years ago with an architecture called Converged Access. This embedded the wireless controller functionality into IOS XE on the 3650 and 3850 access switches. For various reasons, it did not live up to expectations, and Cisco killed it in IOS XE Everest 16.5.1a in late 2017.

Cisco and Aironet

Cisco acquired Aironet Wireless Communications in 1999 for $799M. Since then, Cisco wireless access points have generally been referred to as “Aironet” products by name. ~~This includes the software that runs on the wireless controllers and access points, AireOS.~~

AireOS came from Cisco’s acquisition of Airespace in 2005. Airespace were the developers of the AP/Controller model and the Lightweight Access Point Protocol (LWAPP), which was the precursor to CAPWAP.

(Credit to Jake Snyder for correcting me on the origins of AireOS)

Whatever AireOS version is running on your wireless controller is the same that you have on your access points. Cisco has developed the platform to be what it is today, and very little of it remains what was once the original AireOS.

With this iteration, or rather re-invention of the Wireless Controller, Cisco have highlighted three key improvements to their predecessor wireless software.

Always-On

Controller redundancy is always critical to prevent downtime in the event of failure. Here, Cisco are touting stateful switch over with an active standby model in which client state is maintained across the standby controller, offering no downtime for clients in the event of a failure.

Patches and minor software updates now will not change the base image of the controller. Updates can be done without client downtime. Patches for specific AP models can be done without affecting the base image or other access point models with per-AP device packs. These are installed to the controller and then pushed only to the model of AP they are for.

New AP models can also be joined to the controller without impact to the overall base image with the AP device packs, allowing new hardware to join an existing environment without a major upgrade.

Citing “no disruption” base image/version upgrades, the new 9800 controllers can be updated independently of the access points, whereas previously the software version running on the controller and access points was coupled. Upgrades were done to the controller, and then pushed to the access points. More often than not, this resulted in interruption to clients on affected access points, some rebooting of the controller and AP’s was inevitable, and quite often, some orphaned access points that never quite upgraded properly or failed to rejoin the controller.

Cisco have made many improvements to the upgrade process over the years, including staged firmware upgrades, however in large wireless deployments, firmware upgrades would not generally be considered zero-downtime.

With the new controller architecture using an RF-based intelligent rolling upgrade process, Cisco has aimed at eliminating some of these issues. During the upgrade process, the standby or secondary controller is first upgraded to the new image. You can then specify a percentage of access points you would like upgraded at once (5%-25%), and the controller then determines which AP’s should be upgraded using the AP neighbor information and # of clients on each AP. APs with no clients are upgraded first. Clients on access points that are to be upgraded are steered toward neighboring access points in order to prevent interruption in service.

The idea of steering clients to other access points or 5Ghz radios instead of 2.4Ghz radios isn’t new, and because I’m not a wireless expert I won’t comment on exactly how it’s done, but it is my understanding that it is difficult to guarantee that the client will “listen” to the steering mechanism. I feel even with this intelligent RF behind this upgrade process, some clients will inevitably experience a loss of connectivity during the upgrade process.

Once the access point is upgraded, it then joins the already-upgraded controller, and resumes servicing clients.

After all access points are joined to the upgraded controller, the primary controller begins its upgrade process.

Secure

Encrypted Traffic Analytics was first announced as part of the Catalyst 9K switch launch, and uses advanced analytics and Cisco Stealthwatch to detect malware in encrypted flows, without the need for SSL decryption. ETA is now available for wireless traffic on the 9800 platform, if deployed in a centralized model, meaning all wireless traffic is tunneled back to the controller.

This is a great feature considering the only other option for gaining visibility into encrypted traffic is usually some form of sketchy certificate man-in-the-middle voodoo. In many situations this works okay for corporate domain-joined machines as here you can control the certificate trusts, but if you provide wireless to any BYOD devices or to the general public in any way, this often results in people not using your wireless because of certificate issues.

Deploy Anywhere

Cisco is offering a lot of flexibility in deployment options for this new wireless controller.

Branch offices can look at the embedded software controller on Catalyst 9K switches for up to 200 APs, and 4K clients.

Edit: Since the original publication of this post, I’ve clarified that the option to run the 9800 controller on a Catalyst 9K switch is only available as an SD-Access Fabric Mode deployment option. SD-Access requires DNA Center. This is an expensive proposition for what could truly have been a fantastic option for small/medium branch office deployments.

Private or public cloud options are available on KVM, VMware, Cisco ENCS, and will be available on AWS. These options support 1000, 3000, and up to 6000 APs, and 10K, 32K, and 64K clients. The AWS public cloud option only supports FlexConnect deployment models, which makes sense as tunneling all client traffic back to your controller in this case would get expensive quickly.

Physical appliance options include the 9800-40 at 2000 APs, 32K clients and 40Gbps (4x10Gbps interfaces), as well as the 9800-80 at 6000 APs, 64K clients, and 80Gbps (8x10Gbps interfaces). The 9800-80 also has a modular option which allows for GigE, 10GigE, 40GigE, and 100GigE uplinks.

Each of these options have identical setup, configuration, management, and features.

Lessons Learned?

Overall, the presentation of this new wireless platform seems solid. Cisco have acknowledged the problems with Converged Access, and have seem to have checked off all of the missing boxes from that first attempt. Feature parity was a big one, and Cisco insists here that all features will be the same up to the existing controller software version 8.8 (current version is 8.5 at the time if this post), so that would give Cisco and their customers quite a bit of time to flesh out the new architecture.

Now, AireOS isn’t going to disappear suddenly. Cisco have said that they are going to continue to develop and support the existing line of controllers and AireOS software, until they can be sure that this new architecture has been successfully adapted by their customers. Customers who previously bought into Converged Access may not be lining up to be the first customers to try out the new platform, but the popularity of the Catalyst 9K switches should provide a good foundation for the embedded controller to gain a foothold.

You can check out Cisco’s presentation at Networking Field Day 19 here:

GNS3 – An Interview

Posted on October 20, 2014 by NetworkCanuck

It used to be that the only way to get any direct, hands-on experience with any networking gear was to have access to some actual, physical equipment. It was an expensive prospect, and often out of reach for most beginners to invest significant dollars into owning their own lab gear. Back in 2000, when I first got started in this area, there were no companies offering remote rack rentals, there were no good simulators, and there weren’t companies selling refurbished networking products at a deep discount. Ebay was available, but the products you were looking at there were quite often from questionable sources and probably not genuine. Needless to say it was very difficult to gain experience working hands-on with the products you were hoping to get familiar with.

Certification training, especially on a budget, was something that was very difficult to do on your own.

Over the years projects like Dynamips and Dynagen would provide some limited simulation functionality for Cisco IOS, but in 2007 as part of a university Master’s thesis project, Jeremy Grossman would begin developing GNS3.

GNS3 would evolve to become one of the most widely used networking simulators available and many a networking student working towards certification would find it an invaluable resource for learning.

Now, in 2014, GNS3 continues its evolution.

Now, I’m a sucker for a local success story and Stephen Guppy, CEO and Co-Founder of GNS3 is located just south of me in Calgary. I thought I’d reach out to him to discuss the past, present, and future of GNS3 and talk to him a little bit about their role in training and certification for the networking professional.

In The Beginning…

Stephen recounted the story of how GNS3 began as Jeremy’s university project, and while he now remains the sole developer, there have been a number of others involved over the years. After Jeremy moved to Calgary he met Stephen replying to an ad for a roommate. Over the past few years, GNS3 has been primarily a labor of love but recently the two really started to ask the question – “What does the community want from GNS3?”

After surpassing the 10 million download mark, they decided to launch a crowd funding campaign, in the hopes that they might make GNS3 sustainable in the long-term. So far that campaign has been a great success with over 13,000 contributors and over $553K in funding. The midnight launch of the campaign actually broke Crowdtilt (now Tilt) but they were very responsive and had things back up and running quickly. They met their initial goal in 3.5 hours and hit over 100K in 18 hours.

For contributors to the campaign, the first Alpha release of the new GNS3 became available April 1^st, 2014, and Beta releases in mid-July. Their goal is to have the first version of the “final” product in December.

Fresh Start

The original GNS3 had become somewhat bloated with redundant modules and was a bit of a patchwork over the years with multiple contributors. For this new iteration Jeremy started from the ground up, with less than 5% of the original code going in. The design goal here was to develop a solid framework that could have modules added onto it. This meant simplicity when adding on support for VMware, HP, Juniper, etc.

The GUI and general feel of GNS3 hasn’t changed much so long time users of the product aren’t going through a huge learning curve with the new version either. The end result is a much more robust tool that takes up fewer resources, loads and runs faster, and is generally a lot friendlier to use.

Multi-vendor

Jeremy and Stephen had seen some interesting data over the last two years, and that was a steep decline in Cisco usage, with Stephen indicating a 25%-45% decline year over year. As they expanded into markets outside the US, other vendors were more dominant, including huge demand for Juniper and Huawei. GNS3 for many years had been known as a “Cisco” simulator but in reality, they aren’t. Their goal is to be the “CML for all vendors” and be able to offer the network engineer a platform to work on whatever vendor product they require.

The huge surge in virtualization has made it incredibly easy to integrate various vendor systems into GNS3. Not only that but with more and more vendors offering ‘free’ versions of their platforms to use in lab and test environments, access to the software has never been easier.

The L’s – Legalities and Licensing

Stephen admits they rarely have to address any legal/licensing issues because they’ve simply made sure they were always on the right side of the law.

“A lot of the things we provide are fully, freely available for anybody. The vendors provide it. XRV, 1000V, the 3600 series, Cisco provides that for free, online, anybody can download it. All that we are able to integrate within GNS3.”

They also have direct relationships with a lot of the major vendors, including Cisco, having been in talks with the product managers for CML and VIRL, as well as having other vendors approach them directly asking for integration into GNS3. Stephen mentions the relationship with the vendors should be “symbiotic” and in their discussions with various internal product managers, etc. all have indicated that GNS3 has value for them, providing highly trained people and a platform for test and development.

“We never want to do something illegal and be taken down because a lot of people depend on our product.”

On CML and VIRL

“I hope Cisco pushes the boundaries, GNS3 has needed a little competition.” Stephen says, laughing. Having known about CML and VIRL for a very long time due to their relationships within Cisco, they’ve been very excited for release of these products. Stephen calls these a “declaration to all the other vendors” and a challenge to them to provide the same level of support for the network professional.

CML, the paid, TAC-supported platform has already been released, but VIRL – the free, community-supported platform remains vaporware. However, should Cisco decide to release VIRL sooner than later, I wondered how a “free” product would impact them?

Stephen admits these products will impact GNS3 in some way, as a competing product, but a lot of the feedback from those with early access to these new platforms seems to be that while they were generally good products, people eventually moved back to GNS3.

“Cisco is building a whole new community from the ground up and that’s not easy.”

The Community

In 7 years the GNS3 community has evolved to a group that has very knowledgeable people who understand the product very well. Stephen says plans here are to really nurture this relationship and harness the people within the community, starting with a “legitimate, real community portal”. He cites Thwack (the Solarwinds community portal) as an example.

“You help out, you get points, you get a t-shirt.”

Long-Term Strategy

I’ll admit that over the course of the interview with Stephen I learned a lot about the product that I didn’t know. I had been guilty of seeing GNS3 as a “Cisco simulator” but learned that their focus has not been single-minded, and that support for pretty much any vendor was their goal.

Training and Certification remain a huge part of what they are focused on. Plans include a full CCNA platform including labs and documentation, all for free.

“We want to provide all the material and all the resources you need in order to get your certifications. We would love to do it all the way up to CCIE and equivalent expert certifications in all fields, and provide it for free. All you have to do is be a part of the community.”

This is huge. As anyone who has set a goal for a particular vendor certification, the costs associated with the training and studying are sometimes a limiting factor. For GNS3 to be able to offer community-driven material as well as a platform for labs, is incredible and speaks volumes about the goals Jeremy and Stephen have for their software.

“Networks run our lives. They make our lives possible and the network professional, while it is a very thankless task, is probably one of the most important, or I think the most important profession in the world. Vendors make it extremely difficult for them to be good at their job and I think that’s a huge miss from them. We look at training not as a source of revenue but just as something that needs to be done. Someone needs to step up and be able to provide all of this, and we have the eyes on us to be able to do that, so that’s the ultimate goal.”

“You never have to pay for training again, if you chose to.”

As the new product evolves it’s plain to see that these guys are passionate about what they are doing and the energy they are putting into this new iteration of their product is tremendous. The GNS3 brand itself has undergone a massive overhaul as they reimagine themselves not only as a training tool but as a legitimate choice for network modeling, development, proof-of-concept work, etc.

It’s clear that as GNS3 evolves from “that Cisco simulator” into a full-fledged multi-vendor, multi-purpose network virtualization platform that it seeks to become a core component of every network professional’s toolset.

Cisco UCS – Zero to Hero in 5 Short Years

Posted on June 4, 2014 by NetworkCanuck

I’d love to call Randy Seidl and ask him for an interview. The problem is, I don’t have the street cred that it would take to even make it past his administrative assistant. You see, Mr. Seidl used to work for Hewlett-Packard as their “Senior Vice President of the Americas, Enterprise Servers, Storage, and Networking”.

He doesn’t work for HP any more.

From YouTube/Cisco: “The Worst Predictions in History”

“A year from now the difference will be UCS is dead and we have had phenomenal market share growth in the networking space.”

This is a quote taken from this article over at CRN just prior to HP’s 2010 partner conference, just one year after Cisco launched the UCS platform. HP’s strategy at this point was to try to take market share away from Cisco in their core switching business. I suppose this was a natural response considering Cisco’s foray into enterprise servers was aimed to strike a blow at the heart of HP’s business. HP’s strategy aimed to empower their partners to offer significant discounts and trade-in allowances for any existing Cisco customers, hoping to woo them away from the teal giant. 2-for-1 and 3-for-1 deals weren’t uncommon, and it seemed HP was ready to cut off their nose to spite their face just to grab more of the Ethernet switching pie.

5 years later Cisco remains atop the Ethernet switch market with a 60.4% share.

But we’re not here to talk about Ethernet switching. We’re here to talk about Cisco UCS.

Happy Birthday Unified Computing System

The official Cisco Unified Computing System press release came in March of 2009. By 2010, which marked my first Cisco Live event in Las Vegas, UCS had a lot of hype among the Cisco faithful. I returned home from the conference excited about UCS, because at my day job we were in the process of jumping head-first into virtualization and were looking at different options for servers and storage.

I shared my enthusiasm for UCS but was told Cisco would never touch HP or IBM in market share for servers. We bought Dell.

In just 5 years, Cisco UCS has vaulted to the #1 spot in the Americas(40.9%), and #2 Worldwide (26.3%) for x86 Blade Servers, according to the IDC Worldwide Quarterly Server Tracker for 2014Q1.

HP has fallen from 47.7% to 34.9% in that time.

IBM has plummeted from 34.4% to 10.2% in that time.

Also according to the IDC report, Cisco has the highest industry growth in the total worldwide server market, with 39% revenue growth on a cumulative four quarter basis ending in 2014Q1. This, while HP, IBM, Dell, Oracle, and Fujitsu all report flat or declining results.

Cisco UCS also presently holds 94 performance benchmark records.

The Numbers Don’t Lie

5 years after launch, and 4 years after Mr. Seidl’s bold prediction that UCS would be dead in a year, the numbers reveal the truth of the success of the UCS platform. Cisco UCS has established itself as a key player in the enterprise server market, not only in the Americas, but Worldwide, and growth continues every quarter as the Cisco UCS product team continue to drive innovation and performance within the platform.

Cisco’s UCS business unit deserves a round of applause and a lot of credit.

The numbers don’t lie, but I’d still really like to ask Randy Seidl what he thinks of those numbers.

The official press release can be found here: http://newsroom.cisco.com/release/1426059

Cisco Live 2014 – San Fran-tastic!

Posted on June 2, 2014 by NetworkCanuck

My 5th Cisco Live is in the books and this was a fantastic week of reconnecting with friends, meeting new ones, and drinking from the technology fire hose.

This was my first visit to San Francisco and although I didn’t get to be a tourist very much, what I did see of the city was great. My hotel wasn’t in the greatest part of the downtown area, but even so the walk to and from Moscone never seemed too scary.

As I mentioned in my pre-show post my schedule this year was packed with sessions, meetings and events, much more than previous years. I arrived on Sunday and the whirl wind week of activity began immediately.

Registration and Arrival Tweetup

I managed to get to Moscone in time to register and pick up my badge and bag before the 5:00 pm closing on Sunday. After checking into my hotel and dropping off most of my gear, I went straight back to Moscone South for the Welcome Tweetup. This had grown exponentially over the last few years and the 2014 Tweetup was no exception. It’s always great to be face to face with the people you interact with all year on Twitter and other social media platforms, and this year was made that much more special with the #CiscoChampion program adding a number of new people to the mix.

Social Media Lounge 2014

The appearance of the SDNicorn marked the beginning of what was going to be a week filled with networking (in several terms), socializing and yes, even some shenanigans.

Sessions and Cisco Live Online

If you’ve ever attended Cisco Live or other events like it, you’ll know it’s near impossible to schedule in all the breakout sessions that you want to attend. There are simply too many of them. I’ve had a couple of years where I was focused on a particular technology, mostly because of a specific planned project at my day job, but when I can I prefer to have as much variety in my schedule as possible. Cisco has made this far less stressful because selecting one session over another doesn’t mean I have to miss the content for the one I didn’t attend in person, it simply means I can view it later on Cisco Live Online (formerly Cisco Live 365). This is a fantastic resource all year long, and I find myself going back to watch and re-watch content from the various Cisco Live events worldwide.

This year’s sessions included some Collaboration, UCS, Virtualization, and Nexus sessions. Also, thanks to Robert Novak (@gallifreyan) over at rsts11.com I was provided a complimentary 4-hour lab, and I chose to attend the Intelligent WAN (IWAN) Hands-On Lab. The IWAN lab was fun, given that I got to spend some time working with gear and software that I don’t normally get to play with, particularly UCS-Express (in the form of ISR-2911’s with UCSe blades), and Cisco Prime Infrastructure. A lot of this lab focused on Cisco Prime and seemed more of a DMVPN lab mixed with some WAAS and QoS, but it was still a great learning experience.

#CiscoChampions

Throughout the week there were some fantastic opportunities facilitated via the Cisco Champions group. These included an excellent (and revealing at times) team building event, a live Cisco Champions Radio episode hosted by Amy Lewis (@CommsNinja) in which a large number of us piled into the smallest room possible to record a really entertaining podcast, a tour of the Cisco NERV truck, and a briefing on the upcoming Cisco Modeling Labs (CML) product.

For me, these activities were more about connecting with a group of people who I’ve “met” through the Cisco Champions program via Google+, Twitter, and podcasts, but hadn’t yet met in person. I was absolutely blown away by the diversity of this group and the opportunity to meet face to face and share some conversation, some knowledge, and a quite a few good laughs as well.

#BaconIT

The fine folks over at @CiscoDC along with infamous bacon advocate Amy Lewis hosted an awesome party at the Cable Car City Pub on Monday evening. This was a fun social event that gave everyone an opportunity to wind down after the first full day of sessions and enjoy meeting some more of the social media personalities, Cisco Champions, and Cisco staff. The evening included some bacon-related giveaways, including subscriptions to the bacon of the month club. There was even a food table replete with….bacon.

@CommsNinja tossing bacon to the huddled masses

INE Rewired

INE once again hosted a customer appreciation event that included some excellent prizes (congrats to @bbaize on winning a Macbook Air!) and some more food and drinks. After mercilessly harassing Mark Snow (@highspeedsnow) and Brian McGahan (@brianmcgahan) I managed to procure one of the VIP tokens that provided access to an exclusive section of the Mezzanine Nightclub. They also used the evening to preview their upcoming revised and retooled training platform which promises to be a tremendous platform for studying and certification along your chosen Cisco track. The Rewired platform offers an interactive community approach to learning, including badges, achievements and looks to add a bit more “fun” to the task of studying for a Cisco exam. I’m really looking foward to the launch of this product, and attendees of the event were told they would have beta access sometime in the upcoming months.

Achievement Unlocked: VIP Coin

Mezzanine Nightclub

Customer Appreciation Event

The annual CAE is always a great time, and this year was no exception. We invaded AT&T Park, home of the San Francisco Giants, and were treated to some awesome food, a few beverages, and live performances from Lenny Kravitz and Imagine Dragons. I would have loved to see Kravitz as the headliner, but then again I’m old. He played a fantastic show, playing many of his hits from the 90’s and 00’s. Imagine Dragons’ set got off to a rocky start when the power went out to their amplifiers, but after a brief “Please Standby” everything got back underway. The rocked out the remainder of the evening and the evening’s finale was a brilliant fireworks display launched from a ship in the San Francisco Bay beside the park.


Myself and @pidooma	Party Tweeps
#BaconIT lady @CommsNinja	Birthday Hats
Lenny!	@Lauren and @ColdStorageGuy photobomb

Final Tweetup and Farewell

The Thursday of Cisco Live is always bittersweet as the final sessions of the week wrap up, the last of the prizes are given away at the World of Solutions, and everyone heads to the airport to scatter across the world back to their homes. This year was no exception.

The farewell Tweetup was held at the Social Media ~~Hub~~ Routed Bridge (no idea, ask @amyengineer) and everyone had an opportunity to take a few more photos, play a few more hands of Cards Against ~~Humanity~~ #CLUS, and say their goodbyes. As is tradition the group photo at the Cisco Live sign marked the end of the week, and I bid farewell to my fellow Twitterers, Champions, and friends, and headed to the airport.

Sayonara at The Sign

Cisco Live 2014 was for me, the best Cisco Live yet. They seem to get better every year and I’m not quite sure how that happens, but it’s true. From the moment I left San Francisco I started to look forward to next year, and Cisco Live 2015 in San Diego.

See you there!

All Aboard for SFO – Cisco Live 2014

Posted on May 16, 2014 by NetworkCanuck

Cisco Live 2014 is only two days away and although I like to think I’ve planned everything out well in advance, I’ll likely be doing a lot of last minute panicked preparation early Sunday before my flight.

This will be my fifth Cisco Live, my first one being the 2010 event in Las Vegas. Things have changed for me significantly since that first conference, and every year it just seems to get better and better. To be honest it’s become the one networking/professional development event I absolutely cannot miss every year, such that I pay for the majority of the trip myself.

Budget Scarcity

Working as I do in the public sector, where budgets seemingly shrink every year, while costs and expenses rise, I do not have the benefit of a large professional development budget. Much of the training and certification I have done over the years has been self-paced, self-funded, and conducted in my home office and lab. This includes the costs incurred for Cisco Live. What budget I do get every year (usually between $1000-$1500 CAD) I put towards the conference registration costs. This does help put a dent in even the early-bird conference price of $1995 USD, and I appreciate the fact that I get anything at all towards my continued education.

Travel can be expensive but I’ve managed to get pretty thrifty over the years, and more often than not find pretty good deals on hotels that aren’t part of the Cisco Live “official hotel list” but are in close proximity to the conference location and other amenities.

In 2011 for example, I was able to stay The Excalibur in Las Vegas which is literally across the street from the Luxor, Mandalay Bay, and MGM Grand, which are all official Cisco Live hotels, for about 1/3 the price. The tram running along the strip provided fast transport over to the Mandalay Bay for the conference events.

This year was an even bigger challenge with hotel prices in San Francisco bordering on insane. There was no possible way I was going to be able to attend this year if I had to book one of the official hotels which ranged from $170 to $399 per night.

My schedule is so packed every year I actually don’t spend a lot of time in my hotel, so it’s basically a place to sleep for a few hours before beginning another day of social networking, learning, and fun. With that in mind I booked a room at a hostel-style hotel called The Winsor that’s a decent walking distance away from The Moscone Center. I’ll have a small room with pretty much just a bed and a sink, and a shared bathroom. The reviews on various travel sites said the neighborhood wasn’t great but the hotel was decent, even if it looks a bit shady.

Some examples:

“Decent hotel, but absolutely scary street/neighborhood!”

“The hotel is of course very basic, but definitely acceptable (including in terms of cleanliness) given the excellent price.”

The proximity of liquor, beer, and cigarettes will be convenient, at least.

Evolution

The first year I attended Cisco Live in 2010 I was purely alone. I didn’t know anyone, I wasn’t very active on social media, and I had no idea what to expect from the conference. I focused on maximizing my time in breakout sessions with the intent of learning as much as possible in the week I was there. I lurked on Twitter and saw there seemed to be a community within a community here and quickly saw some of the same names interacting with each other, with mention of “tweetup” and “Tom’s Corner”.

In between 2010 and 2012 I spent a lot of time following some of the more active names from that first event. People like Tom Hollingsworth (@networkingnerd), Jeff Fry (@fryguy_pa), Amy Arnold (@amyengineer), Stephen Foskett (@sfoskett), Tony Mattke (@tonhe), Eric Peterson (@ucgod), and Jennifer Huber (@jenniferlucille) just to name a few.

I discovered the Packet Pushers podcast where a lot of these folks got together with Greg Ferro (@etherealmind) and Ethan Banks (@ecbanks) to discuss current topics and trends in networking, and started listening to the back catalog of recordings.

I learned about Tech Field Day and it’s delegate program and with it another list of names of people I started to follow and interact with on Twitter.

As I built both the group of people that I followed on Twitter as well as those who followed me, I began to feel a real sense of community within these groups and carried that forward into the next visits to Cisco Live.

#CiscoChampion

By 2013 in Orlando the trip to Cisco Live felt less like attending a trade show and more like a reunion with a large group of friends. “Tom’s Corner” had evolved into the Social Media Lounge, and it was clear that Cisco was putting a lot of effort into social media and this large subculture of networkers who populated it. The welcome Tweetup was the largest it had ever been and once again it was a welcome opportunity to reconnect with old friends and meet new people as the social media crowd grew even larger.

I had been approached by Cisco’s Social Media Marketing Manager just before the Orlando show as a candidate for the Cisco Champions program. I welcomed this as an opportunity to continue to grow with social media as a platform to discuss, inform and debate current topics within the networking world. I’ve been able to participate in some great podcast, product briefings and even managed to have a few blog articles published:

http://blogs.cisco.com/perspectives/los-angeles-unified-school-district-ipads-for-everyone/

http://blogs.cisco.com/perspectives/los-angeles-unified-school-district-hack-the-ipads/

http://blogs.cisco.com/perspectives/ioe-napkin-math-and-your-daily-commute/

Now in 2014, the #CiscoChampions group has grown quite a lot and there are several events scheduled for us in San Francisco. My schedule now is even more jam-packed between regular sessions, social media events, Cisco Champion events, Tech Field Day round-tables and somewhere in there, time to eat and sleep.

Other Events

The Customer Appreciation Event is going to be excellent this year with Lenny Kravitz as the headliner, and Imagine Dragons in support. The #BaconIT meetup with Amy Lewis (@commsninja) should prove to be another great evening with friends and bacon (and maybe a beverage or two). And I’m also looking forward to running in another 5K with Colin McNamara (@colinmcnamara) in support of the Wounded Warriors Project. Rumor has it we might be running across the Golden Gate Bridge!

I’m truly looking forward to another great year, and can’t wait to see you there!

Cisco Live 2014 CAE – A Canuck Connection!

Posted on March 12, 2014 by NetworkCanuck

Pretty much every year, the Cisco Live Team post a poll on the Cisco Live web site to potential registrants asking which band they would like to see for the Customer Appreciation Event (CAE). I’m not sure if the polls are a short list of potential performers or if they are just gauging interest in a particular genre of entertainment, but the options all seem to cover a fairly wide range of tastes.

Before attending my first Cisco Live I can remember watching a few YouTube videos of some earlier CAEs, one of which featured none other than KISS. My I was stunned and excited by the opportunity to be relatively up close and personal with a private show by one of the world’s greatest entertainment groups.

The poll one year included Lenny Kravitz, who I voted for, and the results seemed to be largely in favor of him being the CAE performer. Alas that year Kravitz was not ultimately the chosen act, but another great CAE was enjoyed by all in attendance nonetheless. (I had heard rumor that Kravitz had been lined up that year, but a scheduling conflict forced him to back out and another band was selected)

This year I am very excited about the announcement of the entertainment for the CAE. As a proud Canadian, there is a very special link with the artist and my country, even though he resides in the United States. I also know that many of my fellow Cisco Live attendees, bloggers and #CiscoChampions have all been begging for him to be the CAE headliner.

Ladies and Gentlemen, this years Customer Appreciation Event headliner: JUSTIN BIEBER!

I may have to turn comments *off* for this post.

Okay, it’s not The Biebs, although I know there are some of you who are disappointed. The actual CAE artist does have a Canadian connection however in that he recorded an excellent cover of The Guess Who’s “American Woman” in 2000. The Guess Who, not to be confused with England’s “The Who”, are from my hometown of Winnipeg, Manitoba, and I list them among Canada’s greatest accomplishments along with poutine, ~~Pamela Anderson~~ Dan Ackroyd, and The Canadarm.

This year in San Francisco for the 25th Anniversary, the CAE headliner will in fact be Lenny Kravitz at AT&T Park – Home of the San Francisco Giants!

Kravitz is a pure entertainer and an extremely talented artist. Not only as a musician but more recently as an actor. Kravitz is multi-talented singing lead and backup vocals and in many cases recording guitar, bass and drums for his own albums. As a hack ~~musician~~ drummer myself for many years, it’s easy to truly appreciate the dedication to one’s art that this diversity of skill requires.

As networkers and IT professionals, we are all equally dedicated to our own “art” and in the constantly changing world of data networking, we too must continue to develop additional skills and ‘learn new instruments’ as it were, in order to continue to be the best in our fields.

This year at Cisco Live and at the CAE, we are provided not only an opportunity to connect, reconnect, and learn from each other, but to sit back, relax, and enjoy some down time with our peers and colleagues, while enjoying what will likely be an evening of epic entertainment that could easily rival all previous CAEs.

Not to be overlooked is this years special guest – Imagine Dragons! Their 2012 debut release “Night Visions” has had phenomenal success and afforded the band many accolades in 2013, including “Breakthrough Band of 2013” by Billboard and their hit “Radioactive” named “the biggest rock hit of the year” by Rolling Stone.

If you haven’t already, head over to the Cisco Live US page and get registered. Trust me when I say it’s an event you do not want to miss!

This announcement totally makes up for my accomodations!

Note: This is not an official Cisco Live hotel, but I’m traveling on my own dime this year and the San Francisco hotels are pricey…

I look forward to seeing you there!

One Way Audio on Cisco 7925G Wireless Phones

Posted on October 4, 2013 by NetworkCanuck

Knock Knock.

Who’s there?

Voice over Wireless LAN.

Voice over Wireless LAN who?

….

Hello?

….

Hello?

Working with a multitude of different technologies is great. I love it, for the most part. That being said sometimes it can be really frustrating as well. I am neither an expert in voice nor wireless technologies, but I am often times the primary ‘go-to’ person for both of these subjects at work. Now I like working with voice, it’s fun and presents its own interesting challenges sometimes, but for the size of our VoIP deployment, it pretty much just works. Wireless, while still fun to play around with, tends to be my nemesis, as I just haven’t had enough time to really delve into its deeper mysteries. Now, on that rare occasion when the problem is related to both voice AND wireless, things start to get really interesting.

I recently deployed some Cisco 7925G Wireless IP Phones to a number of our sites’ custodians as a replacement for cellular phones. They need to be mobile around the facility in order to troubleshoot issues in places that don’t have a hard line, but don’t require a full-blown cell phone.

Now some caveats; we don’t have sufficient AP coverage for a full-blown VoWLAN deployment, and during testing with the 7925G I did notice some interruption in the call stream when roaming from AP to AP. We also no longer have Cisco as our wireless vendor so I thought there may be some interoperability issues, but felt that 802.11 was after all, a standard right? What could possibly go wrong?

First Reports

The first rumblings of a problem came from some of the custodians saying they had ‘intermittent’ audio. I assumed (somewhat incorrectly) that this meant they were trying to wander around the building or even outside, treating the phone as a cell-phone, and losing sufficient signal from a nearby AP to maintain the call.

I explained to anyone with issues that these were not in fact cellular phones and they needed to stay within reasonable range of an AP to keep their call going. We would add capacity to the wireless as needed in the future, but for now it was the best we could do.

Sent Back

Next I received one of the phones, and it’s charger, in inter-office mail with a sticky note saying simply: “doesn’t work”. I tested the phone with a few different numbers and it seemed fine. I sent it back to the person who mailed it with a note: “works fine”.

As it turns out, I was wrong.

Definitely Broken

I next heard from another analyst who said all calls from the phone at one site were completely dropping. No audio at all. We tested and found that audio coming from the 7925’s was fine, but they were having problems receiving audio. The initial call setup seemed fine and there were a few seconds of clear two-way audio, but almost immediately the receiving audio was failing.

One-way audio – the bane of any voice engineer’s existence. Coupled with the fact that these were wireless phones as well, made troubleshooting the issue even more complicated.

I had initially thought this might be a QoS issue but the wired phones at the site were fine. Wireshark confirmed QoS wasn’t an issue but I could clearly see in the captures that the RTP to the handsets stopped shortly after calls began, resulting in one-way audio.

Viewing the Call Statistics on the phone also confirmed there was definitely some sort of problem. Jitter was extremely high, Receiver lost Packets were many, and the MOS was around 2.

Settings

I began playing around with the WLAN settings on the 7925G handsets, trying to find what might be causing the issue. Some suggestions from folks on Twitter pointed at forcing the phones to use 2.4 GHz only, while others insisted they would work fine on 5 GHz. Hard setting the frequency didn’t appear to resolve anything, so I continued the ever popular troubleshooting technique of randomly turning options on and off.

I came across the setting labeled “Call Power Save Mode” which was set by default to “U-APSD/PS-Poll” and also presented the option “None”.

Now, I had no idea what this option did, but I set it to “None” and performed a test call. Lo and behold, the issue appeared to go away. Two way audio persisted through the entire call, and call statistics on the handset were dramatically improved. Jitter was down to 2/22, only 2 dropped packets, and MOS was up to 4.5.

U-APSD/PS-Poll

So what exactly does this option do? U-APSD or Unscheduled Asynchronous Power Save Delivery is a mechanism that allows frames to be queued on a wireless access point in order to save power on a wireless client. When there is no data for the client to receive, it can go back into standby mode, allowing it to save power and battery life.

From Cisco’s Voice over Wireless LAN Design Guide:

The primary benefit of U-APSD is that it allows the voice client to synchronize the transmission and reception of voice frames with the AP, thereby allowing the client to go into power-save mode between the transmission/reception of each voice frame tuple. The WLAN client frame transmission in the access categories supporting U-APSD triggers the AP to send any data frames queued for that WLAN client in that AC. A U-APSD client remains listening to the AP until it receives a frame from the AP with an end-of-service period (EOSP) bit set. This tells the client that it can now go back into its power-save mode. This triggering mechanism is considered a more efficient use of client power than the regular listening for beacons method, at a period controlled by the delivery traffic indication map (DTIM) interval, because the latency and jitter requirements of voice are such that a WVoIP client would either not be in power-save mode during a call, resulting in reduced talk times, or would use a short DTIM interval, resulting in reduced standby times. The use of U-APSD allows the use of long DTIM intervals to maximize standby time without sacrificing call quality. The U-APSD feature can be applied individually across access categories, allowing U-APSD can be applied to the voice ACs in the AP, but the other ACs still use the standard power save feature.

Best Intentions

So why did turning this feature off resolve the one-way audio problem? It seems this is a technology that should help rather than hinder a wireless VoIP call. In this case it appears to do nothing but cause problems.

I can only speculate here because my understanding of this particular mechanism is limited, but I would suspect that even though U-APSD is a standard as part of IEEE 802.11e, the implementations may be somewhat disparate across vendors. Cisco in this case makes the phone and the wireless network is Ruckus. I suspect if I were using Cisco wireless gear, this wouldn’t be an issue. That’s not to blame Ruckus for the problem of course, it just seems to be one of those minor differences in how vendors implement certain technologies.

This brings about an entirely different topic of discussion, but if this is the case, can anything be done to hold vendors accountable for the little tweaks and changes to technologies that are supposed to be standards designed to improve, not prevent interoperability?

The Problem With “Free”.

Posted on May 14, 2013 by NetworkCanuck

It’s rare to have a day go by during which I don’t hear or read about some product that a vendor is now ‘giving away’ or moving to a ‘freemium’ model. In some of the more contentious verticals in the IT industry this seems to be a key tactic for winning new customers and providing value-add for existing ones.

I’m not in marketing or sales, so I can only assume here that the premise behind these gratuitous offerings is to have new, potential customers try the product, fall in love with it, and want to then add more of that company’s products to their infrastructure. There is also a tiny voice in my head that suggests perhaps these organizations might also want their ‘free’ product to become so critical to your operation, that should they decide to charge a fee or licensing for said product at some point in the future, that you’d be forced to pay because it has become something you simply couldn’t live without.

Ultimately the short or long-term goal of offering these products doesn’t really matter. What matters is there is a very big problem with these free products:

They’re free.

They don’t generate revenue, at least directly, for the vendor providing them. This means they are, in all aspects, simply a cost center…a money sink. An expense that perhaps proves the old saying that “you have to spend money to make money”. But the real issue here for you or I as a potential user, or implementer of these products, is that it is very difficult to get any support.

Hello, Bonjour

This particular ~~rant~~ blog post is centered around one such product that everybody seems to be racing to give away. If you, like me, work in an environment that is moving to support the BYOD craze and have anything other than one large, flat network, then Apple’s Bonjour is probably driving you nuts and causing you to sprout gray hair, if you have any left.

Because this particular protocol and all of it’s relatives (mDNS, Zeroconf) can’t communicate across layer 3 boundaries (they have a TTL of 1) when someone on your BYOD wifi wants to talk to the Apple TV on your corporate wifi, you need something to broker that connection. Enter the Bonjour Gateway (BG).

Aerohive was first to announce and make available their BG product in early 2012. It is built into their HiveOS on any Aerohive access point, or as a virtual machine that will run on VMware. It’s free up to 2 instances of the virtual appliance. I don’t know what the cost might be for anyone wishing to use more than 2, but I would imagine this is an opportunity to sell actual Aerohive hardware to a potential customer.

Cisco has included it as part of their Wireless Lan Controller (WLC) software beginning with version 7.4. This isn’t free, per se, but is obviously a valuable addition for any existing customer.

Ruckus announced in January 2013 their SmartWay™ technology as “beyond bonjour bridging”, and would be available Q2. Again, this is only free in the sense that existing customers would not have to pay for the software upgrade to their existing controllers.

A quick Google search at some other vendor offerings show that pretty much everyone in the wireless space is offering support for Bonjour in some way.

I may be wrong about this but it seems to me that providing a solution for this issue in enterprise networks is/was a priority for each of these vendors. Why then has my experience with getting one of these platforms working been such a disaster?

Aerohive

If you don’t already follow Andrew von Nagy on Twitter (@revolutionwifi), you should. He is a true wifi evangelist and an excellent resource for keeping up-to-date on all things 802.11. His twitter feed was very active with the announcement of the release of Aerohive’s BG.

Working in a K-12 education environment we had already identified this as a need. Staff and students wanted to take advantage of AirPrint and AirPlay and we had to find a solution. I quickly signed up for my free Aerohive BG and HiveManager account. Installation was easy as it comes in the form of an OVA. It’s pretty much ‘drop it into VMware’ and you are ready to go.

I had some problems with devices being able to see the AirPrint and AirPlay services across subnets. After some tinkering I decided to email Aerohive at the provided “free_bonjour_support@aerohive.com” address with my issue. That email must have ended up in the bit bucket because I received no reply. I sent out a tweet about a week later asking @Aerohive how long one could expect to wait for support for the BG. That too was met with silence. Two weeks later I was rather frustrated and sent out another tweet, this one a little more vitriolic:

“Going nowhere fast with Aerohive’s free bonjour gateway. Anyone have alternative suggestions? (That work)”

Now it should be noted that I’m in Canada and this tweet was sent out on November 22nd, 2012 – US Thanksgiving.

Andrew von Nagy responded via twitter and helped me out with some troubleshooting. I have to throw out a big thanks to him for taking the time on a holiday to offer some support.

On that same day, I received a reply to my original email (unsure if Andrew had anything to do with this) and began working with the online support to get the BG working.

A short 10 weeks later, I had resolved the issue (on my own) and closed the support request with Aerohive. From the original email on November 5th to resolution on January 10th….granted there are a few holidays in there…but that’s a long time to get an issue with an initial configuration resolved.

Ruckus

Just around the same time (January 2013) I managed to get that first BG working, we received word from our current wireless vendor, Ruckus, that they too were working on a BG solution. This was direct from David Callisch, VP of Marketing for Ruckus Wireless. He even offered to let us beta test the new firmware. This is great news! Being able to implement this solution on infrastructure we already own and manage should be quick and easy, right?

It’s mid May, and we still haven’t received the beta firmware.

Also, Ruckus recently pulled their latest 9.6 firmware off their support site, so I have a feeling 9.7 and SmartWay™ are going to miss their targeted Q2 release.

“Ruckus Wireless has decided to remove the 9.6.0.0.264 release for ZoneDirector while we investigate an issue that was discovered after the release.”

Aerohive Revisited

In April I received an email from Aerohive that outlined some major bug fixes and enhancements to their free BG. While I had been able to get it working with AirPlay somewhat in my previous attempt we had never been able to get AirPrint to work properly. I hoped that this news would mean we could get both pieces to function properly.

Having deleted the VM for the original installation of Aerohive’s BG, attempted to reinstall it, only to be told that my serial # had already been activated and that I could not reactivate it. Ok, easy fix, right? I fired off an email to “free_bonjour_support@aerohive.com” and explained my situation and asked if I could have a new key or the original key re-enabled.

That email went out April 19th, and I have yet to get any sort of reply.

Free Should Not Mean “free from support”

If these value-added features, or in some cases, fully ‘free’ products are meant to drive potential customers to become paying customers and/or if these products are meant to keep existing customers as loyal, long-term customers with an existing vendor, then I would expect support be as agile and attentive as it would be for any other product or offering from these same vendors.

I shouldn’t be left waiting for an email that never comes, and I certainly shouldn’t have to resort to social media shaming to get action from a vendor. Sadly it seems to be the most effective method of getting things moving, but it should be a last resort not the primary method of seeking resolution.

Perhaps I’m expecting too much from a free product or feature, and I may be misinterpreting the purpose of these add-ons as marketing/sales tools. I might be naive in believing that any truly ‘free’ product is going to become a key part of my infrastructure and solve a major technical hurdle for my users. I can only hope there is actually some sort of benevolent, beneficial reason for vendors to offer these solutions, and hope that they are able to provide some better support in the future.

Otherwise, there are truly free and open products like Avahi that are able to quickly and easily deploy mDNS service discovery options across subnets. If you know a little Linux…

Note: During the writing of this post I had been contacted by our local Aerohive rep who caught wind of a Tweet I sent out yesterday about my BG issue. He’s managed to get me a new serial # for our BG so I can happily reinstall it and give it another go. Social media wins again!

Tony’s Discount Switch Emporium

Posted on August 15, 2012 by NetworkCanuck

Random Acts of (dis)Connectivity

Summer time is busy time. July and August, when all the staff and students are out enjoying their vacation, the IT department for a public school division is hard at work. It’s one of the rare opportunities I’m allowed to unplug, replace, upgrade, reboot, and/or generally break our infrastructure, because nobody is using it. So with a full plate of projects to complete before the end of August – when random things break “on their own” it sometimes causes some confusion.

Last week I had two wireless access points stop communicating to the controller at one of my sites. Now, had there been people in the building I could have initially chalked this up to someone playing around in the wiring closet (we have switches in the weirdest, least secure places) or some kids throwing basketballs at them (yes, we have AP’s in the gyms), but I knew for a fact that this building was empty. Our Facilities guys generally let us know if there is going to be some work done that is going to involve power going down, and this definitely wasn’t the case because the switch these 2 AP’s were on was up and running.

Strange Behaviour

I checked the switch and the ports that the AP’s were on showed up/up. I could even ping their IP addresses. What was strange was I could not ping the controller from that switch, nor could I ping the site gateway. While the switch appeared up and functional for all intents and purposes, it appeared as though it was simply refusing to forward any traffic. Then I checked the logs:

Aug 9 13:11:48: %ILET-1-AUTHENTICATION_FAIL: This Switch may not have been manufactured by Cisco or with Cisco’s authorization. This product may contain software that was copied in violation of Cisco’s license terms. If your use of this product is the cause of a support issue, Cisco may deny operation of the product, support under your warranty or under a Cisco technical support program such as Smartnet. Please contact Cisco’s Technical Assistance Center for more information.

Dammit! I knew that guy selling Cisco gear off the back of his unmarked white truck in the back alley was too good to be true! His prices were so reasonable, even compared to our educational discount. You know it’s so hard to work within an ever-decreasing hardware budget these days.

Alright in reality this batch of switches was bought from a legitimate Cisco Partner, one of Canada’s largest, and I hadn’t touched them since rolling them out a couple of summers ago. They had the out of the box IOS 12.2(22)SE that they had shipped with, and I had about 30 of them around. These were a batch of 2960S switches that were replacing some aging 2950 units that had calcified and grown long grey beards.

I started checking several others and quickly realized this was an isolated issue, limited to this one switch.

Licensing, Contracts, Smartnet, Oh My!

As part of our ongoing efforts to reduce expenses and save what little budget we have, we don’t get Smartnet on our access switches. Ultimately all of the current 2xxx and 3xxx switches (sometimes referred to as the DSBU or Desktop Switching Business Unit switches) come with Cisco’s Enhanced Limited Lifetime Warranty which includes free IOS software updates. The lifetime warranty is arguably Cisco’s response to pressure from HP’s long-standing lifetime warranty on their Procurve switches, and free IOS updates are always nice:

Software Update

Q. Can I obtain a “no additional cost” Cisco IOS^® Software update for the Cisco Catalyst 2960 Series?

A. Yes. Cisco offers ongoing Cisco IOS Software updates for certain fixed-configuration and stackable Cisco Catalyst switches at no additional cost. For the life of the product, updates within the Cisco IOS Software package purchased (LAN Lite and LAN Base) will be made available.

Note that upgrades are different from updates. For example, an upgrade from the IP Base package to IP Services package provides significant new function; therefore, this upgrade requires the purchase of a software license upgrade. Updates are incremental software features and bug fixes that are released within a licensed Cisco IOS Software package.

This statement supersedes any previous warranty or software statement and is subject to change without notice.

Q. How do I get a “no additional cost” Cisco IOS Software update for the Cisco Catalyst 2960 Series?

A. Visit http://www.cisco.com, click “Downloads,” and select “Switch Software.” Downloading software requires a Cisco.com username and password. If you do not have a Cisco.com username, you can obtain one by clicking “Register” at the top of any Cisco.com Webpage.

Since we also keep a couple of spare switches around we don’t need overnight or next-business-day hardware replacement, Smartnet is an expense we can live without. This sometimes causes issues when trying to RMA a product…

Someone Tell TAC

I found a few posts in the Cisco Support Community from folks who had run into this same issue. It seemed a general consensus that there was no real solution. It appears a small batch of 2xxx and 3xxx series switches were having this issue as far back as March of 2010. There had been a Bug ID for the issue but the workaround had been to update the IOS to 12.2(44)SE or later, with no other workaround. Since I was already past that update, I decided it would be easier to just call TAC.

Apparently nobody tells TAC about lifetime warranties or free software. It always seems to turn into an argument with the front line support agents because as soon as they see there is no support contract, they won’t open a case. It’s pretty binary in their world…no contract equals no support.

So how do you get support for a lifetime warranty product? Just tell them you want to RMA the device. Cisco is more than happy to simply drop ship a new switch than to waste time troubleshooting and that’s fine by me. I’m not a CCIE so I don’t get bumped up to a level 2 engineer right away, and I’d rather not waste time on the phone or via email with a level 1 engineer. It’s in the best interest of everyone involved to just send me a replacement.

Good Experience

I’d have to say all of my experiences with Cisco’s RMA process have been excellent. Even without 8×5 or NBD service, they are quick and painless to deal with. A box arrives with a pre-paid UPS waybill and I happily ship the dead product back to them in the same box the replacement came in. They even have a Canadian location I can ship to so I don’t have to fill out reams of Customs paperwork (I’m looking at you, Ruckus Wireless) or pay brokerage fees.

Thanks for reading. As always feedback or comments are welcome. I have to run, Tony is here with some deeply discounted UCS servers and Rolex watches I need to take a look at.