Knock Knock.

Who’s there?

Voice over Wireless LAN.

Voice over Wireless LAN who?

….

….

Hello?

….

Hello?

Working with a multitude of different technologies is great. I love it, for the most part. That being said sometimes it can be really frustrating as well. I am neither an expert in voice nor wireless technologies, but I am often times the primary ‘go-to’ person for both of these subjects at work. Now I like working with voice, it’s fun and presents its own interesting challenges sometimes, but for the size of our VoIP deployment, it pretty much just works. Wireless, while still fun to play around with, tends to be my nemesis, as I just haven’t had enough time to really delve into its deeper mysteries. Now, on that rare occasion when the problem is related to both voice AND wireless, things start to get really interesting.

I recently deployed some Cisco 7925G Wireless IP Phones to a number of our sites’ custodians as a replacement for cellular phones. They need to be mobile around the facility in order to troubleshoot issues in places that don’t have a hard line, but don’t require a full-blown cell phone.

Now some caveats; we don’t have sufficient AP coverage for a full-blown VoWLAN deployment, and during testing with the 7925G I did notice some interruption in the call stream when roaming from AP to AP. We also no longer have Cisco as our wireless vendor so I thought there may be some interoperability issues, but felt that 802.11 was after all, a standard right? What could possibly go wrong?

First Reports

The first rumblings of a problem came from some of the custodians saying they had ‘intermittent’ audio. I assumed (somewhat incorrectly) that this meant they were trying to wander around the building or even outside, treating the phone as a cell-phone, and losing sufficient signal from a nearby AP to maintain the call.

I explained to anyone with issues that these were not in fact cellular phones and they needed to stay within reasonable range of an AP to keep their call going. We would add capacity to the wireless as needed in the future, but for now it was the best we could do.

Sent Back

Next I received one of the phones, and it’s charger, in inter-office mail with a sticky note saying simply: “doesn’t work”. I tested the phone with a few different numbers and it seemed fine. I sent it back to the person who mailed it with a note: “works fine”.

As it turns out, I was wrong.

Definitely Broken

I next heard from another analyst who said all calls from the phone at one site were completely dropping. No audio at all. We tested and found that audio coming from the 7925’s was fine, but they were having problems receiving audio.  The initial call setup seemed fine and there were a few seconds of clear two-way audio, but almost immediately the receiving audio was failing.

One-way audio – the bane of any voice engineer’s existence. Coupled with the fact that these were wireless phones as well, made troubleshooting the issue even more complicated.

I had initially thought this might be a QoS issue but the wired phones at the site were fine. Wireshark confirmed QoS wasn’t an issue but I could clearly see in the captures that the RTP to the handsets stopped shortly after calls began, resulting in one-way audio.

Viewing the Call Statistics on the phone also confirmed there was definitely some sort of problem. Jitter was extremely high, Receiver lost Packets were many, and the MOS was around 2.

Settings

I began playing around with the WLAN settings on the 7925G handsets, trying to find what might be causing the issue. Some suggestions from folks on Twitter pointed at forcing the phones to use 2.4 GHz only, while others insisted they would work fine on 5 GHz. Hard setting the frequency didn’t appear to resolve anything, so I continued the ever popular troubleshooting technique of randomly turning options on and off.

I came across the setting labeled “Call Power Save Mode” which was set by default to “U-APSD/PS-Poll” and also presented the option “None”.

Now, I had no idea what this option did, but I set it to “None” and performed a test call. Lo and behold, the issue appeared to go away. Two way audio persisted through the entire call, and call statistics on the handset were dramatically improved. Jitter was down to 2/22, only 2 dropped packets, and MOS was up to 4.5.

U-APSD/PS-Poll

So what exactly does this option do? U-APSD or Unscheduled Asynchronous Power Save Delivery is a mechanism that allows frames to be queued on a wireless access point in order to save power on a wireless client. When there is no data for the client to receive, it can go back into standby mode, allowing it to save power and battery life.

From Cisco’s Voice over Wireless LAN Design Guide:

The primary benefit of U-APSD is that it allows the voice client to synchronize the transmission and reception of voice frames with the AP, thereby allowing the client to go into power-save mode between the transmission/reception of each voice frame tuple. The WLAN client frame transmission in the access categories supporting U-APSD triggers the AP to send any data frames queued for that WLAN client in that AC. A U-APSD client remains listening to the AP until it receives a frame from the AP with an end-of-service period (EOSP) bit set. This tells the client that it can now go back into its power-save mode. This triggering mechanism is considered a more efficient use of client power than the regular listening for beacons method, at a period controlled by the delivery traffic indication map (DTIM) interval, because the latency and jitter requirements of voice are such that a WVoIP client would either not be in power-save mode during a call, resulting in reduced talk times, or would use a short DTIM interval, resulting in reduced standby times. The use of U-APSD allows the use of long DTIM intervals to maximize standby time without sacrificing call quality. The U-APSD feature can be applied individually across access categories, allowing U-APSD can be applied to the voice ACs in the AP, but the other ACs still use the standard power save feature.

Best Intentions

So why did turning this feature off resolve the one-way audio problem? It seems this is a technology that should help rather than hinder a wireless VoIP call. In this case it appears to do nothing but cause problems.

I can only speculate here because my understanding of this particular mechanism is limited, but I would suspect that even though U-APSD is a standard as part of IEEE 802.11e, the implementations may be somewhat disparate across vendors. Cisco in this case makes the phone and the wireless network is Ruckus. I suspect if I were using Cisco wireless gear, this wouldn’t be an issue. That’s not to blame Ruckus for the problem of course, it just seems to be one of those minor differences in how vendors implement certain technologies.

This brings about an entirely different topic of discussion, but if this is the case, can anything be done to hold vendors accountable for the little tweaks and changes to technologies that are supposed to be standards designed to improve, not prevent interoperability?