The following text is copyright 2005 by Network World, permission is hearby given for reproduction, as long as attribution is given and this notice is included.

Conclusions based on isolated data

By Scott Bradner

In mid July the Internet performance measurement company Keynote Systems produced an Internet phone service quality study. They concluded that there is a "need for considerable improvement in service." I do not know just what led them to that conclusion but I do caution anyone reading the reports of this study to conclude that voice over IP has no future.

I have no way of judging the quality of the actual survey since all that Keynote (http://www.keynote.com) has made available is a press release about the survey. (http://www.keynote.com/news_events/releases_2005/05july12.html) They want an unknown amount of money for the full study. Whatever the amount I'm sure that it's more than I will get for writing this column so I'll get by on the press release.

According to the press release the study seems to explore a reasonable number of the relevant variables including multiple VoIP and connectivity providers, call location, and time of day. Keynote used ten factors to evaluate the "end-user experience." They then reduced these factors into two magic numbers representing reliability and audio quality. That seems to me to be rather over-reduced - for example, lumping 2 am (when no one is using the local link) with midday (when the local loop is congested) quality does not produce the information I'd want to get.

I use VoIP and a lot of people I know do as well. Maybe it's just the environments that we work and live in but it is not my experience that VoIP has a "need for considerable improvement in service." In fact, almost all the time the quality of the VoIP call is perceivably better than my office ISDN phone. I have made calls where the quality sucks (to use a technical term) but that happens a few times a year -- far less frequently than the poor quality PSTN connections I keep getting to and from all sorts of locations.

But lets assume that Keynote is completely correct in its claim that VoIP too often does not "live up to the dial-tone reliability and crystal-clear communication quality" we have come to expect with POTS, at least over the environments where they ran their tests. Can we actually conclude anything about the potential of VoIP from that conclusion?

I don't think that this information, in isolation, is all that meaningful in the real world. Other factors overwhelm these perceptions of poor "end-user experience." One only has to imagine what the report would have been if some prior year Keynote had run exactly the same tests on 1990-era cell phones, which were unambiguously and almost universally crappy. Any reader looking only at the results of the testing would have concluded that cell phones had no future at all. But anyone coming to that conclusion would have to ignore three important factors. The most important factor is portability -- the phone goes where you go instead of you having to go to the phone. A second factor is that technology keeps getting better, and the third factor is that the per-call cost of cell phones would plummet due to competition to a point where its far cheaper than land-line phone calls.

Any conclusion that VoIP has a poor future based on the Keynote study would be to ignore the last two of these factors. Ignoring important factors is not a good way to come to useful conclusions.

disclaimer: Harvard has enough of a past to know that predicting the future is hard but has not expressed a formal opinion on VoIP futures.