Posts Tagged testing

tcpkali, TCP load generator

tcpkali is a lightweight and  easy-to-use tool that allows you to generate a traffic load with multiple TCP sessions. You push the load in one or both directions at the same time. Also the tool works easily over a NAT’ed connection. This tool is great if you need to test QoS for VoIP applications.

Here’s an example of a bidirectional load test:

# listening machine: listen on tcp port 8000, send traffic, and use 4 threads.
# the program will exit in 1 hour.
tcpkali -l 8000  --listen-mode=active -m X -T 1h -w 4

# connecting machine: send traffic using 4 threads and 10 simultaneous sessions
# for 1 minute
tcpkali -m Y -c 10 -T1m -w 4

The above test between directly connected PC Engines APU2 boards has shown 1Gbps of traffic, and the average CPU load was about 50%.

Also here are the packaging instructions for Debian, and a 64-bit binary package for Debian 8.

, , , ,

Leave a comment

Quality Assurance for VoIP calls: integration scripts

The scripts for integrating FreeSWITCH with Sevana AQuA software are now available at github:

More details on what they are doing are available in this older post:

, , , , ,

Leave a comment

Quality Assurance for VoIP calls

UPD: the FreeSWITCH integration scripts are available at

A customer has requested to set up a QA service that would continuously monitor the voice quality in their telephony infrastructure. They use a number of telephony carriers, and a set of applications on top of Plivo and FreeSWITCH. Also the conference module in FreeSWITCH is actively used.

Measuring jitter and packet loss, like it’s done in VoIPmonitor, is not sufficient, as we need to monitor end-to-end performance, including that of the FreeSWITCH server itself. So, there has to be a software component that compares the source audio with the recording on the other end of a call.

There are currently two major player on the market for voice quality measurements:

  1. ITU-T PESQ algorithm is proposed as an ITU recommendation P.862. Its source code is available at the ITU website and on Github. But the algorithm is patented, and the source code license does not allow any production use. The evaluation went quite smoothly, and the algorithm was able to detect even minor distortions, like one 20ms frame loss in a 2-minute call. The PESQ algorithm is designed and calibrated to be used for audio files of 6 to 20 seconds in length. Processing of a 2-minute recording takes approximately 5 seconds on a modern Xeon CPU. Commercial software is provided by OPTICOM and PsyTechnics.
  2. is an Estonian company that provides their own algorithms and software product for voice quality assessment. Their AQuA (Audio Quality Analyzer) software provides a fast and reliable way to compare the audio files: processing of a 2-minutes recording took about half a second on a modern CPU. Sevana has kindly provided a 10-days evaluation license and a fully functional software package, and the customer decided to go ahead with purchasing the license.

The simplest single-server license for Sevana AQuA allows running only one AQuA process at a time, so we wrapped its execution into a Perl script that utilizes a simple exclusive locking mechanism and performs audio file processing one at a time.

AQuA produces two scores in each measurement: the similarity percentage, and the MOS score. Both metrics are useful for quality analysis (for example, a 20ms frame added or lost inside of a silent pause influences the similarity score more significantly than MOS). It also takes a number of command-line options which can increase its tolerance to certain types of distortions, such as frequencies outside of G.711 range.

FreeSWITCH software is used as the SIP server for sending and terminating voice calls and for recording the received audio. It allows recording in several different formats: a) raw codec recording, done in the same thread as RTP processing; b) 16-bit signed PCM in WAV format, and file writing is done in a separate thread; c) compressed voice in a number of formats. The first two options produce similar results (raw codec recording had difficulties in the beginning). In case of raw codec recording, an additional step is required to convert the input files into 16-bit PCM WAV.

The call recording server requires to have a precise clock reference, so a baremetal hardware is required. Virtualized environments add up some uncontrollable imprecision to the virtual machines, although a thorough lab test is requires to verify this. It also depends on the type of hypervisor, as they implement the system clock differently.

The Linux kernel provides access to various clock sources. TSC is commonly used as default, and there is also HPET clock on modern hardware platforms. HPET is supposed to provide a more precise clock source, but it appears that it depends on CPU load: we accidentally discovered that audio recording in FreeSWITCH is significantly distorted when there’s some CPU activity is done in parallel (Debian package builder was working on the same 8-core machine). So far, TSC clock on a baremetal server provided the most reliable results.

The recording is done into a tmpfs mounted partition, in order to avoid any dependency on I/O load. The processing script performs the quality assessment on recorded files, and then moves or deletes them, depending on the measured score.

The SIP service was attached to an unusual UDP port, as port 5060 is frequently accessed by port scanners in public Internet. The DNS NAPTR and SRV records are used in order to use a universal SIP URI string, without having to reconfigure the remote servers if the IP address or UDP port changes.

Jitter buffer is disabled by default in FreeSWITCH, and it has to be activated whenever the calls are terminated on the server. In our case, the “jitterbuffer_msec” variable is set to “50:50” in the dialplan before answering and recording the call. With this, the jitter buffer is not allowed to grow dynamically above 50ms. So, we tolerate most of typical Internet-imposed jitter, but clock drift on the sending side would cause packet drop on the receiver.

The dialplan is designed to accept direct SIP calls from remote servers, and PSTN calls from telephony providers. If a remote server calls our QA service directly, it encodes the source name in the user part of the SIP URI. Also there are two options for a QA call: it can playback the test audio, or send silence. In case of PSTN calls, the caller ID is used as the source identifier. The dialplan activates audio recording into a WAV file on a tmpfs partition, and launches the processing script after the hangup.

The conference dialer is used for testing the conferencing performance on a production FreeSWITCH server. It requires a conferencing profile that does not play any greetings to conference participants. Also in case of more than two participants, only one has to be chosen as a speaker, and all others would be listeners. A dedicated SIP URI on the QA server is reserved to playback the test audio and not to perform any recording.

Each measurement result for QA calls is stored in an SQL database for further processing, and also sent to Syslog for real-time monitoring.

The test audio is a concatenation of speech samples from ITU-T Recommendation P.50 Appendix I, resampled from 16KHz to 8KHz and stored as 16-bit signed PCM audio.

, , , , ,


FreeSWITCH performance on Intel Atom CPU

There are multiple low-power, fanless  appliances on the market, and most of them are powered by Intel Atom processors. I needed an estimation how well an Atom would perform for a FreeSWITCH PBX application.

In this test, I use two Acer Aspire One notebooks with different processors:

  • atom01: Atom N2600 (2 cores, 4 virtual CPUs, 512KB cache and 600MHz per virtual CPU, 12768.02 BogoMIPS)
  • atom02: Atom N570 (2 cores, 4 virtual CPUs, 512KB cache and 1000MHz per virtual CPU, 13302.08 BogoMIPS)

Both notebooks are running 32-bit Debian 7 Wheezy (Kernel version 3.2.0-4-686-pae), and FreeSWITCH version 1.2.13 from pre-built Debian packages.

Test results summary

All calls in this test used transcoding between G.711alaw and G.722. The bottleneck in performance was always at the N2600 (atom01), because of slower CPU. In general, N570 can handle approximately 30% higher load than N2600.

With 10 concurrent calls (21 channels on atom01 and 20 channels on atom02), there is no voice distortion and new call processing does not disturb the ongoing calls. Each virtual CPU is busy at 20-25%

With 20 concurrent calls (41 channels on atom01 and 40 cannels on atom02), there is some minor voice distortion, especially during incoming calls, but quality s still acceptable.

With 27 concurrent calls (55 and 54 channels), voice distortions were too high and not acceptable. Every virtual CPU on atom01 was busy at around 50%, which means full load for the whole CPU.

With 20 concurrent calls without transcoding (PCMA only in all call legs), each CPU core on atom01 was utilized at around 9-10%. So, theoretically the platform can handle up to 40-50 simultaneous calls in non-transcoding mode.

Only the voice quality was tested. CPS was not tested, and it depends heavily on the complexity of the dialplan. But the overall response of the system was quite acceptable.

Read the rest of this entry »

, , , , , ,

1 Comment

VoIP lab design

I’m designing a network layout for IP telephony testing lab. Codename: xlab1 (xlab0 is a Xen machine at my home, and xlab1 is a Xen server hosted at a partner company).

The server has only 4 public IP addresses (one for management, two for redundant SIP front-ends, and one for web front-end), and the goal is to have the flexibility to test any telephony scenario, from a standalone vPBX to a 2600hz’s Kazoo installation.

Physically it’s a 1U server with Intel Core2Quad and 8GB RAM, running Xen hypervisor.

The primary application is a multi-tenant virtual PBX. All Internet communication is done with the two front-end servers, and all the telephony handling, media termination and applications are done by the back-end servers. NAT is only used for software installation on the back-end virtual machines.

On the front-end servers, Kamailio+RTPproxy will accept registrations from UAC’s in the public Internet, and also perform flood protection and topology hiding. They would forward all SIP messages to the back-end servers, based on a DNS-based domain map based on a local dispatcher database. The SIP authentication will be handled by back-end servers.

Also the front-end servers will run a FreeSWITCH process each, used only as an SBC for communication with ITSP trunks.

Also need to think how load-balancing and failover would be organized. Especially that many ITSP’s expect only one IP address at the customer end of the trunk.

Here is the concept drawing of the service components and communication flows:

The project is intended to be open-source, open-design, and open-documentation, so more technical details will follow.

UPD: the front-end kamailio configuration, work in progress

, ,

Leave a comment

Call-home SSH scripts

Sometimes I need to set up quickly some presence in a customer network: to be able to access it remotely, or to run some network management scripts, and so on. Most of such networks are behind NAT or at least a firewall, and incoming connections from outside aren’t always easy.

But outgoing connections from a customer LAN are mostly not a problem at all. So, I bring my own small netbook and place it in the customer LAN. This netbook automatically makes an outgoing SSH connection to my central server (a VPS) and pulls an SSH tunnel so that I can access the netbook from outside.

This is a kind of a backdoor, and it makes sense to make your customer completely aware of what you’re doing.

For such purposes, I have a couple of cheapest 10″ netbooks. I’m using Acer AspireOne and some older eMachines netbooks because they were sold cheaply. Most other netbooks would fit too, but one should be careful about Linux compatibility (especially the video drivers might be a problem). They come with 1GB RAM and 160 or 250 GB hard drives. It’s quite trivial to upgrade them to 2GB RAM, although it’s not really necessary. You must only be careful about buying a new SODIMM with exactly the same clocking as the original one.

The netbooks run standard Ubuntu Linux, with SSH daemon enabled. If the user home directory is encrypted, you won’t be able to login with your public SSH keys, so better not encrypt it.

Read the rest of this entry »

, , , , ,

1 Comment

Distributed Testing Platform: design concept

Author: Stanislav Sinyagin
Document status: concept draft

UPD: the project name is now Mooxu


In many network environments, especially in those of large ISPs or carriers, there’s a need to periodically execute some network tests. For example, an IPTV transport provider would need to make sure that all important multicast streams are available in every part of its edge network. Or a customer support engineer would need to collect byte and packet counters from a particular network port every 5 seconds.

The new software system (Project name: Mooxu) is designed to provide an open-source framework that enables the network operators to build the testing environment for their needs.  Also a number of open-source testing probe modules will be available.

Read the rest of this entry »

, , , , , ,