Author:
  Daniel-Constantin Mierla
  Nov 17, 2010

Sep 22, 2010. Mid of testing period for a new major release Kamailio 3.1.0 passed. It was running fine on voipuser.org for couple of weeks. Siremis 2.0 is also in pre-release phase and deployed on the same site.

Started the day doing the usual morning stuff: check of important messages inside inbox and news over net. Next a quick look at voipuser.org server, no coredump, CPU usage low, then Siremis 2.0 to watch the evolution of monitored metrics over the night:

• first memory usage: steady (good, no memory leaks)
• uptime: no auto-restart over the night
• SIP request receiving load: … well, well, hmm, jumped 15-20 times … alert! alert!

No alert actually, quick check of syslog and could spot pike module alert messages:

Sep 22 11:33:14 boyce kamailio-3.1[15760]: ALERT: <script>:
    ALERT: pike block REGISTER from sip:test@194.0.210.251 (IP:195.2.117.197:5065)

Fired quickly ngrep:

ngrep -d any -qt -W byline port 5060 and host 195.2.117.197

Wow, lots of of SIP requests like next one:

U 2010/09/22 11:34:03.731717 195.2.117.197:5065 -> 194.0.210.251:5060
REGISTER sip:194.0.210.251 SIP/2.0.
Via: SIP/2.0/UDP 127.0.0.1:5065;branch=z9hG4bK-63221083;rport.
Content-Length: 0.
From: "test" <sip:test@194.0.210.251>.
Accept: application/sdp.
User-Agent: friendly-scanner.
To: "test" <sip:test@194.0.210.251>.
Contact: sip:123@1.1.1.1.
CSeq: 1 REGISTER.
Call-ID: 2381089838.
Max-Forwards: 70.
.

Right, User-Agent: friendly-scanner, I believe you, From Latvia with Love!!!.

Panic?!?! Not at all, I was testing Kamailio, what better chance could I have?!?! Quick chat with Dean and TJ, grab my breakfast and start watching. Also, took some screenshots from Siremis 2.0 showed in the next section. After 18 hours, the attacker gave up, very likely he/she moved to another target.

The chart that signaled unusual amount of SIP traffic is a custom chart I built to show the number of SIP requests received during past 10 minutes:

You can spot that at 09:30, suddenly, number of received requests increased dramatically, more than 15 times.

The shared memory usage was still normal, not really affected:

There is a jump of max shared memory usage at 08:30, but it is only 10kB (10 000 bytes) - maybe few more phone got online.

The uptime screenshot I took next morning showed that Kamailio was running fine since Sep 21:

A proof that the attacker didn't go through is that the evolution of 200 transactions (chart counting forwarded requests replied with 200 OK, i.e., answered calls) didn't changed its normal limits for that time:

Being a public service, open world wide to users that subscribe themselves, VoIPUser.org Kamailio configuration used pike module to protect against flooding attacks. The scanner hit it.

Pike is a module that tracks number of received SIP requests by source IP and throws an alert when this number exceeds a threshold in a given period of time.

The pike filtering is done at the top of configuration file, so once the first alert fired, there was no more hit on database or SIP transaction layer. Kamailio doesn't not even reply for such cases, practically just discards such SIP requests.

Building a SIP reply and sending back expose to further risks, think about:

• a negative reply is given to end a potential SIP conversation (being it call, registration, a.s.o.), this is clear not the case
• replying consumes CPU cycles, memory and network bandwidth - regardless that stateless replying with Kamailio is very fast, still some resources are used
• even not really relevant in most of the cases, still you give some indication to attacker

Note that sometime attackers may stop after receiving “200 OK” reply for their requests - this is easy to do with sl_send_reply(“200”, “OK”) in configuration file. If bandwidth is your concern, you can try this trick for a while and if does not work, remove sending the reply from configuration file. Read a blog post for such situation at:

• http://jcs.org/notaweblog/2010/04/11/properly_stopping_a_sip_flood/

The pike module tracks number of SIP messages per source IP address. The readme of pike module is available at:

• http://kamailio.org/docs/modules/stable/modules_k/pike.html

Using only this module to detect scanning attacks in Kamailio configuration file can be achieve with following snippets:

loadmodule "pike.so"
 
...
 
# ----- pike params -----
modparam("pike", "sampling_time_unit", 2)
modparam("pike", "reqs_density_per_unit", 20)
modparam("pike", "remove_latency", 4)
 
...
 
route {
  if (!pike_check_req()) {
    xlog("L_ALERT","ALERT: pike block $rm from $fu (IP:$si:$sp)\n");
    exit;
  }
  ...
}

If you have a trusted set of peers, be sure you don't pike them on. You can check using src_ip test or via a module such as permissions. For example:

  if(src_ip!=TRUSTEDIP)
  {
     if (!pike_check_req()) {
         xlog("L_ALERT","ALERT: pike block $rm from $fu (IP:$si:$sp)\n");
         exit;
     }
  }

You can add htable module with a special hash table that can store the list of banned IPs and forbid traffic from it for a period of time. Here is an example blocking the IP 5 minutes (autoexpires value in seconds for htable definition):

loadmodule "htable.so"
...
modparam("htable", "htable", "ipban=>size=8;autoexpire=300;")
 
...
 
route {
  if($sht(ipban=>$si)!=$null)
  {
     # ip is already blocked - keep the node warm
     pike_check_req();
     xdbg("request from blocked IP - $rm from $fu (IP:$si:$sp)\n");
     exit;
  }
  if (!pike_check_req()) {
     $sht(ipban=>$si) = 1;
     xlog("L_ALERT","ALERT: pike block $rm from $fu (IP:$si:$sp)\n");
     exit;
  }
...
}

So, even if the attacker lowers the rate, it is still banned for 5 minutes. This approach has the benefit of printing the PIKE alert every 5 minutes, being easier to sport in syslog file the IP addresses that persist in flooding. By configuration, htable module will delete the entry automatically after 300sec.

Also, you can print the list of banned IP addresses using Siremis (via MI Commands panel) or kamctl:

kamctl fifo sht_dump ipban

After the VoIPUser.org experience, I added to Kamailio default configuration the logic to detect such attacks, being very easy to enable with #!define WITH_ANTIFLOOD (read comments in the top of default config coming with version 3.1.0).

An attacker might be patient and not sending high amount of SIP requests in short time (very unlikely) not to hit your PIKE threshold. You cannot lower it much because you will ban good users doing NAT keepalives or presence subscriptions. Another situation is distributed attacks, so the malicious SIP traffic is not originated from same IP - PIKE is limited in such case as it tracks based on source IP.

Such cases are harder to detect by rate of received SIP requests. But considering that the scope of a scanning attack is to discover the password of local users in order to abuse of their accounts, you can disable user authentication for a period of time if that user fails to authenticate several times in a row.

Because an attacker may use a valid local user ID, that user might be blocked for a while, but it is still better than losing lot of money or facing other damages, which can even affect the valid user.

The solution is presented in htable module readme:

• http://kamailio.org/docs/modules/stable/modules_k/htable.html

The idea is that if a user fails to authenticate 3 times in a row for SIP registration, it is banned for 15 minutes. You can reuse the same concept when authenticating the other SIP requests.

Htable module supports dynamic keys. For each user, two items are stored in hash table:

• number of failed authentications - in $sht(a⇒$au::auth_count)
• timestamp of last authentication attempt - in $sht(a⇒$au::last_auth)

Note: $au is a config variable that expands at runtime to authentication username.

Configuration file snippets are pasted next:

...
loadmodule "htable.so"
...
modparam("htable", "htable", "a=>size=8;autoexpire=920;")
...
route {
...
  if(is_present_hf("Authorization"))
  {
    if($sht(a=>$au::auth_count)==3)
    {
	$var(exp) = $Ts - 900;
        if($sht(a=>$au::last_auth) > $var(exp))
        {
            sl_send_reply("403", "Try later");
            exit;
        } else {
            $sht(a=>$au::auth_count) = 0;
        }
    }
    if(!www_authenticate("$td", "subscriber"))
    {
        switch ($retcode) {
            case -1:
                sl_send_reply("403", "Forbidden");
            exit;
            case -2:
                if($sht(a=>$au::auth_count) == null)
                    $sht(a=>$au::auth_count) = 0;
                $sht(a=>$au::auth_count) = $sht(a=>$au::auth_count) + 1;
                if($sht(a=>$au::auth_count) == 3)
                    xlog("auth failed 3rd time - src ip: $si\n");
                $sht(a=>$au::last_auth) = $Ts;
            break;
        }
        www_challenge("$td"/*realm*/,"0"/*qop*/);
        exit;
    }
    $sht(a=>$au::auth_count) = 0;
  } else {
    www_challenge("$td","0");
    exit;
  }
...
}
...

You can adjust the duration of banning by changing the value 900 (seconds) in line:

$var(exp) = $Ts - 900;

Fail2ban can scan syslog files for specific messages based on regular expressions and act upon matching by banning IP addresses.

Therefore you can print such message to syslog using xlog(). Fail2ban will match it and ban the traffic coming from the IP address you mention in the message.

Create /etc/fail2ban/filter.d/kamailio.conf with following content:

[Definition]
# filter for kamailio messages
failregex = Blocking traffic from <HOST>

Edit /etc/fail2ban/jail.conf and add:

findtime  = 600

[kamailio-iptables]
enabled  = true
filter   = kamailio
action   = iptables-allports[name=KAMAILIO, protocol=all]
logpath  = /var/log/kamailio.log # update it with your kamailio log path
maxretry = 10
bantime  = 1800

In Kamailio configuration, use next line whenever you want to ban an IP for half an hour:

xlog("Blocking traffic from $si\n");

Note: $si is a config file variable that expands at runtime to source IP address. In the syslog you will get messages like:

... Blocking traffic from 1.2.3.4

For example, plugging it in the above Kamailio snippets:

...
	$var(exp) = $Ts - 900;
        if($sht(a=>$au::last_auth) > $var(exp))
        {
            sl_send_reply("403", "Try later");
            xlog("Blocking traffic from $si\n");
            exit;
        } else {
            $sht(a=>$au::auth_count) = 0;
        }
...

Now, with this logic, if a user fails to authenticate 3 times in a row during 15 minutes, then the IP address of last registration attempt is blocked in firewall for half an hour by fail2ban.

You can do something similar for pike alerts.

• there is no doubt scanning attacks against VoIP increased the rate lately, I started to write this document as today another scanning attack happened on VoIPUser.org, this time from 184.106.166.10
• the IP addresses used to originate the traffic might be just other kind of victims (compromised servers), not really owned by attacker
• Kamailio 3.1.0 default configuration file includes protection against scanning attacks which can be enabled very easily with #!define WITH_ANTIFLOOD
• xlog, pike and htable modules are available in older versions, you can apply same solutions even if you run Kamailio 3.0.x or 1.5.x
• for me, a scanning attack is good from time to time, especially for testing. But be careful with them, the best is you block them immediately
• a secure VoIP server configuration is mainly a matter of your VoIP platform administrator. Be sure you have strong passwords for your subscribers and you do detection of DoS and DDoS attacks as early as possible during the SIP traffic processing

Kamailio 3.1.0 is the most secure version from our project entire history (same applies to SIP Express Router (SER) 3.1.0, the two being practically same application since v3.0.0). Here are some facts:

• it has support for asynchronous TLS - deploying SIP secure communication at large scale is now possible even on less powerful systems and easier than ever. First and only SIP server at this time with such feature
• current one-time-nonce algorithm is far more faster and secure than what Kamailio used in 1.x series
• support for better quality of protection (qop=auth-int) was added - no other SIP routing server has it now
• using config file asynchronous message queues (mqueue module) together with embedded languages such as Lua, you can trigger complex notifications without affecting SIP routing. For example: emails, sms, twitter, a.s.o. You can even initiate a call (click-to-dial fashion) or send other SIP requests (instant message) from configuration file to your netops group, using dialog or uac modules
• using geoip module you can detect the origin of SIP traffic by country, area, a.s.o., therefore is easy to block traffic from well-known countries for originating malicious traffic. In case you have customers only in special countries, you can allow traffic only from that countries

Daniel is co-founder and core developer of Kamailio SIP Server Project (former OpenSER). He has over 9 years of continuous work in SIP and VoIP. As a leader of Kamailio and open source advocate, he is a publisher of many online tutorials about SIP routing and a presenter at VoIP events world wide talking about scalability and security in VoIP networks. Nowadays, Daniel's activity is tied to Asipto (http://www.asipto.com), a German based company located in Berlin, where he coordinates SIP consultancy and training services.

You can contact the author via:

• Web: http://www.asipto.com/index.php/contact-us/
• Linkedin: http://www.linkedin.com/in/miconda
• Twitter: http://twitter.com/miconda