Tag Archives: Kamailio

Profiling Kamailio Module Memory Usage

Kamailio, Notes, Python, VoIP, , ,

Recently we were debugging a slow memory leak in a little-used Kamailio module.

For a long time we’ve used the Kamailio Prometheus exporter to export stats from Kamailio and then Grafana to visualize the metrics.

And one of the things we track per box is the shared memory (shm) usage inside Kamailio:

(kamailio_shmem_real_used_size / kamailio_shmem_total_size) * 100

Our dashboards showed this slow memory leak, with the memory slowly ticking up on a certain group of machines, we knew there was a memory leak, but in which Kamailio module?

Shared memory usage per Kamailio instance

So we know there’s a memory leak on some boxes (different boxes run different modules) but which ones?

kamcmd will show you a breakdown of the shared memory usage per module, so I started with a little cronjob to dump the data every so often, with the idea I would diff the values and look for big increases. 

*/5 * * * * kamcmd mod.stats all shm > /home/nickj/memory_stats_$(date +\%Y_\%m_\%d_\%H_\%M).log

Then the idea of processing the data from the files started to scare me.

So in the end I went overboard and wrote a little script to get the memory usage of each module and export it via Prometheus, so I can track down what module is at fault.

If I were a better C programmer I’d have added this into the main Kamailio Prometheus module, but I’m terrible at C.

I’ve put the code on Github here.

With it I was able to setup this:

Yeap, that’s our problem, here’s the deltas, that module has grown by ~10Mb in 24 hours.

Now to fix the problem…

If anyone else finds this useful I’ve also posted the source for the Grafana Dashboard here.

Kamailio Bytes: KEMI and UAC Module – Event Route

IMS / VoLTE, Kamailio, Mobile Networks, Notes, Python, VoIP, , , ,

The UAC module is super handy for creating and sending SIP requests from Kamailio. It could be triggered via HTTP requests using xHTTP, other SIP messages or on a scheduled basis like with Rtimer.

More and more I’ve been using KEMI to allow me to write Python based Kamailio dialplans, do to all sorts of funky stuff.

The UAC module can handle the replies to requests it originated, it’s handled through the event route blocks in the native Kamailio diaplan with

event_route[uac:reply] {}

But that doesn’t exist inside KEMI.

But inside our kamailio.cfg we can specify the event callback route:

loadmodule "uac"
modparam("uac", "event_callback", "ksr_uac_event")

Then in our Kemi code (mine is in Python) we can pick it up with:

    def ksr_uac_event(self, msg, evname):
        KSR.info("===== uac module triggered event: " + evname + "\n")
        return 1

And that’s it!

Module debug in Kamailio

Kamailio, VoIP, , ,

Sometimes Kamailio modules don’t behave how you expect them to, and you want to dive a little deeper into what’s going on.

If we load the debugger.so module in our kamailio.cfg file, we can get a bit more of an insight.

For example, if we wanted to deep dive into what RTPengine was thinking we’d include:

loadmodule "debugger"
modparam("debugger", "mod_level_mode", 1)
modparam("debugger", "mod_level", "rtpengine=3")

And bam, now when we run Kamailio we’ll get all the logic that rtpengine exposes when being called.

By setting cfgtrace we can enable a config trace as well, and as we step through our Kamailio config file for a SIP message, we can see what’s going on at every step.

So if you’re like me, you can use the debugger module, combined with reading through the source code, to prove every single time that the issue is with my not understanding the function properly, and never the logic of the Kamailio module!

Kamailio – NDB_Redis – undefined symbol: redisvCommand

Kamailio, Notes, VoIP, ,

I ran into this issue the other day while compiling Kamailio from source:

Jul 03 23:49:35 kamailio[305199]: ERROR: <core> [core/sr_module.c:608]: ksr_load_module(): could not open module </usr/local/lib64/kamailio/modules/ndb_redis.so>: /usr/local/lib64/kamailio/modules/ndb_redis.so: undefined symbol: redisvCommand
Jul 03 23:49:35 kamailio[305199]: CRITICAL: <core> [core/cfg.y:4024]: yyerror_at(): parse error in config file /etc/kamailio/kamailio.cfg, line 487, column 12-22: failed to load module

So what was going on?

Kamailio’s NDB Redis module relies on libhiredis which was installed, but the path it installs to wasn’t in the LD_LIBRARY_PATH environment variable.

So I added /usr/lib/x86_64-linux-gnu to LD_LIBRARY_PATH, recompiled and presto, the error is gone!

export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH

Libpython3.11 problems with Kamailio on Ubuntu 22.04

Kamailio, Python, Software, VoIP, , , , , ,

I run Ubuntu on my desktop and I mess with Kamailio a lot.

Recently I was doing some work with KEMI using Python, but installing the latest versions of Kamailio from the Debian Repos for Kamailio wasn’t working.

The following packages have unmet dependencies:
 kamailio-python3-modules : Depends: libpython3.11 (>= 3.11.0) but 3.11.0~rc1-1~22.04 is to be installed

Kamailio’s Python modules expect libpython3.11 or higher, but Ubuntu 22.04 repos only contain the release candidate – not the final version:

root@amanaki:/home/nick# apt-cache policy libpython3.11
libpython3.11:
  Installed: 3.11.0~rc1-1~22.04
  Candidate: 3.11.0~rc1-1~22.04
  Version table:
 *** 3.11.0~rc1-1~22.04 500
        500 http://au.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages
        100 /var/lib/dpkg/status

Luckily the deadsnakes PPA to the rescue!

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get purge kamailio
sudo apt --fix-broken install
sudo apt-get upgrade
apt-get install kamailio kamailio-python3-modules

And done!

Kamailio Websocket – Allowing binding to Port 443 & Port 80

Kamailio, VoIP,

If you’re setting up Kamailio for support for WebSocket and need to bind to TCP port 80 or TCP port 443, you may run into the issue that permission is denied to bind to these ports when you try and start the service.

On Ubuntu, you can fix this with:

sudo setcap 'cap_net_bind_service=+ep' /usr/sbin/kamailio

Where /usr/sbin/kamailio is the install path for kamailio, you can check this using:

root@amanaki:/etc/kamailio# type kamailio
kamailio is hashed (/usr/sbin/kamailio)

Kamailio Bytes: Stripping SIP Multipart Bodies

FreeSWITCH, IMS / VoLTE, Kamailio, Mobile Networks, VoIP, , ,

For some calls in (such as some IMS emergency calls) you’ll get MIME Multipart Media Encapsulation as the SIP body, as the content-type set to:

Content-Type: multipart/mixed;boundary=968f194ab800ab27

If you’re used to dealing with SIP, you’d expect to see:

Content-Type: application/sdp

This Content-Type multipart/mixed;boundary is totally valid in SIP, in fact RFC 5261 (Message Body Handling in the Session Initiation Protocol (SIP)) details the use of MIME in SIP, and the Geolocation extension uses this, as we see below from a 911 call example.

But while this extension is standardised, and having your SIP Body containing multipart MIME is legal, not everything supports this, including the FreeSWITCH bridge module, which just appends a new SDP body into the Mime Multipart

Site note: I noticed FreeSWITCH Bridge function just appends the new SIP body in the multipart MIME, leaving the original, SDP:

Okay, so how do we replace the MIME Multipart SIP body with a standard SDP?

Well, with Kamalio’s SDP Ops Module, it’s fairly easy:

#If the body is multipart then strip it and replace with a single part
if (has_body("multipart/mixed")) {
	xlog("This has a multipart body");
	if (filter_body("application/sdp")) {
		remove_hf("Content-Type");
		append_hf("Content-Type: application/sdp\r\n");
	} else {
		xlog("Body part application/sdp not found\n");
		}
}

I’ve written about using SDPops to modify SDP before.

And with that we’ll take an SIP message like the one shown on the left, and when relayed, end up with the message on the right:

Simple fix, but saved me having to fix the fault in FreeSWITCH.

VoLTE / IMS – Analysis Challenge

IMS / VoLTE, Kamailio, LTE, Mobile Networks, SDM, VoIP, , , , ,

It’s challenge time, this time we’re going to be looking at an IMS PCAP, and answering some questions to test your IMS analysis chops!

Here’s the packet capture:

IMS_PCAPsDownload PCAPs

Easy Questions

  • What QCI value is used for the IMS bearer?
  • What is the registration expiry?
  • What is the E-UTRAN Cell ID the Subscriber is served by?
  • What is the AMBR of the IMS APN?

Intermediate Questions

  • Is this the first or subsequent registration?
  • What is the Integrity-Key for the registration?
  • What is the FQDN of the S-CSCF?
  • What Nonce value is used and what does it do?
  • What P-CSCF Addresses are returned?
  • What time would the UE need to re-register by in order to stay active?
  • What is the AA-Request in #476 doing?
  • Who is the(opens in a new tab)(opens in a new tab)(opens in a new tab) OEM of the handset?
  • What is the MSISDN associated with this user?

Hard Questions

  • What port is used for the ESP data?
  • Which encryption algorithm and algorithm is used?
  • How many packets are sent over the ESP tunnel to the UE?
  • Where should SIP SUBSCRIBE requests get routed?
  • What’s the model of phone?

The answers for each question are on the next page, let me know in the comments how you went, and if there’s any tricky ones!

Pages: 1 2

RTPengine – Installation & Configuration (Ubuntu 20.04 / 22.04)

Kamailio, VoIP, , , , ,

I wrote a post a few years back covering installing RTPengine on Ubuntu (14.04 / 18.04) but it doesn’t apply in later Ubuntu releases such as 20.04 and 22.04.

To make everyone’s lives easier; David Lublink publishes premade repos for Ubuntu Jammy (22.04) & Focal (20.04).

Note: It looks like Ubuntu 23.04 includes RTPengine in the standard repos, so this won’t be needed in the future.

sudo add-apt-repository ppa:davidlublink/rtpengine
sudo apt update
sudo apt-get install ngcp-rtpengine

The Ambient Capabilities in the systemctl file bit me,

Commenting out : 

#AmbientCapabilities=CAP_NET_ADMIN CAP_SYS_NICE

In /lib/systemd/system/ngcp-rtpengine-daemon.service and then reloading the service and restarting and I was off and running:

systemctl daemon-reload
systemctl restart rtpengine

Getting it Running

Now we’ve got RTPengine installed let’s setup the basics,

There’s an example config file we’ll copy and edit:

vi /etc/rtpengine/rtpengine.conf

We’ll uncomment the interface line and set the IP to the IP we’ll be listening on:

Once we’ve set this to our IP we can start the service:

systemctl restart rtpengine

All going well it’ll start and rtpengine will be running.

You can learn about all the startup parameters and what everything in the config means in the readme.

Want more RTP info?

If you want to integrate RTPengine with Kamailio take a look at my post on how to set up RTPengine with Kamailio.

For more in-depth info on the workings of RTP check out my post RTP – More than you wanted to Know

Authenticating Fixed Line Subscribers into IMS

IMS / VoLTE, Mobile Networks, , , , , , ,

We recently added support in PyHSS for fixed line SIP subscribers to attach to the IMS.

Traditional telecom operators are finding their fixed line network to be a bit of a money pit, something they’re required to keep operating to meet regulatory obligations, but the switches are sitting idle 99% of the time. As such we’re seeing more and more operators move fixed line subs onto their IMS.

This new feature means we can use PyHSS to serve as the brains for a fixed network, as well as for mobile, but there’s one catch – How we authenticate subscribers changes.

Most banks of line cards in a legacy telecom switches, or IP Phones, don’t have SIM slots to allow us to authenticate, so instead we’re forced to fallback to what they do support.

Unfortunately for the most part, what is supported by these IP phones or telecom switches is SIP MD5 Digest Authentication.

The Nonce is generated by the HSS and put into the Multimedia-Authentication-Answer, along with the subscriber’s password and sent in the clear to the S-CSCF.

Subscriber with Password made up of all 1's MAA response from HSS for Digest-MD5 Auth

The HSS then generates the the Multimedia-Auth Answer, it generates a nonce (in the 3GPP-SIP-Authenticate / 609 AVP) and sends the Subscriber’s password in the 3GPP-SIP-Authorization (610) AVP in response back to the S-CSCF.

I would have thought a better option would be for the HSS to generate the Nonce and Digest, and then the S-CSCF to just send the Nonce to the Sub and compare the returned Digest from the Sub against the expected Digest from the HSS, but it would limit flexibility (realm adaptation, etc) I guess.

The UE/UA (I guess it’s a UA in this context as it’s not a mobile) then generates its own Digest from the Nonce and sends it back to the S-CSCF via the P-CSCF.

The S-CSCF compares the received Digest response against the one it generated, and if the two match, the sub is authenticated and allowed to attach onto the network.

Failures in cobbling together a USSD Gateway

IMS / VoLTE, Kamailio, Mobile Networks, VoIP, , , ,

One day recently I was messing with the XCAP server, trying to set the Call Forward timeout. In the process I triggered the UE to send a USSD request to the IMS.

Huh, I thought, “I wonder how hard it would be to build a USSD Gateway for our IMS?”, and this my friends, is the story of how I wasted a good chunk of my weekend trying (and failing) to add support for USSD.

You might be asking “Who still uses USSD?” – The use cases for USSD are pretty thin on the ground in this day and age, but I guess balance query, and uh…

But this is the story of what I tried before giving up and going outside…

Routing

First I’d need to get the USSD traffic towards the USSD Gateway, this means modifying iFCs. Skimming over the spec I can see the Recv-Info: header for USSD traffic should be set to “g.3gpp.ussd” so I knocked up an iFC to match that, and route the traffic to my dev USSD Gateway, and added it to the subscriber profile in PyHSS:

  <!-- SIP USSD Traffic to USSD-GW-->
        <InitialFilterCriteria>
            <Priority>25</Priority>
            <TriggerPoint>
                <ConditionTypeCNF>1</ConditionTypeCNF>
                <SPT>
                    <ConditionNegated>0</ConditionNegated>
                    <Group>1</Group>
                    <SIPHeader>
                      <Header>Recv-Info</Header>
                      <Content>"g.3gpp.ussd"</Content>
                    </SIPHeader>
                </SPT>                
            </TriggerPoint>
            <ApplicationServer>
                <ServerName>sip:ussdgw:5060</ServerName>
                <DefaultHandling>0</DefaultHandling>
            </ApplicationServer>
        </InitialFilterCriteria>

Easy peasy, now we have the USSD requests hitting our USSD Gateway.

The Response

I’ll admit that I didn’t jump straight to the TS doc from the start.

The first place I headed was Google to see if I could find any PCAPs of USSD over IMS/SIP.

And I did – Restcomm seems to have had a USSD product a few years back, and trawling around their stuff provided some reference PCAPs of USSD over SIP.

ussd_20012017.pcap_Download
USSD-Restcomm_dialogTimeoutTest.pcap_.pcapngDownload

So the flow seemed pretty simple, SIP INVITE to set up the session, SIP INFO for in-dialog responses and a BYE at the end.

With all the USSD guts transferred as XML bodies, in a way that’s pretty easy to understand.

Being a Kamailio fan, that’s the first place I started, but quickly realised that SIP proxies, aren’t great at acting as the UAS.

So I needed to generate in-dialog SIP INFO messages, so I turned to the UAC module to generate the SIP INFO response.

My Kamailio code is super simple, but let’s have a look:

request_route {

        xlog("Request $rm from $fU");

        if(is_method("INVITE")){
                xlog("USSD from $fU to $rU (Emergency number) CSeq is $cs ");
                sl_reply("200", "OK Trying USSD Phase 1");      #Generate 200 OK
                route("USSD_Response"); #Call USSD_Response route block
                exit;
        }
}

route["USSD_Response"]{
        xlog("USSD_Response Route");
        #Generate a new UAC Request
        $uac_req(method)="INFO";
        $uac_req(ruri)=$fu;     #Copy From URI to Request URI
        $uac_req(furi)=$tu;     #Copy To URI to From URI
        $uac_req(turi)=$fu;     #Copy From URI to To URI
        $uac_req(callid)=$ci;   #Copy Call-ID
                                #Set Content Type to 3GPP USSD
        $uac_req(hdrs)=$uac_req(hdrs) + "Content-Type: application/vnd.3gpp.ussd+xml\r\n";
                                #Set the USSD XML Response body
        $uac_req(body)="<?xml version='1.0' encoding='UTF-8'?>
        <ussd-data>
                <language value=\"en\"/>
                <ussd-string value=\"Bienvenido. Seleccione una opcion: 1 o 2.\"/>
        </ussd-data>";
        $uac_req(evroute)=1;    #Set the event route to use on return replies
        uac_req_send();         #Send it!
}

So the UAC module generates the 200 OK and sends it back.

“That was quick” I told myself, patting myself on the back before trying it out for the first time.

Huston, we have a problem – Although the Call-ID is the same, it’s not an in-dialog response as the tags aren’t present, this means our UE send back a 405 to the SIP INFO.

Right. Perhaps this is the time to read the Spec…

Okay, so the SIP INFO needs to be in dialog. Can we do that with the UAC module? Perhaps not…

But the Transaction Module ™ in Kamailio exposes and option on the ctl API to generate an in-dialog UAC – this could be perfect…

But alas real life came back to rear its ugly head, and this adventure will have to continue another day…

Update: Thanks to a kindly provided PCAP I now know what I was doing wrong, and so we’ll soon have a follow up to this post named “Successes in cobbling together a USSD Gateway” just as soon as I have a weekend free.

Kamailio Bytes: Adding Prometheus + Grafana to Kamailio

Kamailio, VoIP, , , , ,

I recently fell in love with the Prometheus + Grafana combo, and I’m including it in as much of my workflow as possible, so today we’ll be integrating this with another favorite – Kamailio.

Why would we want to integrate Kamailio into Prometheus + Grafana? Observability, monitoring, alerting, cool dashboards to make it look like you’re doing complicated stuff, this duo have it all!

I’m going to assume some level of familiarity with Prometheus here, and at least a basic level of understanding of Kamailio (if you’ve never worked with Kamailio before, check out my Kamailio 101 Series, then jump back here).

So what will we achieve today?

We’ll start with the simple SIP Registrar in Kamailio from this post, and we’ll add on the xhttp_prom module, and use it to expose some stats on the rate of requests, and responses sent to those requests.

So to get started we’ll need to load some extra modules, xhttp_prom module requires xhttp (If you’d like to learn the basics of xhttp there’s also a Kamailio Bytes – xHTTP Module post covering the basics) so we’ll load both.

xHTTP also has some extra requirements to load, so in the top of our config we’ll explicitly specify what ports we want to bind to, and set two parameters that control how Kamailio handles HTTP requests (otherwise you’ll not get responses for HTTP GET requests).

listen=tcp:0.0.0.0:9090
listen=tcp:0.0.0.0:5060
listen=udp:0.0.0.0:5060

http_reply_parse=yes
tcp_accept_no_cl=yes

Then where you load all your modules we’ll load xhttp and xhttp_prom, and set the basic parameters:

loadmodule "xhttp.so"
loadmodule "xhttp_prom.so"

# Define two counters and a gauge
modparam("xhttp_prom", "xhttp_prom_stats", "all")

By setting xhttp_prom module to expose all stats, this exposes all of Kamailio’s internal stats as counters to Prometheus – This means we don’t need to define all our own counters / histograms / gauges, instead we can use the built in ones from Kamailio. Of course we can define our own custom ones, but we’ll do that in our next post.

Lastly we’ll need to add an event route to handle HTTP requests to the /metrics URL:

event_route[xhttp:request] {
	xlog("Got a request!");
	xlog("$ru");
	$var(xhttp_prom_root) = $(hu{s.substr,0,8});
	if ($var(xhttp_prom_root) == "/metrics") {
			xlog("Called metrics");
			prom_dispatch();
			xlog("prom_dispatch() called");
			return;
	} else
		xhttp_reply("200", "OK", "text/html",
        		"<html><body>Wrong URL $hu</body></html>");
}

Restart, and browse to the IP of your Kamailio instance (mine is 10.01.23) port 9090 /metrics and you’ll get something like this:

Kamailio metrics endpoing used by Prometheus

That my friends, is the sort of data that Prometheus gobbles up, so let’s point Prometheus at it and see what data we get back.

Over on my Prometheus server I’ve edited /etc/prometheus/prometheus.yml to target our new Prometheus endpoint.

  - job_name: "kamailo"
    static_configs:
      - targets: ["10.0.1.23:9090"]  
    honor_timestamps: false

So how can we see this data? Well first off if we log into Prometheus we can see the data flowing in:

If we throw some SIP REGISTER traffic at our Kamailio instance and check on the kamailio_registrar_accepted_regs stat we can see our registrations.

After a few clicks in Grafana we can run some graphs for this data too:

So that’s it, Kamailio’s core stats are now exposed to Prometheus, and we can render this information in Grafana.

There’s a copy of the full code used here available in the Github, and in our next post we’ll look at defining our own metrics in Kamailio and then interacting with them.

Kamailio Diameter Routing Agent Support

IMS / VoLTE, Kamailio, Mobile Networks, RFCs & Standards, Software, VoIP, , , , ,

Recently I’ve been working on open source Diameter Routing Agent implementations (See my posts on FreeDiameter).

With the hurdles to getting a DRA working with open source software covered, the next step was to get all my Diameter traffic routed via the DRAs, however I soon rediscovered a Kamailio limitation regarding support for Diameter Routing Agents.

You see, when Kamailio’s C Diameter Peer module makes a decision as to where to route a request, it looks for the active Diameter peers, and finds a peer with the suitable Vendor and Application IDs in the supported Applications for the Application needed.

Unfortunately, a DRA typically only advertises support for one application – Relay.

This means if you have everything connected via a DRA, Kamailio’s CDP module doesn’t see the Application / Vendor ID for the Diameter application on the DRA, and doesn’t route the traffic to the DRA.

The fix for this was twofold, the first step was to add some logic into Kamailio to determine if the Relay application was advertised in the Capabilities Exchange Request / Answer of the Diameter Peer.

I added the logic to do this and exposed this so you can see if the peer supports Diameter relay when you run “cdp.list_peers”.

With that out of the way, next step was to update the routing logic to not just reject the candidate peer if the Application / Vendor ID for the required application was missing, but to evaluate if the peer supports Diameter Relay, and if it does, keep it in the game.

I added this functionality, and now I’m able to use CDP Peers in Kamailio to allow my P-CSCF, S-CSCF and I-CSCF to route their traffic via a Diameter Routing Agent.

I’ve got a branch with the changes here and will submit a PR to get it hopefully merged into mainline soon.

Filtering for 3GPP DNS in Wireshark

5G SA, EPC, IMS / VoLTE, LTE, Mobile Networks, VoIP, , , , ,

If you work with IMS or Packet Core, there’s a good chance you need DNS to work, and it doesn’t always.

When I run traces, I’ve always found I get swamped with DNS traffic, UE traffic, OS monitoring, updates, etc, all combine into a big firehose – while my Wireshark filters for finding EPC and IMS traffic is pretty good, my achilles heel has always been filtering the DNS traffic to just get the queries and responses I want out of it.

Well, today I made that a bit better.

By adding this to your Wireshark filter:

dns contains 33:67:70:70:6e:65:74:77:6f:72:6b:03:6f:72:67:00

You’ll only see DNS Queries and Responses for domains at the 3gppnetwork.org domain.

This makes my traces much easier to read, and hopefully will do the same for you!

Bonus, here’s my current Wireshark filter for working EPC/IMS:

(diameter and diameter.cmd.code != 280) or  (sip and !(sip.Method == "OPTIONS") and !(sip.CSeq.method == "OPTIONS")) or (smpp and (smpp.command_id != 0x00000015 and smpp.command_id != 0x80000015)) or (mgcp and !(mgcp.req.verb == "AUEP") and !(mgcp.rsp.rspcode == 500)) or isup or sccp or rtpevent or s1ap or gtpv2 or pfcp or (dns contains 33:67:70:70:6e:65:74:77:6f:72:6b:03:6f:72:67:00)

Kamailio I-CSCF – SRV Lookup Behaviour

IMS / VoLTE, Kamailio, Mobile Networks, RFCs & Standards, VoIP, , ,

Recently I had a strange issue I thought I’d share.

Using Kamailio as an Interrogating-CSCF, Kamailio was getting the S-CSCF details from the User-Authorization-Answer’s “Server-Name” (602) AVP.

The value was set to:

sip:scscf.mnc001.mcc001.3gppnetwork.org:5060

But the I-CSCF was only looking up A-Records for scscf.mnc001.mcc001.3gppnetwork.org, not using DNS-SRV.

The problem? The Server-Name I had configured as a full SIP URI in PyHSS including the port, meant that Kamailio only looks up the A-Record, and did not do a DNS-SRV lookup for the domain.

Dropping the port number saw all those delicious SRV records being queried.

Something to keep in mind if you use S-CSCF pooling with a Kamailio based I-CSCF, if you want to use SRV records for load balancing / traffic sharing, don’t include the port, and if instead you want it to go to the specified host found by an A-record, include the port.

Kamailio Bytes – Extracting SDP Parameters with Kamailio

Kamailio, VoIP, , , ,

So the other day I needed to extract the IP and Port parameters from an SDP body – Not the whole line mind, but the values themselves.

As with so many things in Kamailio, there’s a lot of ways to achieve an outcome, but here’s how I approached this problem.

Using the SDPops module we can get a particular line in the SDP, for example, we can get the media line with:

#Get SDP line starting with m= and put it into AVP $avp(mline)
sdp_get_line_startswith("$avp(mline)", "m=")
#Print value of $avp(mline)
xlog("m-line: $avp(mline)\n");

This gets us the line, but now we need to extract the data, in the example from the screenshot the M line has the value:

m=audio 4002 RTP/AVP 8 101

But we only want the port from the M line.

This is where I’ve used the Kamailio Dialplan module and regex to extract the port from this line.

With a fairly simple regex pattern, we can get a group match for the Port from the m= line.

So I took this regular expression, and put it into the Kamailio Dialplan database with dialplan ID 400 for this example:

INSERT INTO `dialplan` VALUES (4,400,10,1,'m=audio (\\d*)',0,'m=audio (\\d*)','\\1','SDP M Port Stripper');

Now using Dialplan ID 400 we can translate an inputted m= SDP line, and get back the port used, so let’s put that into practice:

        if(sdp_get_line_startswith("$avp(mline)", "m=")) {
            xlog("m-line: $avp(mline)\n");
            xlog("raw: $avp(mline)");
            xlog("Extracting Port from Media Line");
            dp_translate("400", "$avp(mline)/$avp(m_port_b_leg)");
            xlog("Translated m_port_b_leg is: $avp(m_port_b_leg)");
        }

Now we have an AVP called $avp(m_port_b_leg) which contains the RTP Port from the SDP.

Now we’ve got a few other values we might want to get, such as the IP the RTP is to go to, etc, we can extract this in the same way, with Dialplans and store them as AVPs:

        #Print current SDP Values and store as Vars
        if(sdp_get_line_startswith("$avp(mline)", "m=")) {
            xlog("m-line: $avp(mline)\n");
            xlog("raw: $avp(mline)");
            xlog("Extracting Port from Media Line");
            dp_translate("400", "$avp(mline)/$avp(m_port_b_leg)");
            xlog("Translated m_port_b_leg is: $avp(m_port_b_leg)");
        }

        if(sdp_get_line_startswith("$avp(oline)", "o=")) {
            xlog("o-line: $avp(oline)\n");
            dp_translate("401", "$avp(oline)/$avp(o_line_port_1)");
            xlog("O Line Port 1: $avp(o_line_port_1)");
            dp_translate("402", "$avp(oline)/$avp(o_line_port_2)");
            xlog("O Line Port 2: $avp(o_line_port_2)");
            dp_translate("403", "$avp(oline)/$avp(o_ip_b_leg)");
            xlog("O IP: $avp(o_ip_b_leg)");
        }

And all the Regex you’ll need:

INSERT INTO `dialplan` VALUES 
(4,400,10,1,'m=audio (\\d*)',0,'m=audio (\\d*)','\\1','SDP M Port Stripper'),
(5,401,10,1,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*.d*.\\d*.\\d*)',0,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*.d*.\\d*.\\d*)','\\1','O Port 1'),
(6,402,10,1,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*.d*.\\d*.\\d*)',0,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*.d*.\\d*.\\d*)','\\2','O Port 2'),
(7,403,10,1,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*.d*.\\d*.\\d*)',0,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*[.]\\d*[.]\\d*[.]\\d*)','\\3','O IP');
IMS DNS Failing

Kamailio, IMS & DNS Headches

IMS / VoLTE, Kamailio, Mobile Networks, VoIP, , , ,

I’m sure I’ve ranted about the importance of DNS in IMS networks in the past on here already.

Recently I was rebuilding a P-CSCF and kept getting an error saying that the DNS was failing to resolve:

 4(5993) CRITICAL: <core> [core/dns_cache.c:3136]: dns_srv_sip_resolve(): unknown proto 0
 4(5993) ERROR: tm [ut.h:284]: uri2dst2(): failed to resolve "ims.mnc001.mcc001.3gppnetwork.org" :bug - critical error (-13)
 4(5993) ERROR: tm [t_fwd.c:1759]: t_forward_nonack(): failure to add branches
 4(5993) ERROR: sl [sl_funcs.c:414]: sl_reply_error(): stateless error reply used: Unresolvable destination (478/SL)

This was a rebuild, another P-CSCF was running fine and handling traffic with the same DNS server set.

I checked the netplan config and confirmed the DNS server was set correctly.

If I did an nslookup on the address that was failing to resolve – pointing it at the correct DNS server, the A & SRV records came back OK, and everything was fine.

Stranger still, after clearing the DNS Cache, and running a packet capture, I couldn’t see any DNS queries at all….

The problem? Kamailio uses resolv.conf by default on Ubuntu Server, and that was pointing to localhost.

After updating resolv.conf to point to the DNS server handling the IMS domains, I was good to go again.

A super valuable resource for all things DNS & Kamailio is this doc.

FreeSWITCH, Kamailio & IMS Extensions

FreeSWITCH, IMS / VoLTE, Kamailio, Mobile Networks, VoIP, , , , ,

Recently I’ve been doing some work with FreeSWITCH as an IMS Conference Factory, I’ve written a bit about it before in this post on using FreeSWITCH with the AMR codec.

Pretty early on in my testing I faced a problem with subsequent in-dialog responses, like re-INVITEs used for holding the calls.

Every subsequent message, was getting a “420 Bad Extension” response from FreeSWITCH.

The ReINVITE
The 420 Bad Extension Response

So what didn’t it like and why was FreeSWITCH generating 420 Bad Extension Responses to these subsequent messages?

Well, the “Extensions” FreeSWITCH is referring to are not extensions in the Telephony sense – as in related to the Dialplan, like an Extension Number to identify a user, but rather the Extensions (as in expansions) to the SIP Protocol introduced for IMS.

The re-INVITE contains a Require header with sec-agree which is a SIP Extension introduced for IMS, which FreeSWITCH does not have support for, and the re-INVITE says is required to support the call (Not true in this case).

Using a Kamailio based S-CSCF means it is easy to strip these Headers before forwarding the requests onto the Application Server, which is what I’ve done, and bingo, no more errors!

The Surprisingly Complicated World of SMS: Apple iPhone MT SMS

5G SA, IMS / VoLTE, Kamailio, LTE, Mobile Networks, VoIP, , , , , ,

In iOS 15, Apple added support for iPhones to support SMS over IMS networks – SMSoIP. Previously iPhone users have been relying on CSFB / SMSoNAS (Using the SGs interface) to send SMS on 4G networks.

Getting this working recently led me to some issues that took me longer than I’d like to admit to work out the root cause of…

I was finding that when sending a Mobile Termianted SMS to an iPhone as a SIP MESSAGE, the iPhone would send back the 200 OK to confirm delivery, but it never showed up on the screen to the user.

The GSM A-I/F headers in an SMS PDU are used primarily for indicating the sender of an SMS (Some carriers are configured to get this from the SIP From header, but the SMS PDU is most common).

The RP-Destination Address is used to indicate the destination for the SMS, and on all the models of handset I’ve been testing with, this is set to the MSISDN of the Subscriber.

But some devices are really finicky about it’s contents. Case in point, Apple iPhones.

If you send a Mobile Terminated SMS to an iPhone, like the one below, the iPhone will accept and send back a 200 OK to this request.

The problem is it will never be displayed to the user… The message is marked as delivered, the phone has accepted it it just hasn’t shown it…

SMS reports as delivered by the iPhone (200 OK back) but never gets displayed to the user of the phone as the RP-Destination Address header is populated

The fix is simple enough, if you set the RP-Destination Address header to 0, the message will be displayed to the user, but still took me a shamefully long time to work out the problem.

RP-Destination Address set to 0 sent to the iPhone, this time it’ll get displayed to the user.

Kamailio Bytes – Working with Redis

Kamailio, VoIP, ,

I’ve become a big fan of Redis, and recently I had a need to integrate it into Kamailio.

There are two modules for integrating Kamailio and Redis, each have different functionalities:

  • db_redis is used when you want to use Redis in lieu of MySQL, PostGres, etc, as the database backend, this would be useful for services like usrloc. Not all queries / function calls are supported, but can be used as a drop-in replacement for a lot of modules that need database connectivity.
  • ndb_redis exposes Redis functions from the Kamailio config file, in a much more generic way. That’s what we’ll be looking at today.

The setup of the module is nice and simple, we load the module and then define the connection to the Redis server:

import "ndb_redis.so"
modparam("ndb_redis", "server", "name=MyRedisServer;addr=127.0.0.2;port=6379")

With the above we’ve created a connection to the Redis server at 127.0.0.2, and it’s called MyRedisServer.

You can define multiple connections to multiple Redis servers, just give each one a different name to reference.

Now if we want to write some data to Redis (SET) we can do it from within the dialplan with:

redis_cmd("MyRedisServer", "SET foo bar", "r");

We can then get this data back with:

#Get value of key "foo" from Redis
redis_cmd("MyRedisServer", "GET foo", "r");
#Set avp "foo_value" to output from Redis
$avp(foo_value) = $redis(r=>value);
#Print out value of avp "foo_value" to syslog
xlog("Value of foo is:  $avp(foo_value))

At the same time, we can view this data in Redis directly by running:

nick@oldfaithful:~$ redis-cli GET foo

Likewise we can set the value of keys and the keys themselves from AVPs from within Kamailio:

#Set the Redis Key to be the Received IP, with the value set to the value of the AVP "youravp"
redis_cmd("MyRedisServer", "SET $ct $avp(youravp)", "r");

All of the Redis functions are exposed through this mechanism, not just get and set, for example we can set the TTL so a record deletes after a set period of time:

#Set key with value of the received IP to expire after 120 seconds
redis_cmd("MyRedisServer", "EXPIRE $ct 120", "r");

I recently used Redis for a distributed flooding prevention mechanism, where the Subscriber’s received IP is used as the key in Redis and the value set to the number of failed auth attempts that subscriber has had, by using Redis we’re able to use the same mechanism across different platforms and easily administer it.