Well, there’s another concept I haven’t introduced yet, and that’s ChargerS, this is a concept / component we’ll dig into deeper for derived charging, but for now just know we need to add a ChargerS rule in order to get CDRs rated:
Well, if you’ve got CDR storage in StoreDB enabled (And you probably do if you’ve been following up until this point), then the answer is a MySQL table, and we can retrive the data with:
sudo mysql cgrates -e "select * from cdrs \G"
For those of you with a bit of MySQL experience under your belt, you’d be able to envisage using the SUM function to total a monthly bill for a customer from this.
Of course we can add CDRs via the API, and you probably already guessed this, but we can retrive CDRs via the API as well, filtering on the key criteria:
This would be useful for generating an invoice or populating recent calls for a customer portal.
Maybe creating rated CDRs and sticking them into a database is exactly what you’re looking to achieve in CGrateS – And if so, great, this is where you can stop – but for many use cases, there’s a want for an automated solution – For your platform to automatically integrate with CGrateS.
If you’ve got an Asterisk/FreeSWITCH/Kamailio or OpenSIPs based platform, then you can integrate CGrateS directly into your platform to add the CDRs automatically, as well as access features like prepaid credit control, concurrent call limits, etc, etc. The process is a little different on each of these platforms, but ultimately under the hood, all of these platforms have some middleware that generates the same API calls we just ran to create the CDR.
So far this tutorial has been heavy on teaching the API, because that’s what CGrateS ultimately is – An API service.
Our platforms like Asterisk and Kamailio with the CGrateS plugins are just CGrateS API clients, and so once we understand how to use and interact with the API it’s a breeze to plug in the module for your platform to generate the API calls to CGrateS required to integrate.
I build phone networks, and unfortunately, I’m not able to be everywhere at once.
This means sometimes I have to test things in networks I may not be within the coverage of.
To get around this, I’ve setup something pretty simple, but also pretty powerful – Remote test phones.
Using a Raspberry Pi, Intel NUC, or any old computer, I’m able to remotely control Android handsets out in the field, in the coverage footprint of whatever network I need.
This means I can make test calls, run speed testing, signal strength measurements, on real phones out in the network, without leaving my office.
Base OS
Because of some particularities with Wayland and X11, for this I’d steer clear of Ubuntu distributions, and suggest using Debian if you’re using x86 hardware, and Raspbian if you’re using a Pi.
Setup Android Debug Bridge (adb)
The base of this whole system is ADB, the Android Debug Bridge, which exposes the ability to remotely control an Android phone over USB.
You can also do this over WiFi, but I find for device testing, wired allows me to airplane mode a device or disable data, which I can’t do if the device is connected to ADB via WiFi.
There’s lot of info online about setting Android Debug Bridge up on your device, unlocking the Developer Mode settings, etc, if you’ve not done this before I’ll just refer you to the official docs.
Before we plug in the phones we’ll need to setup the software on our remote testing machine, which is simple enough:
Now we can plug in each of the remote phones we want to use for testing and run the command “adb devices” which should list the phones with connected to the machine with ADB enabled:
[email protected]:~$ adb devices
List of devices attached
ABCDEFGHIJK unauthenticated
LMNOPQRSTUV unauthenticated
You’ll get a popup on each device asking if you want to allow USB debugging – If this is going to be a set-and-forget deployment, make sure you tick “Always allow from this Computer” so you don’t have to drive out and repeat this step, and away you go.
Lastly we can run adb devices again to confirm everything is in the connected state
Scrcpy
scrcpy an open-source remote screen mirror / controller that allows us to control Android devices from a computer.
In our case we’re going to install with Snap (if you hate snaps as many folks do, you can also compile from source):
After SSHing into the box, we can just run scrcpy and boom, there’s the window we can interact with.
If you’ve got multiple devices connected to the same device, you’ll need to specify the ADB device ID, and of course, you can have multiple sessions open at the same time.
scrcpy -s 61771fe5
That’s it, as simple as that.
Tweaking
A few settings you may need to set:
I like to enable the “Show taps” option so I can see where my mouse is on the touchscreen and see what I’ve done, it makes it a lot easier when recording from the screen as well for the person watching to follow along.
You’ll probably also want to disable the lock screen and keep the screen awake
Some OEMs have an additonal tick box if you want to be able to interact with the device (rather than just view the screen), which often requires signing into an account, if you see this toggle, you’ll need to turn it on:
Ansible Playbook
I’ve had to build a few of these, so I’ve put an Ansible Playbook on Github so you can create your own.
In our last post we introduced the CGrateS API and we used it to add Rates, Destinations and define DestinationRates.
In this post, we’ll create the RatingPlan that references the DestinationRate we just defined, and the RatingProfile that references the RatingPlan, and then, as the cherry on top – We’ll rate some calls.
For anyone looking at the above diagram for the first time, you might be inclined to ask why what is the purpose of having all these layers?
This layered architecture allows all sorts of flexibility, that we wouldn’t otherwise have, for example, we can have multiple RatingPlans defined for the same Destinations, to allow us to have different Products defined, with different destinations and costs.
Likewise we can have multiple RatingProfiles assigned for the same destinations to allow us to generate multiple CDRs for each call, for example a CDR to bill the customer with and a CDR with our wholesale cost.
All this flexibility is enabled by the layered architecture.
Define RatingPlan
Picking up where we left off having just defined the DestinationRate, we’ll need to create a RatingPlan and link it to the DestinationRate, so let’s check on our DestinationRates:
From the output we can see we’ve got the DestinationRate defined, there’s a lot of info returned (I’ve left out most of it), but you can see the Destination, and the Rate associated with it is returned:
So after confirming that our DestinationRates are there, we’ll create a RatingPlan to reference it, for this we’ll use the APIerSv1.SetTPRatingPlan API call.
In our basic example, this really just glues the DestinationRate_AU object to RatingPlan_VoiceCalls.
It’s worth noting that you can use a RatingPlan to link to multiple DestinationRates, for example, we might want to have a different RatingPlan for each region / country, we can do that pretty easily too, in the below example I’ve referenced other Destination Rates (You’d go about defining the DestinationRates for these other destinations / rates the same way as we did in the last example).
One last step before we can test this all end-to-end, and that’s to link the RatingPlan we just defined with a RatingProfile.
StorDB & DataDB
Psych! Before we do that, I’m going to subject you to learning about backends for a while.
So far we’ve skirted around CGrateS architecture, but this is something we need to know for now.
To keep everything fast, a lot of data is cached in what is called a DataDB (if you’ve followed since part 1, then your DataDB is Redis, but there are other options).
To keep everything together, databases are used for storage, called StorDB (in our case we are using MySQL, but again, we can have other options) but calls to this database are minimal to keep the system fast.
If you’re an astute reader, you may have noticed many of our API calls have TP in method name, if the API call has TP in the name, it is storing it in the StoreDB, if it doesn’t, it means it’s storing it only in DataDB.
Why does this matter? Well, let’s look a little more closely and it will become clear:
ApierV1.SetRatingProfile will set the data only in DataDB (Redis), because it’s in the DataDB the change will take effect immediately.
ApierV1.SetTPRatingProfile will set the data only in StoreDB (MySQL), it will not take effect until it is copied from the database (StoreDB) to the cache (DataDB).
After we define the RatingPlan, we need to run this command prior to creating the RatingProfile, so it has something to reference, so we’ll do that by adding:
The last piece of the puzzle to define is the RatingProfile.
We define a few key things in the rating profile:
The Tenant – CGrateS is multitenant out of the box (in our case we’ve used tenant named “cgrates.org“, but you could have different tenants for different customers).
The Category – As we covered in the first post, CGrateS can bill voice calls, SMS, MMS & Data consumption, in this scenario we’re billing calls so we have the value set to *call, but we’ve got many other options. We can use Category to link what RatingPlan is used, for example we might want to offer a premium voice service with guaranteed CLI rates, using a different RatingPlan that charges more per call, or maybe we’re doing mobile and we want a different RatingPlan for use when Roaming, we can use Category to switch that.
The Subject – This is loosely the Source / Calling Party; in our case we’re using a wildcard value *any which will match any Subject
The RatingPlanActivations list the RatingPlanIds of the RatingPlans this RatingProfile uses
So let’s take a look at what we’d run to add this:
Okay, so at this point, all going well, we should have some data loaded, we’ve gone through all those steps to load this data, so now let’s simulate a call to a Mobile Number (22c per minute) for 123 seconds.
We cheated a fair bit, to show something that worked, but it’s not something you’d probably want to use in real life, loading static CSV files gets us off the ground, but in reality we don’t want to manage a system through CSV files.
Instead, we’d want to use an API.
Fair warning – There is some familiarity expected with JSON and RESTful APIs required, we’ll use Python3 for our examples, but you can use any programing language you’re comfortable with, or even CURL commands.
So we’re going to start by clearing out all the data we setup in CGrateS using the cgr-loader tool from those imported CSVs:
redis-cli flushall
sudo mysql -Nse 'show tables' cgrates | while read table; do sudo mysql -e "truncate table $table" cgrates; done
cgr-migrator -exec=*set_versions -stordb_passwd=CGRateS.org
sudo systemctl restart cgrates
So what have we just done? Well, we’ve just cleared all the data in CGrateS. We’re starting with a blank slate.
In this post, we’re going to define some Destinations, some Rates to charge and then some DestinationRates to link each Destination to a Rate.
But this time we’ll be doing this through the CGrateS API.
Introduction to the CGrateS API
CGrateS is all API driven – so let’s get acquainted with this API.
I’ve written a simple Python wrapper you can find here that will make talking to CGRateS a little easier, so let’s take it for a spin and get the Destinations that are loaded into our system:
import cgrateshttpapi
CGRateS_Obj = cgrateshttpapi.CGRateS('172.16.41.133', 2080) #Replace this IP with the IP Address of your CGrateS instance...
destinations = CGRateS_Obj.SendData({'method':'ApierV1.GetTPDestinationIDs','params':[{"TPid":"cgrates.org"}]})['result']
#Pretty print the result:
print("Destinations: ")
pprint.pprint(destinations)
All going well you’ll see something like this back:
Initializing with host 172.16.41.133 on port 2080
Sending Request with Body:
{'method': 'ApierV2.Ping', 'params': [{'Tenant': 'cgrates.org'}]}
Sending Request with Body:
{'method': 'ApierV2.GetTPDestinationIDs', 'params': [{"TPid":"cgrates.org"}]}
Destinations from CGRates: []
So what did we just do? Well, we sent a JSON formatted string to the CGRateS API at 172.16.41.133 on port 2080 – You’ll obviously need to change this to the IP of your CGrateS instance.
In the JSON body we sent we asked for all the Destinations using the ApierV1.GetTPDestinationIDs method, for the TPid ‘cgrates.org’,
And it looks like no destinations were sent back, so let’s change that!
Note: There’s API Version 1 and API Version 2, not all functions exist in both (at least not in the docs) so you have to use a mix.
Adding Destinations via the API
So now we’ve got our API setup, let’s see if we can add a destination!
To add a destination, we’ll need to go to the API guide and find the API call to add a destination – in our case the API call is ApierV2.SetTPDestination and will look like this:
So we’re creating a Destination named Dest_AU_Mobile and Prefix 614 will match this destination.
Note: I like to prefix all my Destinations with Dest_, all my rates with Rate_, etc, so it makes it easy when reading what’s going on what object is what, you may wish to do the same!
So we’ll use the Python code we had before to list the destinations, but this time, we’ll use the ApierV2.SetTPDestination API call to add a destination before listing them, let’s take a look:
If we post this to the CGR engine, we’ll create a rate, named Rate_AU_Mobile_Rate_1 that bills 22 cents per minute, charged every 60 seconds.
Let’s add a few rates:
CGRateS_Obj.SendData({"method":"ApierV1.SetTPRate","params":[{"ID":"Rate_AU_Mobile_Rate_1","TPid":"cgrates.org","RateSlots":[{"ConnectFee":0,"Rate":22,"RateUnit":"60s","RateIncrement":"60s","GroupIntervalStart":"0s"}]}],"id":1})
CGRateS_Obj.SendData({"method":"ApierV1.SetTPRate","params":[{"ID":"Rate_AU_Fixed_Rate_1","TPid":"cgrates.org","RateSlots":[{"ConnectFee":0,"Rate":14,"RateUnit":"60s","RateIncrement":"60s","GroupIntervalStart":"0s"}]}],"id":1})
CGRateS_Obj.SendData({"method":"ApierV1.SetTPRate","params":[{"ID":"Rate_AU_Toll_Free_Rate_1","TPid":"cgrates.org","RateSlots":[{"ConnectFee":25,"Rate":0,"RateUnit":"60s","RateIncrement":"60s","GroupIntervalStart":"0s"}]}],"id":1})
TPRateIds = CGRateS_Obj.SendData({"method":"ApierV1.GetTPRateIds","params":[{"TPid":"cgrates.org"}]})['result']
print(TPRateIds)
for TPRateId in TPRateIds:
print("\tRate: " + str(TPRateId))
All going well, when you add the above, we’ll have added 3 new rates:
Rate Name
Cost
Rate_AU_Fixed_Rate_1
14c per minute charged every 60s
Rate_AU_Mobile_Rate_1
22c per minute charged every 60s
Rate_AU_Toll_Free_Rate_1
25c connection, untimed
Rates we just created
Linking Rates to Destinations
So now with Destinations defined, and Rates defined, it’s time to link these two together!
Destination Rates link our Destinations and Route rates, this decoupling means that we can have one Rate shared by multiple Destinations if we wanted, and makes things very flexible.
For this example, we’re going to map the Destinations to rates like this:
All going well, you’ll see the new DestinationRate we added.
Here’s a good chance to show how we can add multiple bits of data in one API call, we can tweak the ApierV1.SetTPDestinationRate method and include all the DestinationRates we need in one API call:
In our next post, we’ll keep working our way up this diagram, by creating RatingPlans and RatingProfiles to reference the DestinationRate we just created.
In our last post we talked about setting rates in CGrates and testing them out, but what’s the point in learning a charging system without services to charge?
This post focuses on intergrating FreeSWITCH and CGrates, other posts cover integrating Asterisk and CGrates, Kamailio and CGrates and Diameter and CGrates.
Future posts in this series will focus on the CGrates side, but this post will be a bit of a sidebar to get our FreeSWITCH environment connected to CGrates so we can put all our rating and charging logic into FreeSWITCH.
CGrates interacts with FreeSWITCH via the Event-Socket-Language in FreeSWITCH, which I’ve written about before, in essence when enabled, CGrates is able to make decisions regarding if a call should proceed or not, monitor currently up calls, and terminate calls when a subscriber has used their allocated balance.
Adding ESL Binding Support in FreeSWITCH
The configuration for CGrates is defined through the cgrates.json file in /etc/cgrates on your rating server.
By default, FreeSWITCH’s event socket only listens on localhost, as it is a pretty huge security flaw to open it to the world, but in order for our CGrates server to be able to access we’ll need to bind it to an IP Address assigned to the FreeSWITCH server so we can reach it from elsewhere on the network.
You may want to have CGrates installed on a different machine to your FreeSWITCH instance, or you may want to have multiple FreeSWITCH instances all getting credit control from CGrates.
Well, inside the cgrates.json config file, is where we populate the ESL connection details so CGrates can connect to FreeSWITCH.
Next we’ll need to tag the extensions we want to charge,
In order to do this we’ll need to set the type of the account (Ie. Prepaid, Postpaid, etc), and the flags to apply, which dictate which of the modules we’re going to use inside CGrateS.
FreeSWITCH won’t actually parse this info, it’s just passed to CGrateS.
And that’s pretty much it, when you restart FreeSWITCH and CGrates you should see in the CGrates log that it is connected to your FreeSWITCH instance, and when you make a call, FreeSWITCH will authorize it through CGrates.
We’ll get back into the nitty gritty about setting up CGrates in a future post, and cover setting up integration like this with other Platforms (Kamailio / Asterisk) and Protocols (Diameter & Radius) in future posts.
Carl Sagan once famously said “If you wish to make an apple pie from scratch, you must first invent the universe”, we don’t need to go that far back, but if you want to deliver an MMS to a subscriber, first you must deliver an SMS.
Wait, but we’re talking about MMS right? So why are we talking SMS?
Modern MMS transport relies on HTTP, which is client-server based, the phone / UE is the client, and the MMSc is the Server.
The problem with this client-server relationship, is the client requests things from the server, but the server can’t request things from the client.
This presents a problem when it comes to delivering the MMS – The phone / UE will need to request the MMSc provide it the message to be received, but needs to know there is a message to request in the first place.
So this is where SMS comes in. When the MMSc has a message destined for a Subscriber, it sends the phone/UE an SMS, informing that there is an MMS waiting, and providing the URL the MMS can be retrieved from.
This is typically done by MAP or SMPP, to link the MMSc to the SMSc to allow it to send these messages.
This SMS contains the URL to retrieve the MMS at, once the UE receives this SMS, it knows where to retrieve the MMS.
It can then send an HTTP GET to the URL to retrieve the MMS, and lastly sends an HTTP POST to confirm to the MMSc it retrieved it all OK.
MMS Mobile Terminated message flow
So that’s the basics, let’s look at each part of the dialog in some more detail, starting with this magic SMS to tell the UE where to retrieve the MMS from.
WAP PUSH from MMSc sent via SMS
So some things to notice, the user data, which would usually carry the body of our SMS instead contains another protocol, “Wireless Session Protocol” (WSP), and this is the method “Push”.
That in turn is followed by MMS Message Encapsulation, again inside the SMS message body, this time with the MMS specific data.
The From: header contains the sender of the MMS, this is how you can see who the MMS is from, while it’s still downloading.
The expiry indicates to the handset, it it doesn’t download the MMS within the specified time period, it shouldn’t bother, as the message will have expired.
And lastly, and perhaps most importantly, we have the X-MMS-Content-Location header, which tells our subscriber where to download the MMS from.
After this, the UE sends an HTTP GET to the URL in the X-MMS-Content-Location header (typically on the “mms” APN), to retrieve the MMS from the MMSc.
HTTP GET from the UE to the MMSc
The HTTP GET is pretty normal, there’s the usual MMS headers we talked about in the last post, and we just GET the path provided by the MMSc in the WAP PUSH.
The response from the MMSc contains the actual MMS itself, which is almost a mirror of the sending process (the Data component is unchanged from when the sender sent it).
Response to HTTP GET for message retrieval
At this stage our subscriber has retrieve the MMS, but may not have retrieved it fully, or may have had an issue retrieving it.
Instead the UE sends an HTTP POST with the MMS-Message-Type m-notifyresp-ind with the transaction ID, to indicate that it has successfully retrieved the MMS, and at this point the MMS can notify the sender if delivery receipts are enabled, and delete the message from the cache.
And finally the MMSc sends back a 200 OK with no body to confirm it got that too.
Some notes on MMS Security
Reading about unauthenticated GET requests, you may be left wondering what security does MMS have, and what stops you from just going through and sending HTTP GET requests to all the possible URL paths to vacuum up all the MMS?
In the standard, nothing!
Typically the MMSc has some layer of security added by the implementer, to ensure the user retrieving the MMS, is the user the MMS is destined for. Because MMS has no security in the standard, this is typically achieved through Header Enrichment, whereby the P-GW adds a HTTP header with the MSISDN or IMSI of the subscriber, and then the MMSc can evaluate if this subscriber should be able to retrieve that URL.
Another attack vector I played with was sending a SMS based MMS-Notify with a different URL, which if retrieved, would leak the subscriber’s IP, as it would cause the UE to try and get data from that URL.
Recently I had a strange issue I thought I’d share.
Using Kamailio as an Interrogating-CSCF, Kamailio was getting the S-CSCF details from the User-Authorization-Answer’s “Server-Name” (602) AVP.
The value was set to:
sip:scscf.mnc001.mcc001.3gppnetwork.org:5060
But the I-CSCF was only looking up A-Records for scscf.mnc001.mcc001.3gppnetwork.org, not using DNS-SRV.
The problem? The Server-Name I had configured as a full SIP URI in PyHSS including the port, meant that Kamailio only looks up the A-Record, and did not do a DNS-SRV lookup for the domain.
Dropping the port number saw all those delicious SRV records being queried.
Something to keep in mind if you use S-CSCF pooling with a Kamailio based I-CSCF, if you want to use SRV records for load balancing / traffic sharing, don’t include the port, and if instead you want it to go to the specified host found by an A-record, include the port.
So the other day I needed to extract the IP and Port parameters from an SDP body – Not the whole line mind, but the values themselves.
As with so many things in Kamailio, there’s a lot of ways to achieve an outcome, but here’s how I approached this problem.
Using the SDPops module we can get a particular line in the SDP, for example, we can get the media line with:
#Get SDP line starting with m= and put it into AVP $avp(mline)
sdp_get_line_startswith("$avp(mline)", "m=")
#Print value of $avp(mline)
xlog("m-line: $avp(mline)\n");
This gets us the line, but now we need to extract the data, in the example from the screenshot the M line has the value:
m=audio 4002 RTP/AVP 8 101
But we only want the port from the M line.
This is where I’ve used the Kamailio Dialplan module and regex to extract the port from this line.
With a fairly simple regex pattern, we can get a group match for the Port from the m= line.
So I took this regular expression, and put it into the Kamailio Dialplan database with dialplan ID 400 for this example:
INSERT INTO `dialplan` VALUES (4,400,10,1,'m=audio (\\d*)',0,'m=audio (\\d*)','\\1','SDP M Port Stripper');
Now using Dialplan ID 400 we can translate an inputted m= SDP line, and get back the port used, so let’s put that into practice:
if(sdp_get_line_startswith("$avp(mline)", "m=")) {
xlog("m-line: $avp(mline)\n");
xlog("raw: $avp(mline)");
xlog("Extracting Port from Media Line");
dp_translate("400", "$avp(mline)/$avp(m_port_b_leg)");
xlog("Translated m_port_b_leg is: $avp(m_port_b_leg)");
}
Now we have an AVP called $avp(m_port_b_leg) which contains the RTP Port from the SDP.
Now we’ve got a few other values we might want to get, such as the IP the RTP is to go to, etc, we can extract this in the same way, with Dialplans and store them as AVPs:
#Print current SDP Values and store as Vars
if(sdp_get_line_startswith("$avp(mline)", "m=")) {
xlog("m-line: $avp(mline)\n");
xlog("raw: $avp(mline)");
xlog("Extracting Port from Media Line");
dp_translate("400", "$avp(mline)/$avp(m_port_b_leg)");
xlog("Translated m_port_b_leg is: $avp(m_port_b_leg)");
}
if(sdp_get_line_startswith("$avp(oline)", "o=")) {
xlog("o-line: $avp(oline)\n");
dp_translate("401", "$avp(oline)/$avp(o_line_port_1)");
xlog("O Line Port 1: $avp(o_line_port_1)");
dp_translate("402", "$avp(oline)/$avp(o_line_port_2)");
xlog("O Line Port 2: $avp(o_line_port_2)");
dp_translate("403", "$avp(oline)/$avp(o_ip_b_leg)");
xlog("O IP: $avp(o_ip_b_leg)");
}
And all the Regex you’ll need:
INSERT INTO `dialplan` VALUES
(4,400,10,1,'m=audio (\\d*)',0,'m=audio (\\d*)','\\1','SDP M Port Stripper'),
(5,401,10,1,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*.d*.\\d*.\\d*)',0,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*.d*.\\d*.\\d*)','\\1','O Port 1'),
(6,402,10,1,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*.d*.\\d*.\\d*)',0,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*.d*.\\d*.\\d*)','\\2','O Port 2'),
(7,403,10,1,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*.d*.\\d*.\\d*)',0,'o=[^ ]* (\\d*) (\\d*) IN IP4 (\\d*[.]\\d*[.]\\d*[.]\\d*)','\\3','O IP');
If you’ve ever received an SMS from your operator, and the sender was the Operator name for example, you may be left wondering how it’s done.
In IMS you’d think this could be quite simple – You’d set the From header to be the name rather than the MSISDN, but for most SMSoIP deployments, the From header is ignored and instead the c header inside the SMS body is used.
So how do we get it to show text?
Well the TP-Originating address has the “Type of Number” (ToN) field which is typically set to International/National, but value 5 allows for the Digits to instead be alphanumeric characters.
GSM 7 bit encoding on the text in the TP-Originating Address digits and presto, you can send SMS to subscribers where the message shows as From an alphanumeric source.
On Android SMSs received from alphanumeric sources cannot be responded to (“no more “DO NOT REPLY TO THIS MESSAGE” at the end of each text), but on iOS devices you can respond, but if I send an SMS from “Nick” the reply from the subscriber using the iPhone will be sent to MSISDN 6425 (Nick on the telephone keypad).
Sometimes FreeSWITCH XML dialplan is a bit cumbersome to do more complex stuff, particularly to do with interacting with APIs, etc. But we have the option of using scripts written in Python3 to achieve our outcomes and pass variables to/from the dialplan and perform actions as if we were in the dialplan.
For starters we’ll need to install the module and enable it, here’s the StackOverflow thread that got me looking at it where I share the setup steps.
Here is a very simple example I’ve put together to show how we interact with Python3 in FreeSWITCH:
We’ll create a script in /usr/share/freeswitch/scripts/ and call it “CallerName.py”
from freeswitch import *
import sys
def handler(session,args):
#Get Variables from FreeSWITCH
user_name = str(session.getVariable("user_name"))
session.execute("log", "Call from Username: " + str(user_name))
#Check if Username is equal to Nick
if user_name == "Nick":
session.execute("log", "Username is Nick!")
#Convert the Username to Uppercase
session.execute("set", "user_name=" + str(user_name).upper())
#And return to the dialplan
return
else:
#If no matches then log the error
session.execute("log", "CRIT Username is not Nick - Hanging up the call")
#And reject the call
session.execute("hangup", "CALL_REJECTED")
Once we’ve created and saved the file, we’ll need to ensure it is owned by and executable by service user:
After adding this to the dialplan, we’ll need to run a “reloadxml” to reload the dialplan, and now when these actions are hit, the Python script we created will be called, and if the user_name variable is set to “nick” it will be changed to “NICK”, and if it it isn’t, the call will be hung up with a “CALL_REJECTED” response.
Obviously this is a very basic scenario, but I’m using it for things like ACLs from an API, and dynamic call routing, using the familiar and easy to work with Python interpreter.
A lesson learned a long time ago in Net Eng, is that packet captures (seldom) lie, and the answers are almost always in the packets.
The issue is just getting those packets.
The Problem
But if you’re anything like me, you’re working on remote systems from your workstation, and trying to see what’s going on.
For me this goes like this:
SSH into machine in question
Start TCPdump
Hope that I have run it for long enough to capture the event of interest
Stop TCPdump
Change permissions on PCAP file created so I can copy it
SFTP into the machine in question
Transfer the PCAP to my local machine
View the PCAP in Wireshark
Discover I had not run the PCAP for long enough and repeat
Being a Mikrotik user I fell in love with the remote packet sniffer functionality built into them, where the switch/router will copy packets matching a filter and just stream them to the IP of my workstation.
If only there was something I could use to get this same functionality on remote machines – without named pipes, X11 forwarding or any of the other “hacky” solutions…
The Solution
Introducing Scratch’n’Sniff, a simple tcpdump front end that encapsulates all the filtered traffic of interest in TZSP the same as Mikrotiks do, and stream it (in real time) to your local machine for real time viewing in Wireshark.
Using it is very simple:
Capture all traffic on port 5060 on interface enp0s25 and send it to 10.0.1.252 python3 scratchnsniff.py --dstip 10.0.1.252 --packetfilter 'port 5060' --interface enp0s25
Capture all sctp and icmp traffic on interface lo and send it to 10.98.1.2: python3 scratchnsniff.py --dstip 10.98.1.2 --packetfilter 'sctp or icmp' --interface lo
If you’re keen to try it out you can grab it from GitHub – Scratch’n’Sniff and start streaming packets remotely.
The hint to the cause of the error is above it – Codec comparison. If we look at the Audio Codec Compare lines, we can see the GSM codec we are trying to use, does not match the codecs configured in FreeSWITCH, hence getting the INCOMPATIBLE_DESTINATION error – None of the codecs offered match the codecs supported in FreeSWITCH.
So where do we go to fix this?
Well the SIP profile itself defines the codecs that are supported on this SIP profile,
FreeSWITCH SIP Profile (Sofia) codec settings
If you’re using a mostly default config, you’ll see this is set to a global variable, called $${global_codec_prefs}, so let’s take a look at vars.xml where this is defined:
FreeSWITCH default codec selection global variable
And there’s our problem, we need to add the GSM codec into that list to allow the calls,
So we change it to add the codecs we want to support, and reload the changes,
The Codec preferences I need for this IMS Application Server
This was a rebuild, another P-CSCF was running fine and handling traffic with the same DNS server set.
I checked the netplan config and confirmed the DNS server was set correctly.
If I did an nslookup on the address that was failing to resolve – pointing it at the correct DNS server, the A & SRV records came back OK, and everything was fine.
Stranger still, after clearing the DNS Cache, and running a packet capture, I couldn’t see any DNS queries at all….
The problem? Kamailio uses resolv.conf by default on Ubuntu Server, and that was pointing to localhost.
After updating resolv.conf to point to the DNS server handling the IMS domains, I was good to go again.
Recently I’ve been doing some work with FreeSWITCH as an IMS Conference Factory, I’ve written a bit about it before in this post on using FreeSWITCH with the AMR codec.
Pretty early on in my testing I faced a problem with subsequent in-dialog responses, like re-INVITEs used for holding the calls.
Every subsequent message, was getting a “420 Bad Extension” response from FreeSWITCH.
The ReINVITEThe 420 Bad Extension Response
So what didn’t it like and why was FreeSWITCH generating 420 Bad Extension Responses to these subsequent messages?
Well, the “Extensions” FreeSWITCH is referring to are not extensions in the Telephony sense – as in related to the Dialplan, like an Extension Number to identify a user, but rather the Extensions (as in expansions) to the SIP Protocol introduced for IMS.
The re-INVITE contains a Require header with sec-agree which is a SIP Extension introduced for IMS, which FreeSWITCH does not have support for, and the re-INVITE says is required to support the call (Not true in this case).
Using a Kamailio based S-CSCF means it is easy to strip these Headers before forwarding the requests onto the Application Server, which is what I’ve done, and bingo, no more errors!
So you have a VoIP service and you want to rate the calls to charge your customers?
You’re running a mobile network and you need to meter data used by subscribers?
Need to do least-cost routing?
You want to offer prepaid mobile services?
Want to integrate with Asterisk, Kamailio, FreeSWITCH, Radius, Diameter, Packet Core, IMS, you name it!
Well friends, step right up, because today, we’re talking CGrates!
So before we get started, this isn’t going to be a 5 minute tutorial, I’ve a feeling this may end up a big multipart series like some of the others I’ve done. There is a learning curve here, and we’ll climb it together – but it is a climb.
Installation
Let’s start with a Debian based OS, installation is a doddle:
We’re going to use Redis for the DataDB and MariaDB as the StorDB (More on these concepts later), you should know that other backend options are available, but for keeping things simple we’ll just use these two.
Next we’ll get the database and config setup,
cd /usr/share/cgrates/storage/mysql/
./setup_cgr_db.sh root CGRateS.org localhost
cgr-migrator -exec=*set_versions -stordb_passwd=CGRateS.org
Lastly we’ll clone the config files from the GitHub repo:
In its simplest form, rating is taking a service being provided and calculating the cost for it.
The start of this series will focus on voice calls (With SMS, MMS, Data to come), where the callingparty (The person making the call) pays, so let’s imagine calling a Mobile number (Starting with 614) costs $0.22 per minute.
To perform rating we need to determine the Destination, the Rate to be applied, and the time to charge for.
For our example earlier, a call to a mobile (Any number starting with 614) should be charged at $0.22 per minute. So a 1 minute call will cost $0.22 and a 2 minute long call will cost $0.44, and so on.
We’ll also charge calls to fixed numbers (Prefix 612, 613, 617 and 617) at a flat $0.20 regardless of how long the call goes for.
So let’s start putting this whole thing together.
Introduction to RALs
RALs is the component in CGrates that takes care of Rating and Accounting Logic, and in this post, we’ll be looking at Rating.
The rates have hierarchical structure, which we’ll go into throughout this post. I took my notepad doodle of how everything fits together and digitized it below:
Destinations
Destinations are fairly simple, we’ll set them up in our Destinations.csv file, and it will look something like this:
Each entry has an ID (referred to higher up as the Destination ID), and a prefix.
Also notice that some Prefixes share an ID, for example 612, 613, 617 & 618 are under the Destination ID named “DST_AUS_Fixed”, so a call to any of those prefixes would match DST_AUS_Fixed.
Rates
Rates define the price we charge for a service and are defined by our Rates.csv file.
This is nice and clean, a 1 second call costs $0.25, a 60 second call costs $0.25, and a 61 second call costs $0.50, and so on.
This is the standard billing mechanism for residential services, but it does not pro-rata the call – For example a 1 second call is the same cost as a 59 second call ($0.25), and only if you tick over to 61 seconds does it get charged again (Total of $0.50).
Per Second Billing
If you’re doing a high volume of calls, paying for a 3 second long call where someone’s voicemail answers the call and was hung up, may seem a bit steep to pay the same for that as you would pay for 59 seconds of talk time.
Instead Per Second Billing is more common for high volume customers or carrier-interconnects.
This means the rate still be set at $0.25 per minute, but calculated per second.
So the cost of 60 seconds of call is $0.25, but the cost of 30 second call (half a minute) should cost half of that, so a 30 second call would cost $0.125.
How often we asses the charging is defined by the RateIncrement parameter in the Rate Table.
We could achieve the same outcome another way, by setting the RateIncriment to 1 second, and the dividing the rate per minute by 60, we would get the same outcome, but would be more messy and harder to maintain, but you could think of this as $0.25 per minute, or $0.004166667 per second ($0.25/60 seconds).
Flat Rate Billing
Another option that’s commonly used is to charge a flat rate for the call, so when the call is answered, you’re charged that rate, regardless of the length of the call.
Regardless if the call is for 1 second or 10 hours, the charge is the same.
DestinationID – Refers to the DestinationID defined in the Destinations.csv file
RatesTag – Referes to the Rate ID we defined in Rates.csv
RoundingMethod – Defines if we round up or down
RoundingDecimals – Defines how many decimal places to consider before rounding
MaxCost – The maximum cost this can go up to
MaxCostStrategy – What to do if the Maximum Cost is reached – Either make the rest of the call Free or Disconnect the call
So for each entry we’ll define an ID, reference the Destination and the Rate to be applied, the other parts we’ll leave as boilerplate for now, and presto. We have linked our Destinations to Rates.
Rating Plans
We may want to offer different plans for different customers, with different rates.
DestinationRatesId (As defined in DestinationRates.csv)
TimingTag – References a time profile if used
Weight – Used to determine what precedence to use if multiple matches
So as you may imagine we need to link the DestinationRateIDs we just defined together into a Rating Plan, so that’s what I’ve done in the example above.
Rating Profiles
The last step in our chain is to link Customers / Subscribers to the profiles we’ve just defined.
How you allocate a customer to a particular Rating Plan is up to you, there’s numerous ways to approach it, but for this example we’re going to use one Rating Profile for all callers coming from the “cgrates.org” tenant:
Category is used to define the type of service we’re charging for, in this case it’s a call, but could also be an SMS, Data usage, or a custom definition.
Subject is typically the calling party, we could set this to be the Caller ID, but in this case I’ve used a wildcard “*any”
ActivationTime allows us to define a start time for the Rating Profile, for example if all our rates go up on the 1st of each month, we can update the Plans and add a new entry in the Rating Profile with the new Plans with the start time set
RatingPlanID sets the Rating Plan that is used as we defined in RatingPlans.csv
Loading the Rates into CGrates
At the start we’ll be dealing with CGrates through CSV files we import, this is just one way to interface with CGrates, there’s others we’ll cover in due time.
CGRates has a clever realtime architecture that we won’t go into in any great depth, but in order to load data in from a CSV file there’s a simple handy tool to run the process,
Obviously you’ll need to replace with the folder you cloned from GitHub.
Trying it Out
In order for CGrates to work with Kamailio, FreeSWITCH, Asterisk, Diameter, Radius, and a stack of custom options, for rating calls, it has to have common mechanisms for retrieving this data.
CGrates provides an API for rating calls, that’s used by these platforms, and there’s a tool we can use to emulate the signaling for call being charged, without needing to pickup the phone or integrate a platform into it.
The tenant will need to match those defined in the RatingProfiles.csv, the Subject is the Calling Party identity, in our case we’re using a wildcard match so it doesn’t matter really what it’s set to, the Destination is the destination of the call, AnswerTime is time of the call being answered (pretty self explanatory) and the usage defines how many seconds the call has progressed for.
The output is a JSON string, containing a stack of useful information for us, including the Cost of the call, but also the rates that go into the decision making process so we can see the logic that went into the price.
So have a play with setting up more Destinations, Rates, DestinationRates and RatingPlans, in these CSV files, and in our next post we’ll dig a little deeper… And throw away the CSVs all together!
So you want to send a Multimedia Message (Aka MMS or MM)?
Let’s do it – We’ll use the MM1 interface from the UE towards the MMSc (MMS Service Center) to send our Mobile Originated MMS.
Transport & Creation
Out of the box, our UE doesn’t get told by the network anything about where to send MMS messages (Unless set via something like Android’s Carrier Settings). Instead, this is typically configured by the user in the APN settings, by setting the MMSc address (Typically an FQDN), port (Typically 80) and often a Proxy (Which will actually handle the traffic). Lastly under the bearer type, if we’re sending the MMS on the default bearer (the one used for general Internet) then the bearer type will need to change from “default” to “default,mms”. Alternately, if you’re using a dedicated APN for MMS, you’ll need to set the bearer type to “mms”.
With the connectivity side setup, we’ll need to actually generate an MMS to send, something that is encapsulated in an MMS – so a picture is a good start.
We compose a message with this photo, put an address in the message and hit send on the UE.
The UE encapsulates the photo and metadata, such as the To address, into an HTTP POST is sent to the IP & Port of your MMSc (Or proxy if you have that set). The body of this HTTP POST contains the MMS Message Body (In this case our picture).
Our MMSc receives this POST, extracts the headers of interest, and the multimedia message body itself (in our case the photo) ready to be forwarded onto their destination.
PCAP Extract showing MMS m-send-req from UE
Header Enrichment / Charging / Authentication
One thing to note is that the From header is empty.
Often times a UE doesn’t know it’s own MSISDN. While there is an MSISDN EF on the SIM file system, often this is not updated with the correct MSISDN, as a customer may have ported over their number from a different carrier, or had a replacement SIM reissued. There’s also some problems in just trusting the From address set by a UE, without verifying it as anyone could change this.
The MMS standards evolved in parallel to the 3GPP specifications, but were historically specified by the Open Mobile Alliance. Because it is at arms length with 3GPP, SIM based authentication was not used on the MM1 interface from a UE to the MMSc.
In fact, there is no authentication on an MMS specified in the standard, meaning in theory, anyone could send one. To counter this, the P-GW or GGSN handling the subscriber traffic often enables “Header Enrichment” which when it detects traffic on the MMS APN, will add a Header to the Mobile Originated request with the IMSI or MSISDN of the subscriber sending it, which the MMSc can use to bill the customer.
m-send-req Request
Let’s take a closer look into the HTTP POST sent by the UE containing the message.
Firstly we have what looks like a pretty bulk-standard HTTP POST header, albeit with some custom headers prefixed with “X-” and the Content-Type is application/vnd.wap.mms-message.
But immediately after the HTTP header in the HTTP message body, we have the “MMS Message Encapsulation” header:
MMS Message Encapsulation Header from MO MMS
This header contains the Destination we set in the MMS when sending it, the request type (m-send-req) as well as the actual content itself (inside the Data section).
So why the double header? Why not just encapsulate the whole thing in the HTTP Post? When MMS was introduced, most phones didn’t have a HTTP stack baked into them like everything does now. Instead traffic would be going through a WAP Gateway.
When usage of WAP fell away, the standard moved to transport the same payload that was transfered over WAP, to instead be transferred over HTTP.
Inside the Data section we can see the MIME Type of the attachments themselves, in this case, it’s a photo of my desk:
With all this information, the MMSc analyses the headers and stores the message body ready for forwarding onto the recipient(/s).
m-send-conf Response
To confirm successful receipt, the MMSc sends back a 200 OK with a matching Transaction ID, so the UE knows the message was accepted.
Unstructured Supplementary Service Data or “USSD” is the stack used in Cellular Networks to offer interactive text based menus and systems to Subscribers.
If you remember topping up your mobile phone credit via a text menu on your flip phone, there’s a good chance that was USSD*.
For a period, USSD Services provided Sporting Scores, Stock Prices and horoscopes on phones and networks that were not enabled for packet data.
Unlike plain SMS-PP, USSD services are transaction stateful, which means that there is a session / dialog between the subscriber and the USSD gateway that keeps track of the session and what has happened in the session thus far.
T-Mobile website from 2003 covering the features of their USSD based product at the time
Today USSD is primarily used in the network at times when a subscriber may not have balance to access packet data (Internet) services, so primarily is used for recharging with vouchers.
Osmocom’s HLR (osmo-hlr) has an External USSD interface to allow you to define the USSD logic in another entity, for example you could interface the USSD service with a chat bot, or interface with a billing system to manage credit.
Using the example code provided I made a little demo of how the service could be used:
Communication between the USSD Gateway and the HLR is MAP but carried GSUP (Rather than the full MTP3/SCCP/TCAP layers that traditionally MAP stits on top of), and inside the HLR you define the prefixes and which USSD Gateway to route them to (This would allow you to have multiple USSD gateways and route the requests to them based on the code the subscriber sends).
(I had hoped to make a Python example and actually interface it with some external systems, but another day!)
The signaling is fairly straight forward, when the subscriber kicks off the USSD request, the HLR calls a MAP Invoke operation for “processUnstructuredSS-Request”
Unfortunately is seems the stock Android does not support interactive USSD. This is exposed in the Android SDK so applications can access USSD interfaces (including interactive USSD) but the stock dialer on the few phones I played with did not, which threw a bit of a spanner in the works. There are a few apps that can help with this however I didn’t go into any of them.
(or maybe they used SIM Toolkit which had a similar interface)
If you’re trying to wean yourself away from SQL for everything database related, NoSQL based options like CouchDB are fantastic, but require some rewiring of the brain.
Searching in these databases, without SQL, was a learning curve.
I’m interacting with CouchDB using Python, and while you can create views to help find data, sometimes I knew one of the keys in the document and the value, I wanted to find all documents matching that.
In SQL world I’d do a SELECT WHERE and find the data I wanted, but in CouchDB it’s a bit harder to find.
Using the db.find() function with the {selector} as an input you can filter for all records that have the key value pair you’re looking for.
for doc in db.find({'selector': {'keyyouwanttofind': 'valueofkeyyouwanttofind'}}):
print(doc)
I recently finished this one, the book covers the developments of early long distance submarine cables, right up until the 1950s, it’s a good mix of human interest and technological achievement.
I got introduced to Clarke from The Idea Factory – Bell Labs and the Great Age of American Innovation which I read last year, and while it may have been written in the 1950s, this book covers the early days of Gutta-Percha wrapped steel cables being thrown in the ocean, and several failed attempts at linking the Atlantic, up to the Coax systems that were eventually replaced by Fibre after the book was published.
As well as his scientific achievements, and this book, Clarke is perhaps best remembered as the author of 2001: A Space Odyssey.
I’d suggest reading The Undersea Network after Voice Across the Sea, as it somewhat bookends this.
A well written introduction to “The Internet”. If you know AS numbers off the top of your head, you may find it lacking in technical detail, but this book isn’t meant to be a guide on IP Transit Engineering, but rather how the internet was built and continues to run, and the people behind it more than the tech.
Having read Tubes, the Undersea Network and Voice Across the Sea, it made sense to learn more about Cable and Wireless’ history, this one has only just arrived, so I’ve yet to finish read it.
FreeSWITCH 1.8 / Anthony Minessale & Giovanni Maruzzelli
My skills were getting rusty having not used FreeSWITCH for a while, and I found this book just as useful on whatever read I’m up to now as it was when I first read it.
Kitten Clone – Inside Alcatel-Lucent / Douglas Coupland
Strange title for a book, but this short read covers Alcatel Lucent’s history, before the NSN / Nokia merger.
A much lighter read than Signaling System No. 7 (SS7/C7), this is a good introduction to the topic, and has been a really good reference for the Demystifying SS7 & Sigtran Networks (With Labs!) series I’ve been writing.
I pick this book up whenever I need a more detail on part of SS7, and I’ve been making an effort to go through the whole thing. It’s very thorough and covers all the “gotchas”.
Want more telecom goodness?
I have a good old fashioned RSS feed you can subscribe to.
