I’ve always been kinda intrigued by the idea of parallel universes, the idea that there are infinite copies of the universe, with myself and all the people I care about, but each with slight differences to the universe I inhabit.
The ChargerS module provides the Butterfly Effect needed to create infinite instances of our CGrateS events, each with subtle differences.
Typically if you’re charging subscribers for calls, someone else (or multiple someones) may charge you for those calls, for example you charge your subscribers for an outbound call, but other carriers you interconnect with will charge you for terminating those calls to their subscribers, and for incoming calls you may want to charge the other carriers that terminate calls into your network.
By defining in CGrates what your suppliers charge you, or what you charge suppliers, or reseller rates, or commissions, or any other varied call charge, allows us to:
See profit on each call
Understand supplier costs
Enable reselling at different rates
Opens the door to Least-Cost Routing (Without knowing the cost, we can’t find the cheapest)
Ensure you don’t have calls where you make a loss (Supplier charged you more than you charged the customer)
So how do we do this?
Well, we do this with ChargerS.
When I first looked at CGrateS, the ChargerS module seems like an extra step that did nothing,
In ngrep you’d see the ChargerSv1.ProcessEvent request, and the response, but it doesn’t really do anything, and it’s a PITA when you don’t have a Charger defined and everything stops working.
I’ve spoken a lot about SIP on this blog, and I’m going to assume some level of familiarity with telephony since we’re talking about CGrateS (which is mostly used for telephony), but the best concept I can relate ChargerS to is Serial Forking in SIP, but for the CGrateS event.
A single CGrateS “event” (JSON RPC) comes into CGrateS from wherever, but with ChargerS, we can fork that single event into multiple CGrateS events, which are all treated as unique events.
This is where it starts to get interesting, let’s say we want to calculate a supplier cost and a retail cost, well, with ChargerS we define a rule for supplier and a rule for retail, one single event comes into CGrateS, but with ChargerS setup to create a retail and supplier event, then there are now two events inside CGrateS, one for the supplier and one for the retail.
Well, now when the ChargerSv1.ProcessEvent request hits chargers two events will come out and get processed by the rest of CGrateS as if they’re unique events / calls to be rated.
We’ve cloned our event, now we’ve got two copies of the same event.
The first copy (the original event), will be treated exactly as it is now, the other will see a new event generated inside CGrateS, it’ll be a copy of the original event, except for a few minor changes.
Let’s take a look at what happens to our event going through ChargerS when we generate a CDRsV2.ProcessExternalCDR API request:
I’ve tried to keep the above example as minimal as possible, but if we have a look we can now see two events, the first is our default charger, where nothing is changed; it’s got the same category as we set on the ProcessExternalCDR request (call) and the RunID is “default” per the default charger.
But look below and we’ve got another copy, this time the RunID is set to charger_retail, because that’s what we’ve set it to inside the RunID parameter for the charger named CHARGER_Retail, this means when filtering CDRs we’ll be able to spot these ones really easily, and know it’s a fork of a different event.
But importantly we’ve changed some of the values in the CGrateS Event, the same way AttributeS changes stuff.
So what have we changed? Well the Category of the new request is now RetailCharge.
Now if we cast our mind back to setting the RatingProfile back in Tutorial 3, you may remember we set the Category on the RatingProfile.
Now is when this matters. By setting different Categories in our Rating Profile, we can create a new RatingProfile, with the category set to RetailCharge, but referencing a whole different RatingPlan, with different destinations and rates, and this second event that was forked by ChargerS, will match that RatingProfile, and the RatingPlans that go with it.
For everything matching we’ll get two CDRs (if we’re calling *cdrs that is) and they’re treated as totally separate records.
Think about it; by defining a new RatingProfile with category Wholesale with your wholesale rate, and then creating a Charger for that category, you’ll have a retail CDR and a wholesale CDR. Same for reseller rates, commissions, anything!
We’re using this in one of our networks to handle rating for all the SMS traffic, we’ve got various suppliers and sources for A2P and P2P traffic, and having additional chargers to calculate different rates in a different currency for billing our suppliers is super useful.
#Second charger used for calculating the A2P charge for SMS in USD
print(CGRateS_Obj_local.SendData({
"method": "APIerSv1.SetChargerProfile",
"params": [
{
"ID": "CHARGER_SMS_A2P",
"FilterIDs": ["*string:~*req.Category:sms", "*notstring:~*req.Account:gsm_0340"],
'AttributeIDs' : ['*constant:*req.RequestType:*rated;*constant:*req.Category:sms_a2p'],
'RunID' : 'charger_a2p',
'Weight': 0,
}
] } ))
There’s a bunch of reasons you might want to re-rate CDRs in CGrateS.
For me recently I wanted to introduce StatS to process historical CDR data, and I’d messed up some rates and wanting to correct it without deleting the existing data.
We can re-rate CDRs with the *rerate flag like so:
Something to be aware of that’s tripped me up, is that if while re-rating any of the CDRs fails, CGrateS will stop rating the CDRs after it. For example if you get something like this:
Then the full list of CDRs you’ve requested to be re-rated won’t have been re-rated, only the CDRs up to the error, then CGrateS will stop processing the records after it.
So keep an eye on ngrep and make sure you’ve got all your rates and destinations defined correctly, I found putting:
To filter out any CDRs with a -1 cost in the CDR filters means I filter out any CDRs that had failed to rate last time (Assuming you don’t want to fix CDRs that have failed to get rated).
The StatS subsystem allows us to calculate statistics based on CGrateS events.
Each StatS object contains one or more “metrics” which are things like Average call duration, Total call duration, Average call cost or totals and average of other fields.
The first thing we’ll need to do is enable stats in our JSON config file:
Well we’ve created a StatQueueProfile named StatQueueProfile_VoiceStats, in which we’ll store a maximum of 10000000 datapoints (this is important because to calculate an average we need to know all the previous datapoints), for a maximum of forever (Because TTL is -1, if we wanted to store for 1 hour we’d set TTL to 1h.
We’re not matching any FilterIDs, but based on what we covered on the post in FilterS, you can imagine using this to match calls from a given Account / customer, or to a specific group of destinations, or maybe from a given supplier, etc, etc.
What we do have that’s interesting is we have defined a series of metrics.
So what happens if we now generate a bunch of calls? Well, for starters as we’ve got no FilterS defined here, every call will match this StatQueueProfile, and so we’ll collect data for each.
The example code I’ve provided in the repo for this post generates a bunch of calls, and we can check the values for all our Metrics with GetQueueStringMetrics for our :
If we’ve got a TTL set, old values that have existed in the QueueProfile longer than the TTL are removed, but we can also manually clear the values by using the ResetStatQueue endpoint:
One thing to keep in mind is you can’t modify a StatQueue object via the API without resetting the values.
string_indexed_fields in the config file
Sidebar on this – By specifying the string_indexed_fields means that CGrateS will not evaluate every field against Filter rules, but instead only those defined here. This means if you’ve got an event with say 20 fields (AnswerTime, Account, Subject, Destination, RunID, SetupTime, Extra Fields, etc, etc) each of these gets evaluated against a filter, which is pretty processor intensive if your FilterS only ever look at Account and Destination, so by specifying which fields are indexed here to only the fields you use in your filters, you can boost your performance. On the flip side, you can leave this blank to evaluate all fields, but you’ll take a performance hit by doing so.
FilterS do what it says in the name, they are a generic way to define filter rules that an event may or may not match.
Think of them as like a WHERE statement in SQL, they allow us to condition match.
So what would we use FilterS for? Well, let’s first checkout some example use cases:
We might want to provide 100 free minutes on Tuesdays, we know from this post on creating Balances in CGrateS how to create the balance, but we’d use FilterS to make sure the balance is only used on the Tuesday, by adding a filter to check for the day of the week to only match on a Tuesday.
We might define an Attribute to rewrite the Destination number into E.164, but we only want to apply that transformation if the number is in 0NSN format, we apply the translation with AttributeS but we would create a filter to match Destinations that match the given prefixes.
We might want to trigger a counter for calls where the duration of the call (Usage) is greater than 1 hour, we can do this with Thresholds to handle the counting and FilterS to only match if the call duration is greater than 1 hour.
A customer may have multiple DIDs / phone numbers they present as the From header, and we need a way to map phone number “99990001” through to “99990099” as the Account and change the Account to “Customer X”, we can do that with AttributeS to update the Account value in the request, and FilterS to control if that AttributeS rule is matched or not.
FilterS are used all over in CGrateS, if you’ve been following along, you’ve already come across FilterS in the FilterIDs fields in the API, which we’re going to look at using today.
There’s two ways to handle Filters inside CGrateS, they both act the same way but each have some pros and cons.
Inline Filters
The first option is an “inline” filter. Take for example this AttributeS rule using an inline Filter.
A filter is made up of 3 components, the match “type”, the element to compare using the match and the values.
Match Types: The above example is matching based on it being a string (match type *string), but we can also match on prefixes, suffixes, destinations, empty, not equal to something, greater than, less than, timings and more.
Match Elements: Next up we’ve got the element, this is what part of a CGrateS event we’re matching with the Match Type we’ve selected. In the above example we’re matching for if the value is type *string and the Element is ~*req.Account. If you look at the requests (~*req.) in CGrateS, you can see the events, there’s all the standard fields like Account, Subject, Category, Tenant, Destination, etc, plus any custom ones you’re using, all of which we can use as an element to compare with our match type.
Match Values: Lastly we’ve got the conditions we’ll match on, in the example above it’s the string “Nick” – So what we’re checking is the match is *string and the element we’re getting the string from is ~*req.Account and if that matches the value “Nick” then ding-ding-ding- we’ve matched.
Obviously the values change based on what we’re doing, if we were prefix matching, we’d put the prefix to match in the value.
Value can also be a list, separated by the pipe (|) symbol for inline filters, so for example we could match “Nick” and also “Nicholas” (if I’m in trouble) with this inline filter:
Let’s look at one more example, if the Usage is greater than 1 hour:
*gt:~*req.Usage:1h
Inline filter for any Australian E164 prefixes
FilterProfiles
Now we’ve covered the basics of creating Filters with the “Inline” method, let’s consider the limits of this.
If I had defined objects in AttributeS, ThresholdS, ResourceS, Balances and StatS to match when ~*req.Account is “Nick” using an inline filter, and then I change my name, I’d have to go to each of those elements and update them, and that’d be a pain (especially because I’d need to also change my domain name.)
Instead I can create a “Filter Profile” – A reference to a filter that I can reference from AttributeS, ThresholdS, ResourceS, Balances and StatS, and then I only need to update the Filter.
Let’s look at how that would look, first we’d create a new Filter Profile object using the API with:
This is the same as the below inline Filter, like the inline filter it’ll match any time the ~*req.Account is a string that matches “Nick” or “Nicholas”
"*string:~*req.Account:Nick|Nicholas"
And then to update our AttributeS example from earlier, rather than defining the inline filter in the FilterIDs section, we just put the ID of the filter we created above:
We saw in the example above that we could do Logical OR operations, if the Account is equal to “Nick” or “Nicholas”. But what one neat thing we can do with FilterProfiles is to do local AND Operations.
Let’s create a new FilterProfile called Filter_Sunday to match when the AnswerTime matches Timing named “Timing_Sunday”:
Now we can define an Attribute that will only match if the Account is equal to “Nick” or “Nicholas” AND the AnswerTime matches our “Timing_Sunday” timing profile:
So we can evaluate as AND by just putting both FilterProfiles in the FilterIDs field:
"FilterIDs": ["FLTR_X", "FLTR_Y"],
It’s up to you where you use Inline Filters vs Filter Profiles. As a general rule, if you don’t mind setting it on every object you’re touching, or you don’t reuse the Filter much, inline Filters is probably the way to go. But if you use multiple subsystems and want to keep your logic more readable, perhaps use Filter Profiles – but again, there’s no hard rules.
Filter Profiles is something we’ve got fairly good coverage of in the CGrateS UI, but as far as I’m aware there’s not a simple “Test Filter” API endpoint, so generally I test out with AttributeS.
Up until this point in the series, I’ve tried to hide all the complexity of CGrateS, so people following along can see some progress and feel like they’re making it somewhere with CGrateS, but it’s time to tear off the plaster and talk about the actual concepts, about what’s under the hood, and how all the components interact, as it’ll make it much easier then for us to learn more about how to use CGrateS.
This will be the last post in the “CGrateS in Baby Steps” series (Which I started in 2022), if you’ve made it this far congratulations, all the future posts will be on specific topics and build upon the concepts we’ve covered here.
This took me a while to grasp – CGrateS is both crazy complex and beautifully simple, but getting to the stage where you can “see through the matrix” on CGrateS and see the beautiful simplicity involves a bit of understanding how everything fits together.
Once you realize once you can see the pattern, and understand the building blocks, everything else CGrateS related becomes super simple.
Agents
in CGrateS Agents are consumers of the services. That’s a super generic answer, but let’s take a closer look at what that actually means with some examples:
Diameter is a protocol that can be used for Online Charging. CGrateS has a common interface for API calls that can perform Online Charging. The CGrateS Diameter Agent translates between Diameter on one side, and CGrateS API calls on the other.
Likewise, if we want to speak Radius, we can use the CGrateS Radius Agent, this translates between RADIUS and the CGrateS API calls.
FreeSWITCH, Asterisk and Kamailio don’t use specific protocols like Diameter or Radius, but rather modules or plugins to connect that application to a CGrateS Agent, and they all just end up talking the same CGrateS API calls.
Lastly, there’s even an HTTP agent so you could define your own agent to talk another protocol if you wanted to use CGrateS for anything else (We’ve been playing with CAMEL based charging with CGrateS and 5GC charging).
The config for each of the CGrateS Agents happens in the cgrates.json config file (Typically in /etc/cgrates).
Because the Agents just translate everything into API calls, logic for billing a call from FreeSWITCH is the same as for Diameter, the same as for RADIUS, the same as for SIP, the same as for Asterisk.
The Agents just translate all the domain-specific stuff into the common CGrateS RPC API, which we’ve been working with up to this point.
This is key part to understand; because once you understand how to do the CGrateS part, moving from Asterisk to FreeSWITCH, to DNS, to RADIUS, to any other Agent, it’s all the same to you.
The Agents just translate domain-specific stuff (Diameter requests, CSV files, Asterisk Calls, FreeSWITCH calls, etc, etc) and act as a translator to translate these requests into CGrateS RPC API calls.
On the left side of the image below are the Agents, and on the right side, the Subsystems that do stuff with things.
Subsystems
So with these API calls, where do they go, what do they do?
Well, it’s the Subsystems that do the things.
What things?
Well, everything of use.
Each subsystem has a purpose, AttributeS transforms stuff, EeS exports CDRs, RALs applies our charging logic, CDRs writes CDRs to StorDB, etc, etc.
In each event we can set flags to denote which subsystems it should be routed to, and we can set the links between components in our cgrates.json file.
Based on the flags, we pass events between these subsystems.
Events
So our Agents create the API calls, which contain Events, which are JSON RPC calls.
They look like all the API examples we’ve played with, because that’s exactly what they are.
We can access them via the JSON RPC API, but when you start a call on Kamailio, the Kamailio Agent generates a JSON RPC API call containing an Event into CGrateS for that call on Kamailio.
When you send a DNS request, the DNS Agent translate this DNS request into a CGrateS JSON RPC API containing the event for the DNS request.
Let’s take an example, we’re going to use the ErS as it’s the simplest to demonstrate with.
So if you setup your enviroment per the tutorial above (but don’t load the CSV yet), we’ll start running some experiments…
Anatomy of an Event
We can “sniff” the events bouncing around between the Agent and the various Subsystems in real time, by using ngrep:
sudo ngrep -t -W byline port 2012 or port 2080 or port 8021 or port 2014 or port 2053 -d any
So let’ we’ve got ngrep running, we can move our CSV file in to be processed in another tab.
Plonking the CSV file into the path ERS is monitoring will mean the ErS Agent will generate a CGrateS JSON RPC “event” for each row in the file, it’ll look something like this:
Sidebar – you’re going to spend a lot of time with `ngrep`.
Alright, that event probably looks familiar, after all, it’s the same structure as the API requests we’ve made to CGrateS so far, to set rates and handle accounts.
But what we’re witnessing here isn’t us making an API request to the JSON RPC interface from a Python script, it’s the ERS Agent inside CGrateS, calling CGrateS.
The ERS Agent inside CGrateS reads the CSV file we dropped in, and based on what we had set in the ERS section of the CGrateS config file (cgrates.json), the ERS Agent create JSON RPC events and sent it to CGrateS for processing.
You may be thinking “Wow, the ERS Agent is really dumb, it just sends an API request (events)”, and you’d be right.
We could replace the ERS Agent with a Python script to read the CSV and send the same request, and we’d get the exact same outcome, but CGrateS is mostly “batteries included” so we don’t have to.
Ok, so you’ve heard me drum in the fact that Agents are pretty simple, and all they do is make JSON RPC requests for the event which are sent to CGrateS. So now what happens?
Well, the event is calling CDRsV1.ProcessEvent, so that means the Event is passed by CGRengine to the CDRs subsystem.
What does CDRs subsystem do with it? Well, that’s going to depend on what’s in our cgrates.json config file,
In the above example, CDRs is setup with connections to the different subsystems, AttributeS, Chargers and RALs are all the subsystems linked from here.
Having these links here does not force the Event to always route to these Subsystems, but unless we’ve got the links there, the Event won’t be able to get routed from CDRs to that subsystem if we want it to.
But we can see what’s going to happen with this request based on our CDRsV1.ProcessEvent event, it’s got Flags set to rals, so we know it wants RALs to be called.
So looking in ngrep we see our CDRsV1.ProcessExternalCDR event makes it to the CDRs module with ID 1.
The API call has flags set to *rals so the CDRs will call RALs , and inside our config the CDRs section has a link in the config (shown in the image below) to RALs (rals_conns) – if we didn’t have that link, CGrateS wouldn’t know how to connect to RALs, and the event would fail.
Note at the bottom the APIOpts section tells us this API call was made by the *cdrs subsystem and the ID is 2 (This is a different request to the original CDRsV1.ProcessExternalCDR request which had ID 1 – we can use this to match responses to requests).
Again, because our config also includes links ChargerS and RALS subsystems, we’ll see requests to (you guessed it) ChargerS (The ChargerSv1.ProcessEvent) and RALS (Responder.GetCost).
# T 2024/12/22 09:09:47.465711 127.0.0.1:2012 -> 127.0.0.1:50456 [AP] #414 {"id":4,"result":{"Category":"call","Tenant":"cgrates.org","Subject":"61812341234","Account":"61412341234","Destination":"61812341234","ToR":"*voice","Cost":14,"Timespans":[{"TimeStart":"2024-01-01T01:00:00+11:00","TimeEnd":"2024-01-01T01:01:00+11:00","Cost":14,"RateInterval":{"Timing":{"ID":"*any","Years":[],"Months":[],"MonthDays":[],"WeekDays":[],"StartTime":"00:00:00","EndTime":""},"Rating":{"ConnectFee":0,"RoundingMethod":"*up","RoundingDecimals":4,"MaxCost":0,"MaxCostStrategy":"","Rates":[{"GroupIntervalStart":0,"Value":14,"RateIncrement":60000000000,"RateUnit":60000000000}]},"Weight":10},"DurationIndex":60000000000,"Increments":[{"Duration":0,"Cost":0,"BalanceInfo":{"Unit":null,"Monetary":null,"AccountID":""},"CompressFactor":1},{"Duration":60000000000,"Cost":14,"BalanceInfo":{"Unit":null,"Monetary":null,"AccountID":""},"CompressFactor":1}],"RoundIncrement":null,"MatchedSubject":"*out:cgrates.org:call:*any","MatchedPrefix":"618","MatchedDestId":"Dest_AU_Fixed","RatingPlanId":"RatingPlan_VoiceCalls","CompressFactor":1}],"RatedUsage":60000000000,"AccountSummary":null},"error":null}
What we’re seeing is the CDRs module, calling RALs, to get the cost information for this event.
Finally the CDRsV1.ProcessEvent that was initially sent by ErS gets a result (we can find the result to the request as it’ll have the same id parameter)
So that’s it, that’s the secret sauce – CGrateS is just a bunch of little APIs we combo together to create something great.
Recap
Agents translate data sources into API calls.
Each little API belongs to a Subsystem, like ChargerS, AttributeS or RALs, and we can chain them together in our config file or through the flags in the API request.
Once you’ve got your head wrapped around this, everything in CGrateS becomes way easier.
From now on I’ll pivot to talking about specific modules, and how we use them, starting with AttributeS (which I wrote last year while still drafting this), and diving into how to use each module in more detail.
I started seeing this error the other day when running CDRsv1.GetCDRs on the CGrateS API:
SERVER_ERROR: unexpected end of JSON input
It seemed related to certain CDRs in the cdrs table of StoreDB.
After some digging, I found the stupid simple problem:
I’d written too much data to extra_fields, leading MySQL to cut off the data mid way through, meaning it couldn’t be reconstructed as JSON by CGrateS again.
Like the rounding issue I had, this wasn’t an issue with CGrateS but with MySQL.
Quick fix:
sudo mysql cgrates -e "ALTER TABLE cdrs MODIFY extra_fields LONGTEXT;"
And new fields can exceed this length without being cut off.
After we setup CgrateS the next thing we’d generally want to do would be to rate some traffic.
Of course, that could be realtime traffic, from Diameter, Radius, Kamailio, FreeSWITCH, Asterisk or whatever your case may be, but it could just as easily be CSV files, records from a database or a text file.
We’re going to be rating CDRs from simple CSV files with the date of the event, calling party, called party, and talk time, but of course your CDR exports will have a different format, and that’s to be expected – we tailor the Event Reader Service to match the format of the files we need.
The Event Reader Service, like everything inside CgrateS, is modular. ERS is a module we load that parses files using the rules we define, and creates Events that CgrateS can process and charge for, based on the rules we define.
But before I can tell you that story, I have to tell you this story…
Nick’s imaginary CSV factory
In the repo I’ve added a DummyCSV.csv, it’s (as you might have guessed) a CSV file.
This CSV file is like a million other CSV formats out there – We’ve got a CSV file with Start Time, End Time, Customer, Talk Time, Calling Party, Called Party, Animal (for reasons) and CallID to uniquely identify this CDR.
Protip: The Rainbow CSV VScode extension makes viewing/editing/querying CSV files in VScode much easier.
Call Start Time
Row 0
Call End Time
Row 1
Customer
Row 2
Talk Time
Row 3
Calling Party
Row 4
Called Party
Row 5
Animal
Row 6
CallID
Row 7
File Format
Next we need to feed this into CGrateS, and for that we’ll be using the Event Reporter Service.
JSON config files don’t make for riveting blog posts, but you’ve made it this far, so let’s power through.
ERS is setup in CGrateS’ JSON config file, where we’ll need to define one or more readers which are the the logic we define inside CGrateS to tell it what fields are what, where to find the files we need to import, and set all the parameters for the imports.
This means if we have a CSV file type we get from one of our suppliers with CDRs in it, we’d define a reader to parse that type of file. Likewise, if we’ve got a CSV of SMS traffic out of our SMSc, we’d need to define another reader to parse the CDRs in that format – Generally we’ll do a Reader for each file type we want to parse.
So let’s define a reader for this CSV spec we’ve just defined:
"ers": {
"enabled": true,
"readers": [
{
"id": "blog_example_csv_parser",
"enabled": true,
"run_delay": "-1",
"type": "*file_csv",
"opts": {
"csvFieldSeparator":",",
"csvLazyQuotes": true,
//csvLazyQuotes Counts the row length and if does not match this value declares an error
//-1 means to look at the first row and use that as the row length
"csvRowLength": -1
},
"source_path": "/var/spool/cgrates/blog_example_csv_parser/in",
"processed_path": "/var/spool/cgrates/blog_example_csv_parser/out",
"concurrent_requests": 1024, //How many files to process at the same time
"flags": [
"*cdrs",
"*log"
],
"tenant": "cgrates.org",
"filters": [
"*string:~*req.2:Nick", //Only process CDRs where Customer column == "Nick"
],
"fields":[]
}]}
This should hopefully be relatively simple (I’ve commented it as best I can).
The ID of the ERS object is just the name of this reader – you can name it anything you like, keeping in mind we can have multiple readers defined for different file formats we may want to read, and setting the ID just helps to differentiate them.
The run_delay of -1 means ERS will run as soon as a file is moved into the source_path directory, and the type is a CSV file – Note that’s moved not copied. We’ve got to move the file, not just copy it, as CGrateS waits for the inode notify.
In the opts section we set the specifics for the CSV we’re reading, field separator if how we’re separating the values in our CSV, and in our case, we’re using commas to delineate the fields, but if you were using a file using semicolons or another delineator, you’d adjust this.
Lastly we’ve got the paths, the source path is where we’ll need to move the files to get processed into, and the processed_path is where the processed files will end up.
For now I’ve set the flags to *log and *cdrs – By calling log we’ll make our lives a bit easier for debugging, and CDRs will send the event to the CDRs module to generate a rated CDR in CGrateS, which we could then use to bill a customer, a supplier, etc, and access via the API or exporting using Event Exporter Service.
Lastly under FilterS we’re able to define the filters that should define if we should process a row or not. You don’t know how much you need this feature until you need this feature. The filter rule I’ve included will only process lines where the Customer field in the CSV (row #2) is equal to “Nick”. You could use this to also filter only calls that have been answered, only calls to off-net, etc, etc – FilterS needs a blog post all on it’s own (and if you’re reading this in the future I may have already written one).
Alright, so far so good, we’ve just defined the metadata we need to do to read the file, but now how do we actually get down to parsing the lines in the file?
Well, that’s where the data in Fields: [] comes in.
If you’ve been following along the CgrateS in baby steps series, you’ll have rated a CDR using the API, that looked something like this:
ERS is going to use the same API to rate a CDR, calling more-or-less the same API, so we’re going to set the parameters that go into this from the CSV contents inside the fields:
"fields":[
//Type of Record (Voice)
{"tag": "ToR", "path": "*cgreq.ToR", "type": "*constant", "value": "*voice"},
//Category set to "call" to match RatingProfile_VoiceCalls from our RatingProfile
{"tag": "Category", "path": "*cgreq.Category", "type": "*constant", "value": "call"},
//RequestType is *rated as we won't be deducting from an account balance
{"tag": "RequestType", "path": "*cgreq.RequestType", "type": "*constant", "value": "*rated"},
]
That’s the static values out of the way, next up we’ll define our values we pluck from the CSV. We can get the value of each row from “~*req.ColumnNumber” where ColumnNumber is the column number starting from 0.
//Unique ID for this call - We get this from the CallID field in the CSV
{"tag": "OriginID", "path": "*cgreq.OriginID", "type": "*variable","value":"~*req.7"},
//Account is the Source of the call
{"tag": "Account", "path": "*cgreq.Account", "type": "*variable", "value": "~*req.4"},
//Destination is B Party Number - We use 'Called Party Number'
{"tag": "Destination", "path": "*cgreq.Destination", "type": "*variable", "value": "~*req.5"},
{"tag": "Subject", "path": "*cgreq.Subject", "type": "*variable", "value": "~*req.5"},
//Call Setup Time (In this case, CGrateS can already process this as a datetime object)
{"tag": "SetupTime", "path": "*cgreq.SetupTime", "type": "*variable", "value": "~*req.0"},
//Usage in seconds - We use 'Call duration'
{"tag": "Usage", "path": "*cgreq.Usage", "type": "*variable", "value": "~*req.3"},
//We can include extra columns with extra data - Like this one:
{"tag": "Animal", "path": "*cgreq.Animal", "type": "*variable", "value": "~*req.6"},
]
You’ll need to restart CGrateS after putting the config changes in, but your instance will probably fail to start as we’ll need to create the directories we specified CGrateS should monitor for incoming CSV files:
But before we can put this all into play, we’ll need to setup some rates. My previous posts have covered how to do this, so for that I’ve included a Python script to setup all the rates, which you can run once you’ve restarted CGrateS.
Alright, with that out of the way, we can test it out, move our Dummy.csv file to /var/spool/cgrates/blog_example_csv_parser/in and see what happens.
Having rated CDRs in CGrateS is great, but in reality, you probably want to get them into a billing system, CSV file, S3 bucket, CRM, invoice, Grafana, SQL table, etc, etc.
The Event Exporter Service (EES (previously called CDRe)) handles exporting CDRs from CGrateS.
Like everything in CGrateS, it’s highly configurable, and, again, like everything in CGrateS, supports every combination of services you can think of, plus a stack you haven’t thought of.
CDRs can be exported one of two ways, in real time, as the CDR is generated (online), or after the fact, exporting from the database containing the CDRs (offline).
Exporting in realtime (online) is a great option if you don’t want (or need) to store the CDRs in CGrateS; if you’re just using CGrateS to rate calls and spit them into a seperate system, this is a fantastic option, as it allows your CGrateS instances to remain light and not get clogged up with lots of old CDRs – That said, of course you can export the CDRs in realtime and still store them in CGrateS, that’s also a totally valid approach as well.
The more traditional approach is offline CDR export, where periodically or when an event is triggered, you scrape up a pile of CDRs and send them to your external systems.
For both options, we’ll need to define at least one exporter in our cgrates.json config file. For this example we’ll define a HTTP POST that we will trigger for realtime (online) CDR exporting, and a CSV file we dump to periodically when called from the API.
So first things first, we enable the EES module in the config:
"ees": {
"enabled": true,
"exporters": [
]
}
We’ll start with defining one exporter, named CSVExporter, that will output files to a folder named “testCSV” in the /tmp/ directory, but you can plonk these files wherever you like:
We’ve got a lot of different types of export available to us, but type *file_csv is the easiest, so that’s where we’ll start.
Setting synchronous to true will mean we’ll only run one export job at a time, but it also means we’ll get back the result via the API, which will allow us to keep track of the ID of the last record we updated, so we don’t export the same record multiple times, more on this later.
Flags allows us to, if we wanted, bounce the event through AttributeS, for example, by adding *attributes to the flags, but in this case, it’s just logging to syslog.
Of course, just enabling ees won’t actually send calls to it, we’ll need to add “ees_conns“: [“*localhost”], to “apiers”: and “cdrs” so they know to bounce the events through it:
If you’ve already got CDRs on your system from our previous tutorial, fantastic, but if not, let’s get up and running with a quick and dirty script to define some destinations, a charger, an account balance and then use some of the balance to generate a CDR:
import cgrateshttpapi
import pprint
import uuid
import datetime
now = datetime.datetime.now()
CGRateS_Obj = cgrateshttpapi.CGRateS('localhost', 2080)
#Define Destinations
CGRateS_Obj.SendData({'method':'ApierV2.SetTPDestination','params':[{"TPid":'cgrates.org',"ID":"Dest_AU_Mobile","Prefixes":["614"]}]})
#Load TariffPlan we just defined from StorDB to DataDB
CGRateS_Obj.SendData({"method":"APIerSv1.LoadTariffPlanFromStorDb","params":[{"TPid":'cgrates.org',"DryRun":False,"Validate":True,"APIOpts":None,"Caching":None}],"id":0})
#Define default Charger
print(CGRateS_Obj.SendData({"method": "APIerSv1.SetChargerProfile","params": [{"Tenant": "cgrates.org","ID": "DEFAULT",'FilterIDs': [],'AttributeIDs' : ['*none'],'Weight': 0,}]}))
account = "Nick_Test_123"
#Add a balance to the account with type *sms with 100 sms events
pprint.pprint(CGRateS_Obj.SendData({"method": "ApierV1.SetBalance","params": [{"Tenant": "cgrates.org","Account": account,"BalanceType": "*sms","DestinationIDs": 'Dest_NZ_Mobile;Dest_AU_Mobile',"Categories": "*any","Balance": {"ID": "100_SMS_Bundle_AU_NZ_Mobile","Value": 100,"Weight": 25}}]}))
#Process CDR Event for a single SMS
pprint.pprint(CGRateS_Obj.SendData({"method": "CDRsV2.ProcessExternalCDR","params": [{"OriginID": str(uuid.uuid1()),"ToR": "*sms","RequestType": "*pseudoprepaid","AnswerTime": now.strftime("%Y-%m-%d %H:%M:%S"),"SetupTime": now.strftime("%Y-%m-%d %H:%M:%S"),"Tenant": "cgrates.org","Account": account,"Destination" : "61412345678","Usage": "1",}]}))
Right, with that out of the way, we should now have something in our CDRs table, a quick SQL query confirms this is the case:
So, as you may have guessed, we’ve called the ExportCDRs API endpoint, we’ve specified which ExporterIDs we want to reference (these link back to the objects in the config, and the one we have defined currently is named CSVExporter).
Setting Verbose: True means that CGrateS gives us back a lot of info from the API call, here’s what we get back:
Now that looks pretty positive, we got 12 events of SMS usage exported, which we can see in the file /tmp/testCSV/CSVExporter_21e9bc2.csv – and if we cat out the file, yeap, there’s all the CDRs.
But it’s a bit of a mess, there’s a lot of fields in there, so let’s adjust what goes into the CSV.
Let’s start by filtering what goes into the exporter, to only give us SMS events, of course you could adjust the filters here to target exporting only the records you want, based on anything you can define with Filters (and there’s a lot you can define with filters).
Now we’re only exporting SMS records, so let’s clean up the output of the CSV to just give us the data we want, which is the CDR ID, time, account, destination and usage.
Now after a restart of CGrateS, our exports look like this:
Stunning, truly beautiful, look at that output!
Right, well you may at this point have noticed a problem if you’ve run this more than once. The problem is that is every time we run this, we get all the CDRs since the beginning of time.
But where filtering by date/time falls down, is that if an offline CDR of a call on Monday, only got ingested on Tuesday, it would be missed by the export.
But, setting Verbose: True on the ExportCDRs API call gives us a handy trick, we’ve been told what the highest ID in the CDRs table we just exported in the response from the API in LastExpOrderID field.
If we jump over to the SQL database we use for StorDB, we can see that 33 is the ID of the highest CDR in the system.
So let’s try something, let’s run the exporter again, but this time let’s get all the CDRs where the ID is higher than 33:
#Process CDR Event for a single SMS
pprint.pprint(CGRateS_Obj.SendData({"method": "CDRsV2.ProcessExternalCDR","params": [{"OriginID": str(uuid.uuid1()),"ToR": "*sms","RequestType": "*pseudoprepaid","AnswerTime": now.strftime("%Y-%m-%d %H:%M:%S"),"SetupTime": now.strftime("%Y-%m-%d %H:%M:%S"),"Tenant": "cgrates.org","Account": account,"Destination" : "61412345678","Usage": "1",}]}))
#Trigger export where the OrderID is above 33
result = CGRateS_Obj.SendData({"method":"APIerSv1.ExportCDRs","params":[
{"ExporterIDs": ["CSVExporter"],
"Verbose" : True,
"ExtraArgs" : {
"OrderIDStart" : int(33),
},
"Accounts" : [account]}
]})
pprint.pprint(result)
Boom, now if we have a look at the output we can see the export covered two records, and the last ID was 35.
So as long as we keep track of the LastExpOrderID value, and feed that as in input every time we run ExportCDRs, we can ensure we never miss a CDR, and never get the same CDR twice.
I got an email the other day asking a simple question:
How do I know if a subscriber is VoLTE roaming or not when they send an SMS to charge for it?
My immediate reaction was to look at the SIP headers, P-Access-Network-Info will tell you where the subscriber is located, end of.
Right?
Well not quite, this will tell the SMSc the location of the subscriber sending the SMS. If the PLMN in the P-Access-Network-Info != the home PLMN, the sub is roaming.
But does this information get passed to the OCS / OFCS?
The SMSc uses “Event based charging” to perform credit control, so let’s have a look at what AVPs are present in the Credit Control Request from the SMSc:
Hmm, the SMS-Information AVP (2000) contains a bunch of information about the SMS being sent, but I don’t see anything about the location of the sender in there.
Originator-Interface is just set to “SIP”, of course in a 2G/3G roaming scenario the Originator-SCCP-Address would be that of the Visited PLMN, but for us it is our SCCP address.
Maybe the standard allows for an additional optional AVP in the SMS-Information-AVP we’re missing? Let’s check TS 32.299:
Nope.
So how to deal with this?
While the standards aren’t totally clear on this, we added an IMS-Info AVP and inside that populated the Access-Network-Information directly from the SIP header, and then picked that off inside our OCS in order to apply the correct rules.
The docs describe AttributeS as a Key-Value-Store, but that’s probably selling it short – You can do some really cool stuff with AttributeS, and in this post, we’re going to learn about using AttributeS to transform stuff.
Note: Before we get started, I’d suggest copying this config file to use for testing.
Let’s look at a really basic example, where we add some data into AttributeS, match based on Account in CGrateS, and get back that data.
Well, for starters we’re calling the SetAttributeProfile endpoint, this is where we go to create / update Attribute Profiles, but in this case, because we’re hitting it for the first time with this ID, we’re creating a new entry called “ATTR_Nick_Key_Value_Example“, this will match any Contexts (more on them later) where the FilterIDs is a string, where the request Account, is equal to 1234.
Let’s run this against the CGrateS API and take a look at the result:
This tells us we matched the Attribute with the ID ATTR_Nick_Key_Value_Example, and inside Event we can see that ExampleKey was added with value ExampleValue.
Okay, you’re saying, well what was the point of that?
Well, what if as a key in the attributes, we had the password for the SIP account, which we passed to our SIP switch (Kamailio, FreeSWITCH or Asterisk for example), and used that to authenticate?
Now if the CGrateS Agent for your SIP Switch, includes the *attributes flag, and the call is coming from 1234, we’ll get back a key called “SIP_password” with the value “sosecretiputitonthewebsite”, which you can use to auth the SIP account.
We can also return multiple AttributeS, for example, we created two Attributes (ATTR_Nick_Password_Example and ATTR_Nick_Key_Value_Example) which match on the account 1234. This means we’ll get back the SIP Password from ATTR_Nick_Password_Example and the key:value we set in ATTR_Nick_Key_Value_Example:
The order can be controlled by the Weight flag in the attribute, and if you want to stop matching any other AttributeS rules after the current Attribute, you can set the Blocker=True flag when you create/update the Attribute.
Okay, I hear you saying, that’s all well and good, I can add arbitrary key/values to stuff. Here endeth the lesson right?
Well not quite, because we can add key/values, but we can also rewrite variables using AttributeS.
Let’s imagine we’ve got 3 phone numbers (DIDs) associated with an account inside CGrateS, for example’s sake let’s say we have 12340001, 12340002 and 12340003, and we want any calls from these numbers to be billed to a CGrateS account called “NickTest1234”.
Our SIP switch doesn’t need to know anything about “NickTest1234”, just the 3 DIDs it can use to call out from your SIP stack. But to do this, we’d need CGrateS to transform any events from these DIDs to replace the Account value inside CGrateS, with NickTest1234.
In the example code to go with this I’ve put together a simple for loop to add these – You can find the code on Github (link at the bottom).
So with these defined, let’s try and rate something, we’ll add a default Charger, and add an SMS balance, before simulating an SMS where the account is set to 12340003:
#Define default Charger
print(CGRateS_Obj.SendData({"method":"APIerSv1.SetChargerProfile","params":[{"Tenant":"cgrates.org","ID":"DEFAULT","FilterIDs":[],"AttributeIDs":["*none"],"Weight":0}]}))
#Add an SMS Balance
print(CGRateS_Obj.SendData({"method":"ApierV1.SetBalance","params":[{"Tenant":"cgrates.org","Account":"Nick_Test_123","BalanceType":"*sms","Categories":"*any","Balance":{"ID":"SMS_Balance_1","Value":"100","Weight":25}}],"id":13}))
import uuid
import datetime
now = datetime.datetime.now()
result = CGRateS_Obj.SendData({
"method": "CDRsV2.ProcessExternalCDR",
"params": [
{
"OriginID": str(uuid.uuid1()),
"ToR": "*sms",
"RequestType": "*pseudoprepaid",
"AnswerTime": now.strftime("%Y-%m-%d %H:%M:%S"),
"SetupTime": now.strftime("%Y-%m-%d %H:%M:%S"),
"Tenant": "cgrates.org",
#This is going to be transformed to Nick_Test_123 by Attributes
"Account": "12340003",
"Usage": "1",
}
]
})
pprint.pprint(result)
Right, so all going well, here’s what you should see in the CDRs table:
Bingo! Despite the fact the Account in the ProcessExternalCDR was set to 12340003, and had no mention of “NickTest1234”, CGrateS transformed it to NickTest1234.
How did that happen? Well, inside our cgrates.json file we have set the cdrs and chargers modules to have a link to Attributes, which means that when we call CDRs or Chargers modules via the API, these will in turn bounce the data through AttributesS for any transformations.
This means we don’t need to run AttributeSv1.ProcessEvent ourselves, when we call CDRsV2.ProcessExternalCDR, the CDRs module will call AttributeSv1.ProcessEvent for us.
We can actually see this happening, using ngrep, which as you work more with CGrateS, is a tool you’ll get very familiar with, let’s take a peek:
sudo ngrep -t -W byline port 2012 -d lo
Now if we run the CDRsV2.ProcessExternalCDR again, we’ll see the CDRs module has called Attributes for us:
Boom, there it is, same as we ran, but it’s being handled by CGrateS for us.
If you look carefully you’ll see the context in the API request is set to “*cdrs”, this means the CDRs module is calling Attributes.
When we define each of our Attributes, as we did earlier in the post, we can set what contexts they are valid in, for example we may want to apply the transformation when called by CDRs, but not other modules, you can restrict that when you define the Attribute by setting “Contexts”: [“*cdrs”].
Okay, so we’ve done some account replacement, what else can we do?
Well, let’s look at some other use cases,
Here in Australia we’ve got a few valid dialing formats, you could dial E.164 format (Numbers look like: +61212341234), 0NSN format (Numbers look like: 02 1234 1234) or NSN format (Numbers look like: 1234 1234 assuming you’re in the 03 area code yourself). If we want to define all our Destinations in E.164 format, we’ll need to to normalise the format using AttributeS, so the numbers always come as E.164.
And then under AttributeS we’ve defined a rule to replace anything matching the 0NSN regex, to strip the first digit and append a 61, to put it in E.164 format, and in SN format as the second entry.
And there you have it folks; our number format standardized.
We can combo / cascade AttributeS rules together, with the aid of the Weight and Blocker flags in the API.
Let’s imagine the 61212341234 number has been ported from Operator1 to Operator2, and the Destinations we’ve defined in CGrateS for this prefix are currently set to DST_Operator1. But because this number has been ported we should use DST_Operator2, so we charge the Operator2, as this number has been ported.
This means we don’t need to duplicate destination definitions to show this number has been ported, as this will be updated as the call gets rated, so we just assign the Attribute to each ported number.
So let’s match where the Subject of the call is 61212341234 (even though we’re going to input the Subject as 12341234), and rewrite the Destination attribute to DST_Operator2:
From the results we can see we matched two AttributeS rules, the first, ATTR_0NSN_to_E164_02_Area_Code reformatted the Subject of the call from 12341234 to 61212341234, then the updated Subject was passed through to ATTR_Ported_61212341234, which updated the Destination attribute to DST_Operator2.
If you’re having issues, make sure you have loaded the config file, are running the latest version, and if in doubt (and not on a production system), this script will clear all the data for you so you can rule out anything interfering.
In our last post we looked at Actions and ActionPlans, and one of the really funky things we can do is setting ActionPlans to trigger on a time schedule or setting ActionTriggers to trigger on an event.
We’re going to build on the examples we had on the last post, so we’ll assume your code is up to the point where we’ve added a Signup Bonus to an account, using an ActionPlan we assigned when creating the account.
In this post, we’re going to create an action that charges $6, called “Action_Monthly_Charge“, and tie it to an ActionPlan called “ActionPlan_Monthly_Charge“, but to demo how this works rather than charging this Monthly, we’re going to charge it every minute.
Then with our balances ticking down, we’ll set up an ActionTrigger to trigger when the balance drops below $95, and alert us.
Defining the Monthly Charge Action
The Action for the Monthly charge will look much like the other actions we’ve defined, except the Identifier is *debitso we know we’re deducting from the balance, and we’ll log to the CDRs table too:
Next we’ll need to wrap this up into an ActionPlan, this is where some of the magic happens. Inside the action plan we can set a once off time, or a recurring time, kinda like Cron.
We’re setting the time to *every_minute so things will happen quickly while we watch, this action will get triggered every 60 seconds. In real life of course, for a Monthly charge, we’d want to trigger this Action monthly, so we’d set this value to *monthly. If we wanted this to charge on the 2nd of the month we’d set the MonthDays to “2”, etc, etc.
If you think the accounts will start getting debited every 60 seconds after applying this, you’d be wrong, we need to associate this ActionPlan with an Account first, this is how we control which accounts get which ActionPlans tied to them, to do this we’ll use the SetAccout API again we’ve been using to create accounts:
Well, for starters the ActionPlan named “ActionPlan_Signup_Bonus” is going to be triggered, as in the ActionPlan it’s Timing is set to *asap, so CGrateS will apply the corresponding Action (“Action_Add_Signup_Bonus“) right away, which will credit the account $99.
But a minute after that, we’ll trigger the ActionPlan named “ActionPlan_Monthly_Charge”, as the timing for this is set to *every_minute, when the Action “Action_Monthly_Charge” is triggered, it’s going to be deducting $6 from the balance.
We can check this by using the GetAccount API:
# Get Account Info
pprint.pprint(CGRateS_Obj.SendData({'method': 'ApierV2.GetAccount', 'params': [
{"Tenant": "cgrates.org", "Account": str(Account)}]}))
You should see a balance of $99 to start with, and then after 60 seconds, it should be down to $93, and so on.
Triggering Actions based on Balances with ActionTriggers
Okay, so we’ve set up recurring charges, now let’s get notified if the balance drops below $95, we’ll start, like we have before, with defining an Action, this will log to the CDRs table, HTTP post and write to syslog:
Now we’ll define an ActionTrigger to check if the balance is below $95 and trigger our newly created Action (“Action_HTTP_Notify_95“) when that condition is met:
We’ve defined the ThresholdType of *min_balance, but we could equally set this to ThresholdType to *max_balance, *balance_expired or trigger when a certain Counter has been triggered enough times.
Adding an ActionTrigger to an Account
Again, like the ActionPlan we created before, before the ActionTrigger we just created will be used, we need to associate it with an Account, for this we’ll use the AddAccountActionTriggers API, specify the Account and the ActionTriggerID for the ActionTrigger we just created.
If we run this all together, creating the account with the “ActionPlan_Signup_Bonus” will give the account a $99 Balance. But after 60 seconds, “ActionPlan_Monthly_Charge” will kick in, and every 60 seconds after that, at which point the balance will get to below $95 when CGrateS will trigger the ActionTrigger “ActionTrigger_95_Remaining” and get the HTTP POST to the HTTP endpoint and log entry:
We can check on this using the ApierV2.GetAccount method, where we’ll see the ActionTrigger we just defined.
Checking out the LastExecutionTime we can see if the ActionTrigger been triggered or not.
So using this technique, we can notify a customer when they’ve used a certain amount of their balance, but we can lock out Accounts who have spent more than their allocated spend limit by setting an Action that suspends the Account once it reaches a certain level. We notify customers when balance expires, or if a certain number of counters has been triggered.
In our last post we added a series of different balances to an account, these were actions we took via the API specifically to add a balance.
But there’s a lot more actions we may want to do beyond just adding balance.
CGrateS has the concept of “Actions” which are, as the name suggests, things we want to do to the system.
Some example Actions would be:
Adding / Deducting / Resetting a balance
Adding a CDR log
Enable/Disable an account
Sending HTTP POST request or email notification
Deleting / suspending account
Transferring balances
We can run these actions on a timed basis, or when an event is triggered, and group Actions together to run multiple actions via an ActionTrigger, this means we can trigger these Actions, not just by sending an API request, but based on the state of the subscriber / account.
Let’s look at some examples,
We can define an Action named “Action_Monthly_Fee” to debit $12 from the monetary balance of an account, and add a CDR with the name “Monthly Account Fee” when it does so. We can use ActionTriggers to run this every month on the account automatically.
We can define an Action named “Usage_Warning_10GB” to send an email to the Account owner to inform them they’ve used 10GB of usage, and use ActionTriggers to send this when the customer has used 10GB of their *data balance.
Let’s start basic; to sweeten the deal for new Accounts, we’ll give them $99 of balance to use in the first month they have the service. Rather than hitting the AddBalance API, we’ll define an Action named “Action_Add_Signup_Bonus” to credit $99 of monetary balance to an account.
If you go back to our last post, you should know what we’d need to do to add this balance manually with the AddBalance API, but let’s look at how we can create the same balance add functionality using Actions:
#Add a Signup Bonus of $99 to the account with type *monetary expiring a month after it's added
Action_Signup_Bonus = {
"id": "0",
"method": "ApierV1.SetActions",
"params": [
{
"ActionsId": "Action_Add_Signup_Bonus",
"Actions": [
{
"Identifier": "*topup","BalanceId": "Balance_Signup_Bonus",
"BalanceUuid": "",
"BalanceType": "*monetary",
"Directions": "*out",
"Units": 99,
"ExpiryTime": "*month",
"Filter": "",
"TimingTags": "",
"DestinationIds": "",
"RatingSubject": "",
"Categories": "",
"SharedGroups": "",
"BalanceWeight": 1200,
"ExtraParameters": "",
"BalanceBlocker": "false",
"BalanceDisabled": "false",
"Weight": 10
}
]}]}
pprint.pprint(CGRateS_Obj.SendData(Action_Signup_Bonus))
Alright, this should look pretty familiar if you’ve just come from Account Balances. You’ll notice we’re no longer calling, SetBalance, we’re now calling SetActions, to create the ActionsId with the name “Action_Add_Signup_Bonus“. In “Action_Add_Signup_Bonus” we’ve got an actions we’ll do when “Action_Add_Signup_Bonus” is called. We can define multiple actions, but for now we’ve only got one action defined, which has the Identifier (which defines what the action does) set to *topup to add balance. As you probably guessed, we’re triggering a top up, and setting the BalanceId, BalanceType, Units, ExpiryTime and BalanceWeight just as we would using SetBalance to add a balance.
So how do we use the Action we just created? Well, there’s a lot of options, but let’s start with the most basic – Via the API:
Boom, now we’ll get a CDR created when the Action is triggered.
But let’s push this a bit more and add some more steps in the Action:
As well as adding balance and putting in a CDR to record what we did, let’s also send a notification to our customer via an HTTP API (BYO customer push notification system) and log to Syslog what’s going on.
So what have we done here? We’ve made an ActionPlan named “Action_Add_Signup_Bonus”, which, when associated with an account, will run the Action “Action_Add_Signup_Bonus” as soon as it’s tied to the account, thanks to the Time “*asap“.
Now if we create or update an Account using the SetAccount method, we can set the ActionPlanIds to reference our “ActionPlan_Signup_Bonus” and it’ll be triggered straight away.
Now if we were to run a GetAccount API call, we’ll see the Account balance assigned that was created by the action Action_Add_Signup_Bonus which was triggered by ActionPlan assigned to the account:
But here’s where it gets interesting, in the ActionPlan we just defined the Time was set to “*asap“, which means the Action is triggered as soon as it was assigned to the account, but if we set the Time value to “*monthly“, the Action would get triggered every month, or *every_minute to trigger every minute, or *month_end to trigger at the end of every month.
I’m trying to keep these posts shorter as there’s a lot to cover. Stick around for our next post, we’ll look at some more ActionTriggers to keep decreasing the balance of the account, and setting up ActionTriggers to send a notification to the customer to tell them when their balance is getting low, or any other event based Action you can think of!
So far we’ve used CGrateS to rate a basic CDR and get a cost for it, but in the real world, we’d usually associate the cost with an account, which would represent a business or a person, who will ultimately be charged for using the service.
Running this onto the API should create an account named “Nick_Test_123”, but let’s confirm that’s the case:
#Print the Account Information
pprint.pprint(CGRateS_Obj.SendData({'method':'ApierV2.GetAccount','params':[{"Tenant":"cgrates.org","Account": "Nick_Test_123"}]}))
Running this will give us the information about the account we just created:
There’s not really much to see on our account at this stage, other than the UpdateTime, there’s nothing really going on, we don’t have any Balances.
Adding Balance for Voice
Accounts exist for spending, so let’s add a balance to this account to send from.
We’ll use the SetBalance API to create a new balance with 5 minutes of talk time, that we can use for making a call, and talking, for (you guessed it) – 5 minutes, so and we’ll use the balance “5_minute_voice_balance” that we’ll create:
#Add a balance to the account with type *voice with 5 minutes of Talk Time
Create_Voice_Balance_JSON = {
"method": "ApierV1.SetBalance",
"params": [
{
"Tenant": "cgrates.org",
"Account": "Nick_Test_123",
"BalanceType": "*voice",
"Categories": "*any",
"Balance": {
"ID": "5_minute_voice_balance",
"Value": "5m",
"Weight": 25
}
}
]
}
print(CGRateS_Obj.SendData(Create_Voice_Balance_JSON))
Now if we run the GetAccount API command again, we should see the new balance we just created:
#Print the Account Information
pprint.pprint(CGRateS_Obj.SendData({'method':'ApierV2.GetAccount','params':[{"Tenant":"cgrates.org","Account": "Nick_Test_123"}]}))
So now we’ve got a new balance named ‘5_minute_voice_balance‘:
The type is *voice, because this balance is storing talk time
The weight of this balance is 25, this means this balance should take priority over any balances with a lower value than 25 (that’s right, we can (and will) do tiered balances)
The value is 300000000000 nanoseconds, which equates to 5 minutes (yes, that’s the correct number of zeros)
Okay, but Nick_Test_123 probably wants to make some calls, so let’s generate a 2.5 minute call event and check out what happens.
Alright, now we’ve got a call event, let’s call the GetAccount API again to check the balance:
#Print the Account Information
pprint.pprint(CGRateS_Obj.SendData({'method':'ApierV2.GetAccount','params':[{"Tenant":"cgrates.org","Account": "Nick_Test_123"}]}))
And there you have it, we’ve used 150 seconds of our 300 second (5 minutes) of talk time in this balance, leaving with us 150000000000 nanoseconds (150 seconds) remaining!
And with that progress, now is a great time to pause and talk about some theory that’s really important to grasp!
Balance Types
But *voice is just one balance type – We can support multiple balance types; we’ve just given a balance of *voice for talk time, but we could also give a credit to the balance for *sms or *data, you name it (*generic) and cash (*monetary) and we can have multiple separate balances for each.
This means we can have one account with something like:
100 minutes of Local / National Calls (Expires at the end of the month)
40 minutes of Mobile Calls (Expires 24 hours after it’s been created)
80 minutes of Mobile Calls (During “Happy Hour” from 6 to 7pm)
50 minutes of International Calls (Expires in 30 days)
And not just voice balance, the same account could also have:
1GB of Data usage
50 SMS to on-net destinations
$200 of Cash (expiring never)
Phew! That’s a lot of balances, but we can do it all through CGrateS!
What Balance to Use
So if we’ve got a stack of balances defined, how does CGrateS know what balance to use?
Firstly CGrateS is going to evaluate the BalanceType, this is set on events, so if we get an event for *data CGrateS will check out the balances available for *data, and evaluate the balances by Weight, with the highest weight evaluated first. If we get to the end of all the available balances for that BalanceType, CGrateS then evaluates *generic and then *monetary balances, again, ordered by Weight.
We can set what balance gets used based on the Destination; using DestinationIDs we can filter the Balance to only apply for calls to Local/National numbers, so a call to an International destination won’t use that balance.
We can also set an Expiry on the Balance, for example we can give a customer 30 days to use the balance, after which it expires and can’t be used, likewise we can set Timings so enable scenarios like a “Happy hour” with extra calls between 6pm and 7pm.
When we define a balance we can also set the Blocker flag to True, if this is set, it means CGrateS will not look evaluate any balances after reaching that balance.
Adding a Balance for Local / National & Mobile Calls
Let’s jump back into the practice, and define two new Balances; one for Local/National calls, and another for Mobile calls. But first we’ll need to know what destinations are mobiles and what are local/national (fixed). We’ve covered setting Destinations previously, so let’s set up the Destinations:
CGRateS_Obj.SendData({'method':'ApierV2.SetTPDestination','params':[{"TPid":"cgrates.org","ID":"Dest_AU_Fixed","Prefixes":["612", "613", "617", "618"]}]})
CGRateS_Obj.SendData({'method':'ApierV2.SetTPDestination','params':[{"TPid":"cgrates.org","ID":"Dest_AU_Mobile","Prefixes":["614"]}]})
CGRateS_Obj.SendData({'method':'ApierV2.SetTPDestination','params':[{"TPid":"cgrates.org","ID":"Dest_AU_TollFree","Prefixes":["6113", "6118"]}]})
#Load TariffPlan we just defiend from StorDB to DataDB
print(CGRateS_Obj.SendData({"method":"APIerSv1.LoadTariffPlanFromStorDb","params":[{"TPid":"cgrates.org","DryRun":False,"Validate":True,"APIOpts":None,"Caching":None}],"id":0}))
Alright, now let’s add a balance for our local/national (fixed) calls.
To do this, we’ll add two new balances, but we’ll need to differentiate this from the 5_minute_voice_balance we created earlier, and to achieve this we weill:
Set a higher Weight value than we have set on 5_minute_voice_balance (25) so this balance will get consumed before 5_minute_voice_balance does
Set the DestinationIDs to match the destinations (Dest_AU_Mobile for Mobile and Dest_AU_Fixed for Local/National) we want the balance to apply to
ProTip: When you we create our Balance we can set what Destinations we want to use this balance for, if you want to specify multiple balances, we can do it by setting the Balance names as a string delimited by semicolons, like “DestinationIDs”: “Dest_AU_Fixed;Dest_AU_Mobile;Dest_AU_TollFree”
We’ll also set a balance expiry, which we’ll cover shortly, but now let’s define out 100 minutes for Local/National expiring at the end of the month:
#Add a balance to the account with type *voice with 100 minutes of talk time to Local / National Destinations expiring at the end of the month
Create_Local_National_Voice_Balance_JSON = {
"method": "ApierV1.SetBalance",
"params": [
{
"Tenant": "cgrates.org",
"Account": "Nick_Test_123",
"BalanceType": "*voice",
"Categories": "*any",
"Balance": {
"ID": "Local_National_100_minutes_voice_balance",
"Value": "100m",
"ExpiryTime": "*month_end",
"Weight": 60,
"DestinationIDs": "Dest_AU_Fixed",
}
}
]
}
print(CGRateS_Obj.SendData(Create_Local_National_Voice_Balance_JSON))
We’ll also add our 24 hours to use to use 40 minutes of talk to mobiles, and a GetAccount to check the result:
#Add a balance to the account with type *voice with 40 minutes of talk time to Mobile Destinations expiring in 24 hours
Create_Mobile_Voice_Balance_JSON = {
"method": "ApierV1.SetBalance",
"params": [
{
"Tenant": "cgrates.org",
"Account": "Nick_Test_123",
"BalanceType": "*voice",
"Categories": "*any",
"Balance": {
"ID": "Mobile_40_minutes_voice_balance",
"Value": "40m",
"ExpiryTime": "*daily",
"Weight": 60,
"DestinationIDs": "Dest_AU_Mobile",
}
}
]
}
print(CGRateS_Obj.SendData(Create_Mobile_Voice_Balance_JSON))
# Get Account Info Again
pprint.pprint(CGRateS_Obj.SendData({"method": "ApierV2.GetAccount", "params": [
{"Tenant": "cgrates.org", "Account": "Nick_Test_123"}]}))
Notice in the API in the expiry I put *daily and *month_end, but in the output it’s got a real date and time (I wrote this 14/10/23 around 07:00 UTC, hence why those dates are what they are). I could have specified the date and time in the API of a specific time I wanted the balance to expire (You can too, just replace “*daily” with “2024-01-01T00:00:00Z” for example), but that’s a pain in the butt, especially considering most of the time these values will be something common. The *month_end is a special “meta” value, there’s a heap of these that allow us to do things like “current time + 20 minutes” (+20m), this time next month (*monthly), “this time tomorrow” (*daily), or “this time next week” (+168h) – You can find the full list of special dates here.
From a product perspective, setting an expiry on balances means we can set credit to expire 2 years after the subscriber tops up, but the same logic can be used so a subscriber could purchase a 7 day addon pack, that expires in 7 days, or a monthly plan can automatically expire in 30 days.
Now if we call the ProcessExternalCDR API again with a call to a Mobile and a Fixed number, we’ll see the respective balances get deducted.
#Generate a new call event for a 2.5 minute (150 second) call to a mobile number
Process_External_CDR_JSON = {
"method": "CDRsV2.ProcessExternalCDR",
"params": [
{
"OriginID": str(uuid.uuid1()),
"ToR": "*voice",
"RequestType": "*pseudoprepaid",
"AnswerTime": now.strftime("%Y-%m-%d %H:%M:%S"),
"SetupTime": now.strftime("%Y-%m-%d %H:%M:%S"),
"Subject": "61412341234",
"Destination": "61412341234",
"Tenant": "cgrates.org",
"Account": "Nick_Test_123",
"Usage": "30s",
}
]
}
print(CGRateS_Obj.SendData(Process_External_CDR_JSON))
#Generate a new call event for a 2.5 minute (150 second) call to a fixed line local/national number
Process_External_CDR_JSON = {
"method": "CDRsV2.ProcessExternalCDR",
"params": [
{
"OriginID": str(uuid.uuid1()),
"ToR": "*voice",
"RequestType": "*pseudoprepaid",
"AnswerTime": now.strftime("%Y-%m-%d %H:%M:%S"),
"SetupTime": now.strftime("%Y-%m-%d %H:%M:%S"),
"Subject": "61212341234",
"Destination": "61212341234",
"Tenant": "cgrates.org",
"Account": "Nick_Test_123",
"Usage": "30s",
}
]
}
print(CGRateS_Obj.SendData(Process_External_CDR_JSON))
# Get Account Info Again
pprint.pprint(CGRateS_Obj.SendData({"method": "ApierV2.GetAccount", "params": [
{"Tenant": "cgrates.org", "Account": "Nick_Test_123"}]}))
We should see the minutes reduced by 30 seconds for our Local_National_100_minutes_voice_balance and Mobile_40_minutes_voice_balance balances, while our 5_minute_voice_balance hasn’t been touched.
One last thing we’ll try before we end, our Mobile_40_minutes_voice_balance has still got 39.5 minutes left, and our 5_minute_voice_balance has still got minutes remaining, so if we try and make a call that’s 2450 seconds (~41 minutes), we should consume all the remaining minutes in Mobile_40_minutes_voice_balance and the go onto consume the remaining 1 minute out of 5_minute_voice_balance.
Let’s test this theory!
#Generate a new call event for a 42 minute call to a mobile to use all of our Mobile_40_minutes_voice_balance and start consuming 5_minute_voice_balance
Process_External_CDR_JSON = {
"method": "CDRsV2.ProcessExternalCDR",
"params": [
{
"OriginID": str(uuid.uuid1()),
"ToR": "*voice",
"RequestType": "*pseudoprepaid",
"AnswerTime": now.strftime("%Y-%m-%d %H:%M:%S"),
"SetupTime": now.strftime("%Y-%m-%d %H:%M:%S"),
"Subject": "61412341234",
"Destination": "61412341234",
"Tenant": "cgrates.org",
"Account": "Nick_Test_123",
"Usage": "2450s",
}
]
}
print(CGRateS_Obj.SendData(Process_External_CDR_JSON))
# Get Account Info Again
pprint.pprint(CGRateS_Obj.SendData({"method": "ApierV2.GetAccount", "params": [
{"Tenant": "cgrates.org", "Account": "Nick_Test_123"}]}))
Well, there’s another concept I haven’t introduced yet, and that’s ChargerS, this is a concept / component we’ll dig into deeper for derived charging, but for now just know we need to add a ChargerS rule in order to get CDRs rated:
Well, if you’ve got CDR storage in StoreDB enabled (And you probably do if you’ve been following up until this point), then the answer is a MySQL table, and we can retrive the data with:
sudo mysql cgrates -e "select * from cdrs \G"
For those of you with a bit of MySQL experience under your belt, you’d be able to envisage using the SUM function to total a monthly bill for a customer from this.
Of course we can add CDRs via the API, and you probably already guessed this, but we can retrive CDRs via the API as well, filtering on the key criteria:
This would be useful for generating an invoice or populating recent calls for a customer portal.
Maybe creating rated CDRs and sticking them into a database is exactly what you’re looking to achieve in CGrateS – And if so, great, this is where you can stop – but for many use cases, there’s a want for an automated solution – For your platform to automatically integrate with CGrateS.
If you’ve got an Asterisk/FreeSWITCH/Kamailio or OpenSIPs based platform, then you can integrate CGrateS directly into your platform to add the CDRs automatically, as well as access features like prepaid credit control, concurrent call limits, etc, etc. The process is a little different on each of these platforms, but ultimately under the hood, all of these platforms have some middleware that generates the same API calls we just ran to create the CDR.
So far this tutorial has been heavy on teaching the API, because that’s what CGrateS ultimately is – An API service.
Our platforms like Asterisk and Kamailio with the CGrateS plugins are just CGrateS API clients, and so once we understand how to use and interact with the API it’s a breeze to plug in the module for your platform to generate the API calls to CGrateS required to integrate.
In our last post we introduced the CGrateS API and we used it to add Rates, Destinations and define DestinationRates.
In this post, we’ll create the RatingPlan that references the DestinationRate we just defined, and the RatingProfile that references the RatingPlan, and then, as the cherry on top – We’ll rate some calls.
For anyone looking at the above diagram for the first time, you might be inclined to ask why what is the purpose of having all these layers?
This layered architecture allows all sorts of flexibility, that we wouldn’t otherwise have, for example, we can have multiple RatingPlans defined for the same Destinations, to allow us to have different Products defined, with different destinations and costs.
Likewise we can have multiple RatingProfiles assigned for the same destinations to allow us to generate multiple CDRs for each call, for example a CDR to bill the customer with and a CDR with our wholesale cost.
All this flexibility is enabled by the layered architecture.
Define RatingPlan
Picking up where we left off having just defined the DestinationRate, we’ll need to create a RatingPlan and link it to the DestinationRate, so let’s check on our DestinationRates:
From the output we can see we’ve got the DestinationRate defined, there’s a lot of info returned (I’ve left out most of it), but you can see the Destination, and the Rate associated with it is returned:
So after confirming that our DestinationRates are there, we’ll create a RatingPlan to reference it, for this we’ll use the APIerSv1.SetTPRatingPlan API call.
In our basic example, this really just glues the DestinationRate_AU object to RatingPlan_VoiceCalls.
It’s worth noting that you can use a RatingPlan to link to multiple DestinationRates, for example, we might want to have a different RatingPlan for each region / country, we can do that pretty easily too, in the below example I’ve referenced other Destination Rates (You’d go about defining the DestinationRates for these other destinations / rates the same way as we did in the last example).
One last step before we can test this all end-to-end, and that’s to link the RatingPlan we just defined with a RatingProfile.
StorDB & DataDB
Psych! Before we do that, I’m going to subject you to learning about backends for a while.
So far we’ve skirted around CGrateS architecture, but this is something we need to know for now.
To keep everything fast, a lot of data is cached in what is called a DataDB (if you’ve followed since part 1, then your DataDB is Redis, but there are other options).
To keep everything together, databases are used for storage, called StorDB (in our case we are using MySQL, but again, we can have other options) but calls to this database are minimal to keep the system fast.
If you’re an astute reader, you may have noticed many of our API calls have TP in method name, if the API call has TP in the name, it is storing it in the StoreDB, if it doesn’t, it means it’s storing it only in DataDB.
Why does this matter? Well, let’s look a little more closely and it will become clear:
ApierV1.SetRatingProfile will set the data only in DataDB (Redis), because it’s in the DataDB the change will take effect immediately.
ApierV1.SetTPRatingProfile will set the data only in StoreDB (MySQL), it will not take effect until it is copied from the database (StoreDB) to the cache (DataDB).
After we define the RatingPlan, we need to run this command prior to creating the RatingProfile, so it has something to reference, so we’ll do that by adding:
The last piece of the puzzle to define is the RatingProfile.
We define a few key things in the rating profile:
The Tenant – CGrateS is multitenant out of the box (in our case we’ve used tenant named “cgrates.org“, but you could have different tenants for different customers).
The Category – As we covered in the first post, CGrateS can bill voice calls, SMS, MMS & Data consumption, in this scenario we’re billing calls so we have the value set to *call, but we’ve got many other options. We can use Category to link what RatingPlan is used, for example we might want to offer a premium voice service with guaranteed CLI rates, using a different RatingPlan that charges more per call, or maybe we’re doing mobile and we want a different RatingPlan for use when Roaming, we can use Category to switch that.
The Subject – This is loosely the Source / Calling Party; in our case we’re using a wildcard value *any which will match any Subject
The RatingPlanActivations list the RatingPlanIds of the RatingPlans this RatingProfile uses
So let’s take a look at what we’d run to add this:
Okay, so at this point, all going well, we should have some data loaded, we’ve gone through all those steps to load this data, so now let’s simulate a call to a Mobile Number (22c per minute) for 123 seconds.
We cheated a fair bit, to show something that worked, but it’s not something you’d probably want to use in real life, loading static CSV files gets us off the ground, but in reality we don’t want to manage a system through CSV files.
Instead, we’d want to use an API.
Fair warning – There is some familiarity expected with JSON and RESTful APIs required, we’ll use Python3 for our examples, but you can use any programing language you’re comfortable with, or even CURL commands.
So we’re going to start by clearing out all the data we setup in CGrateS using the cgr-loader tool from those imported CSVs:
redis-cli flushall
sudo mysql -Nse 'show tables' cgrates | while read table; do sudo mysql -e "truncate table $table" cgrates; done
cgr-migrator -exec=*set_versions -stordb_passwd=CGRateS.org
sudo systemctl restart cgrates
So what have we just done? Well, we’ve just cleared all the data in CGrateS. We’re starting with a blank slate.
In this post, we’re going to define some Destinations, some Rates to charge and then some DestinationRates to link each Destination to a Rate.
But this time we’ll be doing this through the CGrateS API.
Introduction to the CGrateS API
CGrateS is all API driven – so let’s get acquainted with this API.
I’ve written a simple Python wrapper you can find here that will make talking to CGRateS a little easier, so let’s take it for a spin and get the Destinations that are loaded into our system:
import cgrateshttpapi
CGRateS_Obj = cgrateshttpapi.CGRateS('172.16.41.133', 2080) #Replace this IP with the IP Address of your CGrateS instance...
destinations = CGRateS_Obj.SendData({'method':'ApierV1.GetTPDestinationIDs','params':[{"TPid":"cgrates.org"}]})['result']
#Pretty print the result:
print("Destinations: ")
pprint.pprint(destinations)
All going well you’ll see something like this back:
Initializing with host 172.16.41.133 on port 2080
Sending Request with Body:
{'method': 'ApierV2.Ping', 'params': [{'Tenant': 'cgrates.org'}]}
Sending Request with Body:
{'method': 'ApierV2.GetTPDestinationIDs', 'params': [{"TPid":"cgrates.org"}]}
Destinations from CGRates: []
So what did we just do? Well, we sent a JSON formatted string to the CGRateS API at 172.16.41.133 on port 2080 – You’ll obviously need to change this to the IP of your CGrateS instance.
In the JSON body we sent we asked for all the Destinations using the ApierV1.GetTPDestinationIDs method, for the TPid ‘cgrates.org’,
And it looks like no destinations were sent back, so let’s change that!
Note: There’s API Version 1 and API Version 2, not all functions exist in both (at least not in the docs) so you have to use a mix.
Adding Destinations via the API
So now we’ve got our API setup, let’s see if we can add a destination!
To add a destination, we’ll need to go to the API guide and find the API call to add a destination – in our case the API call is ApierV2.SetTPDestination and will look like this:
So we’re creating a Destination named Dest_AU_Mobile and Prefix 614 will match this destination.
Note: I like to prefix all my Destinations with Dest_, all my rates with Rate_, etc, so it makes it easy when reading what’s going on what object is what, you may wish to do the same!
So we’ll use the Python code we had before to list the destinations, but this time, we’ll use the ApierV2.SetTPDestination API call to add a destination before listing them, let’s take a look:
If we post this to the CGR engine, we’ll create a rate, named Rate_AU_Mobile_Rate_1 that bills 22 cents per minute, charged every 60 seconds.
Let’s add a few rates:
CGRateS_Obj.SendData({"method":"ApierV1.SetTPRate","params":[{"ID":"Rate_AU_Mobile_Rate_1","TPid":"cgrates.org","RateSlots":[{"ConnectFee":0,"Rate":22,"RateUnit":"60s","RateIncrement":"60s","GroupIntervalStart":"0s"}]}],"id":1})
CGRateS_Obj.SendData({"method":"ApierV1.SetTPRate","params":[{"ID":"Rate_AU_Fixed_Rate_1","TPid":"cgrates.org","RateSlots":[{"ConnectFee":0,"Rate":14,"RateUnit":"60s","RateIncrement":"60s","GroupIntervalStart":"0s"}]}],"id":1})
CGRateS_Obj.SendData({"method":"ApierV1.SetTPRate","params":[{"ID":"Rate_AU_Toll_Free_Rate_1","TPid":"cgrates.org","RateSlots":[{"ConnectFee":25,"Rate":0,"RateUnit":"60s","RateIncrement":"60s","GroupIntervalStart":"0s"}]}],"id":1})
TPRateIds = CGRateS_Obj.SendData({"method":"ApierV1.GetTPRateIds","params":[{"TPid":"cgrates.org"}]})['result']
print(TPRateIds)
for TPRateId in TPRateIds:
print("\tRate: " + str(TPRateId))
All going well, when you add the above, we’ll have added 3 new rates:
Rate Name
Cost
Rate_AU_Fixed_Rate_1
14c per minute charged every 60s
Rate_AU_Mobile_Rate_1
22c per minute charged every 60s
Rate_AU_Toll_Free_Rate_1
25c connection, untimed
Rates we just created
Linking Rates to Destinations
So now with Destinations defined, and Rates defined, it’s time to link these two together!
Destination Rates link our Destinations and Route rates, this decoupling means that we can have one Rate shared by multiple Destinations if we wanted, and makes things very flexible.
For this example, we’re going to map the Destinations to rates like this:
All going well, you’ll see the new DestinationRate we added.
Here’s a good chance to show how we can add multiple bits of data in one API call, we can tweak the ApierV1.SetTPDestinationRate method and include all the DestinationRates we need in one API call:
In our next post, we’ll keep working our way up this diagram, by creating RatingPlans and RatingProfiles to reference the DestinationRate we just created.
In our last post we talked about setting rates in CGrates and testing them out, but what’s the point in learning a charging system without services to charge?
This post focuses on intergrating FreeSWITCH and CGrates, other posts cover integrating Asterisk and CGrates, Kamailio and CGrates and Diameter and CGrates.
Future posts in this series will focus on the CGrates side, but this post will be a bit of a sidebar to get our FreeSWITCH environment connected to CGrates so we can put all our rating and charging logic into FreeSWITCH.
CGrates interacts with FreeSWITCH via the Event-Socket-Language in FreeSWITCH, which I’ve written about before, in essence when enabled, CGrates is able to make decisions regarding if a call should proceed or not, monitor currently up calls, and terminate calls when a subscriber has used their allocated balance.
Adding ESL Binding Support in FreeSWITCH
The configuration for CGrates is defined through the cgrates.json file in /etc/cgrates on your rating server.
By default, FreeSWITCH’s event socket only listens on localhost, as it is a pretty huge security flaw to open it to the world, but in order for our CGrates server to be able to access we’ll need to bind it to an IP Address assigned to the FreeSWITCH server so we can reach it from elsewhere on the network.
You may want to have CGrates installed on a different machine to your FreeSWITCH instance, or you may want to have multiple FreeSWITCH instances all getting credit control from CGrates.
Well, inside the cgrates.json config file, is where we populate the ESL connection details so CGrates can connect to FreeSWITCH.
Next we’ll need to tag the extensions we want to charge,
In order to do this we’ll need to set the type of the account (Ie. Prepaid, Postpaid, etc), and the flags to apply, which dictate which of the modules we’re going to use inside CGrateS.
FreeSWITCH won’t actually parse this info, it’s just passed to CGrateS.
And that’s pretty much it, when you restart FreeSWITCH and CGrates you should see in the CGrates log that it is connected to your FreeSWITCH instance, and when you make a call, FreeSWITCH will authorize it through CGrates.
We’ll get back into the nitty gritty about setting up CGrates in a future post, and cover setting up integration like this with other Platforms (Kamailio / Asterisk) and Protocols (Diameter & Radius) in future posts.
So you have a VoIP service and you want to rate the calls to charge your customers?
You’re running a mobile network and you need to meter data used by subscribers?
Need to do least-cost routing?
You want to offer prepaid mobile services?
Want to integrate with Asterisk, Kamailio, FreeSWITCH, Radius, Diameter, Packet Core, IMS, you name it!
Well friends, step right up, because today, we’re talking CGrates!
So before we get started, this isn’t going to be a 5 minute tutorial, I’ve a feeling this may end up a big multipart series like some of the others I’ve done. There is a learning curve here, and we’ll climb it together – but it is a climb.
Installation
Let’s start with a Debian based OS, installation is a doddle:
We’re going to use Redis for the DataDB and MariaDB as the StorDB (More on these concepts later), you should know that other backend options are available, but for keeping things simple we’ll just use these two.
Next we’ll get the database and config setup,
cd /usr/share/cgrates/storage/mysql/
./setup_cgr_db.sh root CGRateS.org localhost
cgr-migrator -exec=*set_versions -stordb_passwd=CGRateS.org
Lastly we’ll clone the config files from the GitHub repo:
In its simplest form, rating is taking a service being provided and calculating the cost for it.
The start of this series will focus on voice calls (With SMS, MMS, Data to come), where the callingparty (The person making the call) pays, so let’s imagine calling a Mobile number (Starting with 614) costs $0.22 per minute.
To perform rating we need to determine the Destination, the Rate to be applied, and the time to charge for.
For our example earlier, a call to a mobile (Any number starting with 614) should be charged at $0.22 per minute. So a 1 minute call will cost $0.22 and a 2 minute long call will cost $0.44, and so on.
We’ll also charge calls to fixed numbers (Prefix 612, 613, 617 and 617) at a flat $0.20 regardless of how long the call goes for.
So let’s start putting this whole thing together.
Introduction to RALs
RALs is the component in CGrates that takes care of Rating and Accounting Logic, and in this post, we’ll be looking at Rating.
The rates have hierarchical structure, which we’ll go into throughout this post. I took my notepad doodle of how everything fits together and digitized it below:
Destinations
Destinations are fairly simple, we’ll set them up in our Destinations.csv file, and it will look something like this:
Each entry has an ID (referred to higher up as the Destination ID), and a prefix.
Also notice that some Prefixes share an ID, for example 612, 613, 617 & 618 are under the Destination ID named “DST_AUS_Fixed”, so a call to any of those prefixes would match DST_AUS_Fixed.
Rates
Rates define the price we charge for a service and are defined by our Rates.csv file.
This is nice and clean, a 1 second call costs $0.25, a 60 second call costs $0.25, and a 61 second call costs $0.50, and so on.
This is the standard billing mechanism for residential services, but it does not pro-rata the call – For example a 1 second call is the same cost as a 59 second call ($0.25), and only if you tick over to 61 seconds does it get charged again (Total of $0.50).
Per Second Billing
If you’re doing a high volume of calls, paying for a 3 second long call where someone’s voicemail answers the call and was hung up, may seem a bit steep to pay the same for that as you would pay for 59 seconds of talk time.
Instead Per Second Billing is more common for high volume customers or carrier-interconnects.
This means the rate still be set at $0.25 per minute, but calculated per second.
So the cost of 60 seconds of call is $0.25, but the cost of 30 second call (half a minute) should cost half of that, so a 30 second call would cost $0.125.
How often we asses the charging is defined by the RateIncrement parameter in the Rate Table.
We could achieve the same outcome another way, by setting the RateIncriment to 1 second, and the dividing the rate per minute by 60, we would get the same outcome, but would be more messy and harder to maintain, but you could think of this as $0.25 per minute, or $0.004166667 per second ($0.25/60 seconds).
Flat Rate Billing
Another option that’s commonly used is to charge a flat rate for the call, so when the call is answered, you’re charged that rate, regardless of the length of the call.
Regardless if the call is for 1 second or 10 hours, the charge is the same.
DestinationID – Refers to the DestinationID defined in the Destinations.csv file
RatesTag – Referes to the Rate ID we defined in Rates.csv
RoundingMethod – Defines if we round up or down
RoundingDecimals – Defines how many decimal places to consider before rounding
MaxCost – The maximum cost this can go up to
MaxCostStrategy – What to do if the Maximum Cost is reached – Either make the rest of the call Free or Disconnect the call
So for each entry we’ll define an ID, reference the Destination and the Rate to be applied, the other parts we’ll leave as boilerplate for now, and presto. We have linked our Destinations to Rates.
Rating Plans
We may want to offer different plans for different customers, with different rates.
DestinationRatesId (As defined in DestinationRates.csv)
TimingTag – References a time profile if used
Weight – Used to determine what precedence to use if multiple matches
So as you may imagine we need to link the DestinationRateIDs we just defined together into a Rating Plan, so that’s what I’ve done in the example above.
Rating Profiles
The last step in our chain is to link Customers / Subscribers to the profiles we’ve just defined.
How you allocate a customer to a particular Rating Plan is up to you, there’s numerous ways to approach it, but for this example we’re going to use one Rating Profile for all callers coming from the “cgrates.org” tenant:
Category is used to define the type of service we’re charging for, in this case it’s a call, but could also be an SMS, Data usage, or a custom definition.
Subject is typically the calling party, we could set this to be the Caller ID, but in this case I’ve used a wildcard “*any”
ActivationTime allows us to define a start time for the Rating Profile, for example if all our rates go up on the 1st of each month, we can update the Plans and add a new entry in the Rating Profile with the new Plans with the start time set
RatingPlanID sets the Rating Plan that is used as we defined in RatingPlans.csv
Loading the Rates into CGrates
At the start we’ll be dealing with CGrates through CSV files we import, this is just one way to interface with CGrates, there’s others we’ll cover in due time.
CGRates has a clever realtime architecture that we won’t go into in any great depth, but in order to load data in from a CSV file there’s a simple handy tool to run the process,
Obviously you’ll need to replace with the folder you cloned from GitHub.
Trying it Out
In order for CGrates to work with Kamailio, FreeSWITCH, Asterisk, Diameter, Radius, and a stack of custom options, for rating calls, it has to have common mechanisms for retrieving this data.
CGrates provides an API for rating calls, that’s used by these platforms, and there’s a tool we can use to emulate the signaling for call being charged, without needing to pickup the phone or integrate a platform into it.
The tenant will need to match those defined in the RatingProfiles.csv, the Subject is the Calling Party identity, in our case we’re using a wildcard match so it doesn’t matter really what it’s set to, the Destination is the destination of the call, AnswerTime is time of the call being answered (pretty self explanatory) and the usage defines how many seconds the call has progressed for.
The output is a JSON string, containing a stack of useful information for us, including the Cost of the call, but also the rates that go into the decision making process so we can see the logic that went into the price.
So have a play with setting up more Destinations, Rates, DestinationRates and RatingPlans, in these CSV files, and in our next post we’ll dig a little deeper… And throw away the CSVs all together!
