I am connected on a VDSL line, not by choice, but here we are. DSL is many things, but consistent it not one of them, so I thought it’d be interesting to graph out the SNR and the line rate of the connection.
This is an NBN FTTN circuit, I run Mikrotiks for the routing, but I have a Draytek Vigor 130 that acts as a dumb modem and connects to the Tik.
Draytek exposes this info via SNMP, but the OIDs / MIBs are not part of the standard Prometheus snmp_exporter, so I’ve added them into snmp_exporter.yaml and restarted the snmp_exporter service.
Recently a Tweet from Dean Bubly got me thinking about how data is charged in cellular:
In the cellular world, subscribers are charged for data from the IP, transport and applications layers; this means you pay for the IP header, you pay for the TCP/UDP header, and you pay for the contents (the cat videos it contains).
This also means if an operator moves mobile subscribers from IPv4 to IPv6, there’s an extra 20 bytes the customer is charged for for every packet sent / received, which the customer is charged for – This is because the IPv6 header is longer than the IPv4 header.
In most cases, mobile subs don’t get a choice as to if their connection is IPv4 or IPv6, but on a like for like basis, we can say that if a customer moves is on IPv6 every packet sent/received will have an extra 20 bytes of data consumed compared to IPv4.
This means subscribers use more data on IPv6, and this means they get charged for more data on IPv6.
For IoT applications, light users and PAYG users, this extra 20 bytes per packet could add up to something significant – But how much?
We can quantify this, but we’d need to know the number of packets sent on average, and the quantity of the data transferred, because the number of packets is the multiplier here.
So for starters I’ve left a phone on the desk, it’s registered to the network but just sitting in Idle mode – This is an engineering phone from an OEM, it’s just used for testing so doesn’t have anything loaded onto it in terms of apps, it’s not signed into any applications, or checking in the background, so I thought I’d try something more realistic.
So to get a clearer picture, I chucked a SIM in my regular everyday phone I use personally, registered it to the cellular lab I have here. For the next hour I sniffed the GTP traffic for the phone while it was sitting on my desk, not touching the phone, and here’s what I’ve got:
Overall the PCAP includes 6,417,732 bytes of data, but this includes the transport and GTP headers, meaning we can drop everything above it in our traffic calculations.
For this I’ve got 14 bytes of ethernet, 20 bytes IP, 8 bytes UDP and 5 bytes for TZSP (this is to copy the traffic from the eNB to my local machine), then we’ve got the transport from the eNB to the SGW, 14 bytes of ethernet again, 20 bytes of IP , 8 bytes of UDP and 8 bytes of GTP then the payload itself. Phew. All this means we can drop 97 bytes off every packet.
We have 16,889 packets, 6,417,732 bytes in total, minus 97 bytes from each gives us 1,638,233 of headers to drop (~1.6MB) giving us a total of 4.556 MB traffic to/from the phone itself.
This means my Android phone consumes 4.5 MB of cellular data in an hour while sitting on the desk, with 16,889 packets in/out.
Okay, now we’re getting somewhere!
So now we can answer the question, if each of these 16k packets was IPv6, rather than IPv4, we’d be adding another 20 bytes to each of them, 20 bytes x 16,889 packets gives 337,780 bytes (~0.3MB) to add to the total.
But before you go on about what an outrage this IPv6 transport is, being charged for those extra bytes, that’s only one part of the picture.
There’s a reason operators are finally embracing IPv6, and it’s not to put an extra 7% of traffic on the network (I think if you asked most capacity planners, they’d say they want data savings, not growth).
IPv6 is, for lack of a better term, less rubbish than IPv4.
There’s a lot of drivers for IPv6, and some of these will reduce data consumption. IPv6 is actually your stuff talking directly to the remote stuff, this means that we don’t need to rely on NAT, so no need to do NAT keepalives, and opening new sessions, which is going to save you data. If you’re running apps that need to keep a connection to somewhere alive, these data savings could negate your IPv6 overhead costs.
Will these potential data savings when using IPv6 outweigh the costs?
That’s going to depend on your use case.
If you’ve extremely bandwidth / data constrained, for example, you have an IoT device on an NTN / satellite connection, that was having to Push data every X hours via IPv4 because you couldn’t pull data from it as it had no public IP, then moving it to IPv6 so you can pull the data on the public IP, on demand, will save you data. That’s a win with IPv6.
If you’re a mobile user, watching YouTube, getting push notifications and using your phone like a normal human, probably not, but if you’re using data like a normal user, you’ve probably got a sizable data allowance that you don’t end up fully consuming, and the extra 20 bytes per packet will be nothing in comparison to the data used to watch a 2k video on your small phone screen.
Arguably, the most capable (toolbox-carryable) test set out there is the Fluke 990 CopperPro, combined TDR, Butt Set, Toner, RFL, Noise meter, etc, etc, it’s a really cool gizmo (And probably worthy of a blog post in itself someday).
Alas mine had stopped taking much of a charge, you’d plug it into the charger, and after 20 minutes of use, it’d report low power and shut down.
Sitting on the charger the Fluke would show the red charging light for about 20 minutes, then the charging light would turn off.
If I unplugged and replugged the charger I’d get red charging light for another few minutes, then turn off again.
Naively, I ruled it to be an issue with the battery (Which is close to 10 years old), and ordered a new one.
But with the new battery in hand, the same issue.
Stupidly, it turns out I was using a 15v charger, technically the unit supports a 15v charger, but it seems after a period, the internal voltage regulator overheats and shuts off, meaning the battery never got a good charge.
Now with the new battery I’m getting 15 hours of runtime out of a charge, and still squeezing 7 out of 8 year old battery it originally came with.
Dumb mistake but hopefully an easy fix for anyone with the same issue.
I recently finished this one, the book covers the developments of early long distance submarine cables, right up until the 1950s, it’s a good mix of human interest and technological achievement.
I got introduced to Clarke from The Idea Factory – Bell Labs and the Great Age of American Innovation which I read last year, and while it may have been written in the 1950s, this book covers the early days of Gutta-Percha wrapped steel cables being thrown in the ocean, and several failed attempts at linking the Atlantic, up to the Coax systems that were eventually replaced by Fibre after the book was published.
As well as his scientific achievements, and this book, Clarke is perhaps best remembered as the author of 2001: A Space Odyssey.
I’d suggest reading The Undersea Network after Voice Across the Sea, as it somewhat bookends this.
A well written introduction to “The Internet”. If you know AS numbers off the top of your head, you may find it lacking in technical detail, but this book isn’t meant to be a guide on IP Transit Engineering, but rather how the internet was built and continues to run, and the people behind it more than the tech.
So I run a lot of VMs. It’s not unusual when I’m automating something with Ansible or setting up a complex lab to be running 20+ VMs at a time, and often I’ll create a base VM and clone it a dozen times.
Alas, Ubuntu 20.04 has some very irritating default behaviour, where even if the MAC addresses of these cloned VMs differ they get the same IP Address from DHCP.
That’s because by default Netplan doesn’t use the MAC address as the identifier when requesting a DHCP lease. And if you’ve cloned a VM the identifier it does use doesn’t change even if you do change the MAC address…
Irritating, but easily fixed!
Editing the netplan config:
Run a netplan-apply and you’re done.
Now you can clone that VM as many times as you like and each will get it’s own unique IP address.
Give me a lever long enough and a fulcrum on which to place it, and I shall move the world.
For me, the equivalent would be:
Give me a packet capture of the problem occurring and a standards document against which to compare it, and I shall debug the networking world.
And if you’re like me, there’s a good chance when things are really not going your way, you roll up your sleeves, break out Wireshark and the standards docs and begin the painstaking process of trying to work out what’s not right.
Today’s problem involved a side by side comparison between a pcap of a known good call, and one which is failing, so I just had to compare the two, which is slow and fairly error-prone,
So I started looking for something to diff PCAPs easily. The data I was working with was ASN.1 encoded so I couldn’t export as text like you can with HTTP or SIP based protocols and compare it that way.
In the end I stumbled across something even better to compare frames from packet captures side by side, with the decoding intact!
Turns out yo ucan copy the values including decoding from within Wireshark, which means you can then just paste the contents into a diff tool (I’m using the fabulous Meld on Linux, but any diff tool will do including diff itself) and off you go, side-by-side comparison.
Select the first packet/frame you’re interested in (or even just the section), expand the subkeys, right click, copy “All Visible items”. This copy contains all the decoded data, not just the raw bytes, which is what makes it so great.
Next paste it into your diff tool of choice, repeat with the one to compare against, scroll past the data you know is going to be different (session IDs, IPs, etc) and ta-da, there’s the differences.
I wanted to be able to use my desktop computer which lives in my office, on the TV in the living room.
Long HDMI cables would involve me climbing around under the house, and making more holes in the walls, and most wireless keyboard/mouse combos wouldn’t reach that far and USB has a limit of 5 meters.
So instead I put together a rather simple solution that I’m quite happy with.
I ran a lot of Cat5 in the house a while ago, and I’ve got 4 Cat5e sockets behind the TV, and a patch panel at my desk.
I purchased online a HDMI over RJ45/Cat5e adapter, and a USB over RJ45/Cat5e adapter online, for about $5 each.
These are passive devices (baluns) meaning they aren’t converting anything to IP or Ethernet for transport.
This means no additional latency (beyond velocity factor of the cable and the distance, but I digress), so it’s not a remote-desktop experience, it’s like sitting in front of a screen, because that’s what it is.
I’d tried in the past to use the USB port on the Mikrotik, an external HDD and the SMB server in RouterOS, to act as a simple NAS for sharing files on the home network. And the performance was terrible.
This is because the device is a Router. Not a NAS (duh). And everything I later read online confirmed that yes, this is a router, not a NAS so stop trying.
But I recently got a new Mikrotik CRS109, so now I have a new Mikrotik, how bad is the SMB file share performance?
To test this I’ve got a USB drive with some files on it, an old Mikrotik RB915G and the new Mikrotik CRS109-8G-1S-2HnD-IN, and compared the time it takes to download a file between the two.
Mikrotik Routerboard RB951G
While pulling a 2Gb file of random data from a FAT formatted flash drive, I achieved a consistent 1.9MB/s (15.2 Mb/s)
nick@oldfaithful:~$ smbget smb://10.0.1.1/share1/2Gb_file.bin
Password for [nick] connecting to //share1/10.0.1.1:
Using workgroup WORKGROUP, user nick
Downloaded 2.07GB in 1123 seconds
In terms of transfer speed, a consistent 2.8MB/s (22.4 Mb/s)
nick@oldfaithful:~$ smbget smb://10.0.1.1/share1/2Gb_file.bin
Password for [nick] connecting to //share1/10.0.1.1:
Using workgroup WORKGROUP, user nick
Downloaded 2.07GB in 736 seconds
Profiler shows 25% CPU usage on “other”, which drops down as soon as the file transfers stop,
So better, but still not a NAS (duh).
Still not a NAS. So stop trying to use it as a NAS.
As my download speed is faster than 22Mbps I’d just be better to use cloud storage.
I’ve been adding SNMP support to an open source project I’ve been working on (PyHSS) to generate metrics / performance statistics from it, and this meant staring down SNMP again, but this time I’ve come up with a novel way to handle SNMP, that made it much less painful that normal.
The requirement was simple enough, I already had a piece of software I’d written in Python, but I had a need to add an SNMP server to get information about that bit of software.
For a little more detail – PyHSS handles Device Watchdog Requests already, but I needed a count of how many it had handled, made accessible via SNMP. So inside the logic that does this I just increment a counter in Redis;
Then when you run it, presto, you’re exposing that data via SNMP.
You can verify it through SNMP walk or start integrating it into your NMS, in the above example OID 184.108.40.206.220.127.116.11.0.2, contains the value of Answer_280_attempt_count from Redis, and with that, you’re exposing info via SNMP, all while not really having to think about SNMP.
*Ok, you still have to sort which OIDs you assign for what, but you get the idea.
We’ve already touched on how subscribers are authenticated to the network, how the network is authenticated to subscribers and how the key hierarchy works for encryption of user data and control plane data.
If the IMSI was broadcast in the clear over the air, anyone listening would have the unique identifier of the subscriber nearby and be able to track their movements.
We want to limit the use of the IMSI over the air to a minimum.
During the first exchange the terminal is forced to send it’s IMSI, it’s the only way we can go about authenticating to the network, but once the terminal is authenticated and encryption of the radio link has been established, the network allocates a temporary identifier to the terminal, called the Temporary Mobile Subscriber Identity (TMSI) by the serving MME.
The TMSI is given to the terminal once encryption is setup, so only the network and the terminal know the mapping between IMSI and TMSI.
The TMSI is used for all future communication between the Network and the Terminal, hiding the IMSI.
The TMSI can be updated / changed at regular intervals to ensure the IMSI-TMSI mapping cannot be ascertained by a process of elimination.
The TMSI is short – only 4 bytes long – and this only has significance for the serving MME.
For the network to ascertain what MME is serving what TMSI the terminal is also assigned a Globally Unique Temporary (UE) Identity (GUTI), to identify the MME that knows the TMSI to IMSI mapping.
The GUTI is made up of the MNC/MCC combination, then an MME group ID to identify the MME group the serving MME is in, a MME code to identify the MME that allocated the TMSI and finally the TMSI itself.
The decision to use the TMSI or GUTI in a dialog is dependant on the needs of the dialog and what information both sides have. For example in an MME change the GUTI is needed so the original IMSI can be determined by the new MME, while in a normal handover the TMSI is enough.
Let’s look at how the Tracking Area Updates work from the point of view of the network.
Let’s take an example of a UE which has been sent the Tracking Area List TA0 and TA1, which is currently in ECM_IDLE state served by eNBs in Tracking Area 1.
The UE is moving towards another eNB in Tracking Area 2. As the UE listens on the Broadcast Channel the power of the new eNB overtakes that of the previous eNB, but the UE notes the Tracking Area of the new eNB, which is not on the UE’s Tracking Area List.
So the UE must make a Tracking Area Update to inform the network.
The first thing to do is to establish a radio connection.
Once the radio connection is setup a S1-AP connection is setup, upon which an NAS message – EMM Tracking Area Update Request is sent which contains the GUIT and old Tracking Area ID, which is sent to the MME.
The MME then sends back a new Tracking Area List for the UE and new TMSI to update the GUTI of the subscriber.
The UE updates it’s GUTI, updates it’s Tracking Area List, sends an EMM TRACKING AREA UPDATE COMPLETE and the UE returns to ECM_IDLE state.
As we’ve seen earlier, the eNB needs a connection to an MME and a S-GW.
However different eNBs may connect to different S-GWs or different MMEs, and our UE may connect to any eNB, so we need a way to handover between S-GWs and MMEs.
Handover to new S-GW
Let’s take a look at a scenario where a UE is moving from one eNB to another, and each of the two eNBs is in a different S-GW.
At the start we have a connection from the MME to the S-GW, a GTP-C tunnel for control information and a GTP-U tunnel (called the S5/58 bearer) that carriers the user data over GTP-U between the P-GW and the S-GW.
As the UE moves to the eNB in TA2 we need the MME to modify the tunnel from the P-GW to the S-GW to change it from connecting the P-GW to the old S-GW and instead connecting the P-GW to the new S-GW.
The MME establishes a new tunnel for control to the new S-GW, and sends a message to the new S-GW to modify the tunnel from the P-GW to the old S-GW to point to the new S-GW.
As we saw before larger Tracking Areas minimize the number of UEs between terminals to update their location.
The problem is the cells/eNBs on the edge of the Tracking Area have to handle almost all of the Tracking Area Update requests, to inform the network the UE has moved to a new TA.
There’s an obvious imbalance between edge cells that handle almost all of Tracking Area Updates and the central cells inside a Tracking Area that handle very few many Tracking Area Update messages.
As we know we only have one radio interface, and sending Tracking Area Updates eats into our valuable radio resources that can’t be used to carry user data. Because of this users can experience a lower bit rate on edge cells.
To get around this we group Tracking Areas together into Tracking Area Lists.
A Tracking Area List is provided by the network to the UE, and contains a list of Tracking Areas, so long as the UE stays within the list of Tracking Areas, there is no need for it to send a Tracking Area Update.
You might think this just makes our problem worse, as now at the edges of the cells in the Tracking Area List we have even more signaling traffic, the clever part comes from the fact the network gives out different Tracking Area Lists to different UEs.
In the example below we can see UE2 has a different Tracking Area List to UE1.
This means the cell edges are different for UE1 and UE2, which spreads the signaling load across Tracking Areas, so while UE2 will send a Tracking Area Update when it reaches the border from TA1 to TA4, UE1 will send a Tracking Area Update when it passes from TA6 to TA9.
The other limitation of this is now to reach a UE paging must be sent on all cells in the Tracking Area List.
As we saw with the Network Triggered Service Request, the network needs to know which eNB / cell the UE is currently being served by.
The UE knows which cell it should use as it’s always listening on the broadcast channel to know the received power levels of the nearby eNBs.
If our UE is in ECM IDLE state and the network needs to contact the UE, the eNB sends sends a Paging Request on the Beacon (Broadcast) Channel with the UE’s RNTI.
The UE is always listening on the Beacon Channel for it’s own RNTI, and when it hears it’s own RNTI it follows the process to come back from ECM_IDLE state to ECM_CONNECTED state.
For this to work the network needs to know which eNB to send the Paging request to.
For this to work our UE would need to inform the network each time it changes eNB, but, as we’ve touched upon several times, minimizing power consumption is a constant architecture constraint in LTE.
So if the UE has to transmit each time a UE moves to a different eNB / Cell, the UE power consumption would be high and the battery life of the UE would be low.
If we imagine driving along a freeway at speed, with each eNB serving an area of 1km, at 60kph, our UE would change cells every minute, and if the UE needs to transmit to let the network know it’s changing location, we’d be transmitting data every 60 seconds even if the UE is sitting in our pocket, all these transmissions would lead to lower battery life on the UE.
To work around the power wastage of each UE transmitting data to the network to let it know each time it changes eNB, 3GPP designers decided to group eNBs in the same geographic area into Tracking Areas or TAs.
This means instead of the network knowing exactly which eNB a UE is located in, it has it’s location down to a tracking area made up of several eNBs. (Tens to hundreds of cells per TA)
To go back to our freeway example, we might group all the eNBs along a freeway into one Tracking Area, all of which broadcast the ID of each eNB and the Tracking Area of each eNB.
As the UE moves from one eNB to another eNB in the same Tracking Area, there’s no need for the UE to send a Tracking Area Update message as it’s reamining in the same Tracking Area.
Tracking Area Update messages only need to be sent when the UE moves to an eNB in a different Tracking Area.
Paging a Tracking Area
As the network knows the location of our UE down to a tracking area, when it comes time to Page a UE a Paging Request is simply sent from the MME to all eNBs in the Tracking Area that the UE is in.
This means the RNTI of the UE is broadcast out of all eNBs in that tracking Area, and the UE establishes connectivity once again with it’s nearest eNB.
As we discussed before when no data has been sent by a UE for a period of time the eNB will switch from an ECM-Connected state to an ECM-Idle state where there is no radio connection.
So let’s look at the release procedure.
When the transmission timeout (typically 10 – 30 seconds) has expired, meaning a user hasn’t sent data for that length of time, the eNB sends the MME a S1-AP UE Context Release Request with the cause of User Inactivity to denote why the change is being made.
The MME then sends a GTP-C message requesting release of the tunnel between the S-GW and the eNB (GTP-C Release request).
The S-GW sends back a GTP-C Release Access Bearers response, indicating it has cleared down the GTP tunnel between itself and the eNB,
The MME then sends a S1-AP UE Context Release Command to the eNB, and the eNB sends an RRCConnectionRelease which releases the RNTI assigned to that UE removing it’s radio resources.
Finally a S1-AP UE Context Release Complete is sent from eNB to the MME to let the MME know the process has completed.
At this stage the RNTI is no longer active so the UE cannot use the RNTI and therefore cannot be assigned radio resources.
The UE is now in ECM_Idle mode, however as it still has an IP Address allocated and can be bought back it’s in EMM_Regsitered mode.
UE is disconnected from the network with no radio resources and does not have an IP Address
EMM-Registered & ECM-Connected
UE is connected to the network with an IP address
Radio resources (RNTI allocated)
Location of the UE known
All tunnels & connections established
EMM-Regsitered & ECM-Idle
UE has an IP address & appears to be connected
No radio resources (RNTI) currently in use
No tunnels or connection from the eNB to the S-GW & MME.
Tunnel between S-GW and P-GW and the tunnel between the MME and S-GW
A relative location (tracking area) of the UE is available
As we just saw when a terminal moves to ECC-Idle while in EMM-Registered state, it releases it’s radio resources, so what happens when the UE needs to send / receive data again?
While one option could have been to go through the full attach procedure again when the UE is triggered, the 3GPP team wanted the re-connection process to be as fast as possible.
As we saw in the last post we don’t drop the S-GW <-> P-GW tunnel, which saves time on re-establishing a connection. The S1 tunnel is also not completely released; the TEID value from the S-GW end of the tunnel is saved by the MME so it can be reused by the new tunnel when the UE reconnects, without needing to inform the S-GW.
One of the common themes we cover over and over in the 4G discussion is the desire to preserve energy on the UE RF side of things, to extend battery life as much as possible.
The 3GPPs requirements for LTE also included the smallest round trip times, defining less than 5 ms in unload condition, so traffic to the UE must be routed as quickly as possible.
Mobiles are by their very nature, mobile.
This requires UEs to constantly monitor the RF conditions and the signal measurements from different base stations so the UE can determine if it’s time to handoff to another cell due to going further from one eNB and closer to another, or another eNB offering better RF conditions (Strong signal etc).
This requires regular exchanges of messages and checks, but this would take a lot of energy and eat up battery usage.
Instead we avoid maintaining the radio connection all the time with the aid of an inactivity timer on the eNB.
For as long as user data is flowing over the air interface the connection is maintained, for example web browsing, the inactivity timer is constantly reset as traffic flows.
However when the eNB detects no packets sent or received by the UE the timer starts counting down from it’s set value.
When the inactivity timer reaches 0 the RRC Connection is released and the UE no longer has an RNTI.
The UE is still listening to an eNB, it’s just not sending data to it it and visa-versa.
As the radio bearer has been removed the UE the S1-AP and S1-UP bearers between the eNB and the MME and the eNB and the S-GW respectively, can be torn down.
This means the MME is no long sure of exactly which eNB the UE is listening on.
This is referred to as ECM_IDLE state as there is no radio connection, and the network is unaware of the precise location of the UE.
An ECM_ACTIVE state is the state when the UE is connected to an eNB with an RNTI and it’s inactivity timer has not reached 0.
The dotted line bearers shown in the image above frequently change between active and inactive based on the ECM_ACTIVE / ECM_INACTIVE state of the bearers.
Want more telecom goodness?
I have a good old fashioned RSS feed you can subscribe to.