This was a fun one.

We were testing international roaming for a customer, roaming into the US where one of our team members (shoutout to Cody) is based.

So we sent Cody some SIMs and asked them to run the basic tests for us, but no IMS APN would attach.

We’d get them to power up the phone, fire up a trace on the roaming PGWs, but never seeing an attempt to attach on the IMS APN (No Create Session Request from the SGW in the vPMN).

The operator we were roaming into swore their side was correct – that IMSI was allowed for the IMS APN for testing, but whenever we’d run a trace, same thing, default bearer just fine but no CSR for the IMS APN, as if the IMS APN was blocked on the roaming network (Which is common on networks where you haven’t launched VoLTE but do have data roaming).

The steps we would do is:

1. Person in the US turns on phone tries to attach
2. I fire up my computer, get a trace running
3. Person in the US airplanes the device
4. I monitor for CSRs on the PGWs for that IMSI

But no CSR for the IMS APN would ever come through.

After a few attempts, here’s what we found was happening:

1. The phone would get powered up
2. The phone would roam onto the vPMN in the US and the default bearer would come up
3. The phone would try to attach to the ims APN, this would work for this SIM which was whitelisted, and the Create Session Request for the ims APN went to the PGW in the hPMN
4. The ims APN came up as expected
5. The phone would send a SIP REGISTER (So far so good)
6. Our IMS had an issue with Rx routing in this scenario, so the SIP REGISTER would timeout, and when it timed out, a 504 error was sent back to the phone by the P-CSCF and it set the Retry-After header to 3600 seconds.
7. The phone would not try again for as long as that timer value was set.
8. At this point we’d start a trace, airplane the device, and see no IMS APN attach attempt.

This 504 Timeout would all happen when the phone fired up before we had any traces running, so we weren’t capturing that.

I’d wrongly assumed that airplaning the device after starting a trace would reset the state fully, but it doesn’t, neither does a restart of the phone.

When we’d started a trace and airplane the phone, the phone wouldn’t try to attach to the ims APN as it was still inside the Retry-After time window from when we’d first fired it up.

Per RFC 3261, the phone should not try again during this time, which in our case meant no attempt to attach to the IMS APN, this makes sense – it protects the network against the thundering herd problem, but made this otherwise simple fault really hard to find.

Anywho, that was my facepalm failure of the day!