Update (22 December 2014): The following instructions have been updated and tested with iOS 8.
Right now, my challenge is upgrading almost 200 iPads to iOS 7 with minimal pain (read: zero device handling). Factor in less-than-ideal Internet bandwidth and Apple’s disinterest in allowing proxies to cache iOS updates, and it’s been a bit of a headache.
First, a word of advice: ask your users not to upgrade when prompted. Do this before Apple release a major update, to buy yourself some time to test it on your network and to check that the update is being cached properly.
Hopefully your iPad fleet is already using your Squid proxy. Ours is configured (via Apple’s Profile Manager) to use a PAC file when it’s on our WiFi network. The PAC file directs all but onsite requests to Squid.
Unfortunately, iOS doesn’t use the proxy for everything; system update authorizations, in particular, don’t get out unless permitted on your firewall. Here’s the relevant rule on our iptables firewall (no_proxy_ok is one of our custom chains, as is tcp_allowed):
-A no_proxy_ok -p tcp -m comment -m tcp -m multiport -d 17.0.0.0/8 -j tcp_allowed --dports 80,443,5223,2195,2196 --comment "allow Apple services (e.g. APNs, updates)"
Mercifully, the update itself is requested via the proxy, but getting it to cache is non-trivial. Obviously max_object_size needs to be big enough to accommodate a 1GB+ file. I went with 2GB:
maximum_object_size 2048000000 bytes
But this wasn’t enough to get the update to cache. A bit of sleuthing led to the first problem: Apple adds HTTP headers like these to its updates, so Squid discards them:
Cache-Control: max-age=0, no-cache, no-store Pragma: no-cache
The workaround is to break HTTP a little by adding this line above any other refresh_pattern entries in your squid.conf:
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
This forces Squid to treat objects from *.appldnld.apple.com and *.phobos.apple.com as “fresh” (i.e. cacheable) for 90 days (129600 minutes), no matter what appldnld.apple.com and phobos.apple.com say.
Finally, I made sure appldnld.apple.com requests were excluded from Squid’s delay pools and filtering ACLs; you may need to make similar tweaks. I also found that maximum_object_size wasn’t being applied correctly to cache_dir, so I defined it explicitly, i.e.:
cache_dir aufs /var/spool/squid3 256000 128 256 max-size=2048000000
iOS 7 is rolling out smoothly as I type.
Excellent! Thank you so much for posting this!
I’m going to modify the config on my Cachebox (based on squid 2.7, http://www.appliansys.com) this very afternoon. Hopefully I can improve my daily caching ratio a bit…
Having only 2.5 mbps for 400 users, with Apple’s devices being the most popular ones, this trick has helped us to regain control of our internet connection. We had blocked iTunes completely because we couldn’t get the darn files to stay in cache…
For some reason this was not working on Debian 7.1 with squid 3.1.20
I added `ignore no-cache` and made it explicit for zip files.
refresh_pattern -i appldnld\.apple\.com.*\.(zip) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate ignore-no-cache
Interesting … no-cache headers are client-initiated, so maybe the on-device updater is trying harder to fetch an uncached file now. Thanks for sharing!
Excellent tutorial. I am a school IT Manager also. I have managed to cache apple store updates using your advice.
However, our main bandwidth nightmare is that our school distributes our teacher’s notes (in ibook and pdf format via iTunesU) and 150+ iPads have to download the same book in a period of few hours. Therefore, I have added another refresh_pattern:
refresh_pattern -i ([^.]+.|)(phobos).(apple.|)com/.*\.(pdf|ibooks|ipa) 10080 100% 10080 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate;
However, books are not cached (TCP_MISS/200, as the URL contains a dynamic part (something like http://xxx.phobos.apple.com/…/…/test.ibook?downloadKey=1384882721_3e35e8cd987509c269f6a14089f81a00 )
I have tried to turn squid3 dynamic content on and off. I have tried to add the dynamic refresh_pattern (/cgi-bin/|\?) 0 0% 0;
Have you tried anything similar?
Best regards.
This has probably already been resolved by the person but in case it’s helpful, I noticed the following:
It appears your regex had “ibooks” instead of “ibook” which you indicated in the URL provided.
2014/03/03 01:12:57| parse_refreshpattern: Unknown option ‘appldnld\.apple\.com’: ignore-no-store
Is anyone getting this with the Windows version of 2.7STABLE8
This is probably the coolest thing I have ever seen.
Looks like you need Intune for this then. Which of course I won’t be able to use, since it costs money! Thanks anyway.
Thanks for this :)
Do you know if this can be used to cache app downloads from the App Store?
I don’t think it works with App Store downloads because of their URL structure. But OS X Server does cache apps (if you enable this feature).
Does this work for a transparent squid proxy as well ? Also have you tried something similar for OS X, although OS X software updates may be https ( https://support.apple.com/en-us/HT202943 ) ? I am glad iOS software update is not https
It should work with transparent proxies, but I haven’t tested it in this scenario. OS X updates are easiest to cache with OS X Server. Enable “Software Update” for operating system updates, and “Caching” for apps (includes both iOS and OS X).
Thanks! Do you know of a way to cache OS X updates with just Squid ? Trying to figure out if possible without buying a OS X server.
You’ll have to experiment with this yourself, sorry. I have no need for this myself.
No worries. Thanks!
Thank you. confirmed working with IOS 9.2.1 (squid 3.5.13-2, centos6.7)
Hi Luke,
Thanks for the information it has been an absolute godsend.
Just wondering if there have been any changes to this configuration and are you caching apps as well?
Thanks in advance!
I’ve just done some updated testing on this. Results below. In short, iOS apps can’t be cached by Squid, but macOS apps can be (directives in reply below).
I am looking to Cache IOS and Android Apps as well. Any help is greatly appreciated.
I haven’t looked into Android app caching, but can confirm this doesn’t work with iOS apps — they are retrieved with a unique authorisation key in the query string each time, which makes them uncacheable (unless you’re running a macOS Server with Caching enabled). macOS apps can be cached with a line like this, however:
Is this possible on a pfSense (2.3.x) firewall with squid installed?
+1 Also interested:
Is this possible on a pfSense (2.3.x) firewall with squid installed?
hello, please can you help me? i can’t send whatsapp msg with iphone across squid (android works well)