Operation | # samples | 10th %ile | Median | Mean | 90th %ile | 99th %ile | 99.9th %ile |
Chrome Websocket / DSL | 897 | 89.6 ms | 94.0 ms | 105.0 ms | 105.0 ms | 273.0 ms | 881.0 ms |
Chrome / DSL | 897 | 102.0 ms | 111.0 ms | 121.0 ms | 118.0 ms | 337.0 ms | 1.07 sec |
Firefox / DSL | 1793 | 98.9 ms | 105.0 ms | 116.0 ms | 116.0 ms | 273.0 ms | 1.07 sec |
Safari / DSL | 1755 | 98.5 ms | 103.0 ms | 111.0 ms | 107.0 ms | 270.0 ms | 1.19 sec |
iPhone / DSL | 1182 | 145.0 ms | 156.0 ms | 232.0 ms | 294.0 ms | 1.03 sec | 3.94 sec |
Nexus One / DSL | 1158 | 132.0 ms | 147.0 ms | 589.0 ms | 233.0 ms | 1.91 sec | 68.4 sec |
iPhone / AT&T 2G | 1063 | 487.0 ms | 629.0 ms | 875.0 ms | 1.22 sec | 5.53 sec | 15.9 sec |
Nexus One / T-Mobile 3G | 949 | 286.0 ms | 320.0 ms | 345.0 ms | 381.0 ms | 915.0 ms | 2.27 sec |
Nexus One / AT&T 2G | 240 | 351.0 ms | 438.0 ms | 601.0 ms | 668.0 ms | 5.49 sec | 6.31 sec |
| Nexus One / T-Mobile 2G | 510 | 378.0 ms | 409.0 ms | 429.0 ms | 479.0 ms | 790.0 ms | 2.61 sec |
[Fine print: All of these measurements involve a client in the S.F. Bay Area area talking to an Amazon EC2 server in Virginia. In light of the previous post, both the client and server were minimally loaded. The first result uses Web Sockets, the others all use XMLHttpRequest (only one of my test browsers support Web Sockets). The client device, if not mentioned, is a MacBook Pro; the browser, if not mentioned, is the device's built-in browser. The iPhone is a first-generation model (shocking, I know). All cellular network tests involved a full strength signal (according to the phone's status display), except for "iPhone / AT&T 2G" which had 3 or 4 out of 5 bars. The DSL samples were all taken at roughly the same period of time, from the same location, and thus experienced the same DSL conditions. Your mileage may vary. Past performance is no guarantee of future results.]
This is a very un-scientific study: the sample counts are small, and each test was made from a single location over a brief period. Still, some trends are clear:
- As in my previous experiments, median latencies from the MacBook are quite good, and the tail is not terrible either.
- There are no glaring differences between browsers.
- Web sockets, currently supported only by Chrome (among the browsers I tested), continue to look like a good idea.
- Even over the DSL line (via WiFi), both cell phones were substantially slower than the MacBook, with a very long tail.
- All of the cellular connections were much slower than DSL. 3G is much faster than 2G, and not all 2G connections are created equal.
No surprises here. I wasn't sure what to expect from the cellular connections. It's perhaps surprising that they're so much slower than DSL in this test. The amount of data exchanged is small -- around 600 bytes, most of which is request headers. I don't know the nominal bandwidth of these 2G connections, but let's suppose it's 128Kbps. 600 bytes is 4800 bits, which would take roughly 37 milliseconds to transmit at 128Kbps. Yet even the 10th percentile figures are at least 200 milliseconds slower than the same device communicating via WiFi + DSL.
Impact of request size
I decided to directly test the impact of request size on latency. According to Firebug, when I run this benchmark in Firefox, the request headers are as follows:
for a total of 475 bytes. (The other components of the request and response are small by comparison, so I don't bother to detail them here.) I was able to easily surpress the User-Agent, Accept, and Accept-Language headers using XMLHttpRequest.setRequestHeader (e.g. request.setRequestHeader("User-Agent", "")), reducing the request headers to 263 bytes. Repeating the Firefox / DSL test, this yielded a small but definite improvement:
Impact of request size
I decided to directly test the impact of request size on latency. According to Firebug, when I run this benchmark in Firefox, the request headers are as follows:
Host: metrics2.amistrongeryet.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://metrics2.amistrongeryet.com/CloudMetricsEC2/netbench.jsp
Cookie: sid=s1; browserId=dummy; sid=s1
Operation | # samples | 10th %ile | Median | Mean | 90th %ile | 99th %ile | 99.9th %ile |
| Regular headers | 642 | 100.0 ms | 110.0 ms | 116.0 ms | 117.0 ms | 458.0 ms | 510.0 ms |
| Reduced headers | 641 | 98.8 ms | 104.0 ms | 113.0 ms | 116.0 ms | 399.0 ms | 488.0 ms |
With additional effort, it should be possible to further shrink both the request and response, but on DSL the savings don't appear to be large. The setRequestHeader trick doesn't seem to work on the Nexus browser, so I wasn't able to directly reproduce this comparison on a cellular link. Instead, I did a side-by-side test of the regular request headers vs. "padded" headers (adding an extra header containing 500 bytes of text). Here are those results:
All tests were on a Nexus One with a full-strength signal. Once again, request size has only a moderate impact.
Impact of WiFi
Another variable to be isolated is a wired connection vs. WiFi. Here are figures for Chrome running on two different Macs, both connected to the same DSL line, one via WiFi and one via Ethernet:
There's a difference, but it's fairly minor, at least through the 90th percentile (and this sample is too small to place much weight in the 99th percentile figures). Clearly, WiFi does not explain the speed difference between a smartphone and a conventional computer when both are connecting to a DSL line.
The conclusion seems to be that phones are just plain fundamentally slower. This could be a property of raw processing speed, the browser implementation, competition from background tasks, or all of the above; but it does not seem to relate to factors I can control in an experiment. Encouragingly, the newer phone (Nexus One) and newer network (3G) both perform better than the original iPhone and 2G, so perhaps we can look forward to a general trend of improvement.
Appendix: Histograms
There are no surprises in any of the histograms for these benchmarks, but as histograms seem to be a theme for this blog, I'll toss in a few. Here's the "Chrome / DSL" XMLHttpRequest benchmark:
Many of the histograms look like that -- a sharp spike around 100ms, and a tail out to 1 second or so, with not much structure in the tail. (More structure might be visible with a larger sample count.) The iPhone and Nexus DSL tests had a broader peak -- here's the iPhone:
The block of timings in the 200-300ms range don't seem to be explained by WiFi or other network issues, so perhaps they reflect garbage collection, CPU contention, or some other environmental factor on the phone.
The cellular links had much broader peaks; here's the iPhone again (this is the same AT&T 2G test reported above):
Other cellular links have histograms similar to this one, though not as wide.
Operation | # samples | 10th %ile | Median | Mean | 90th %ile | 99th %ile | 99.9th %ile |
| T-Mobile 3G | 310 | 257.0 ms | 282.0 ms | 299.0 ms | 357.0 ms | 527.0 ms | 579.0 ms |
| T-Mobile 3G (padded) | 309 | 285.0 ms | 318.0 ms | 338.0 ms | 400.0 ms | 653.0 ms | 1.5 sec |
| T-Mobile 2G | 426 | 380.0 ms | 426.0 ms | 454.0 ms | 549.0 ms | 1.01 sec | 1.25 sec |
| T-Mobile 2G (padded ) | 426 | 421.0 ms | 471.0 ms | 499.0 ms | 621.0 ms | 948.0 ms | 1.12 sec |
All tests were on a Nexus One with a full-strength signal. Once again, request size has only a moderate impact.
Impact of WiFi
Another variable to be isolated is a wired connection vs. WiFi. Here are figures for Chrome running on two different Macs, both connected to the same DSL line, one via WiFi and one via Ethernet:
Operation | # samples | 10th %ile | Median | Mean | 90th %ile | 99th %ile | 99.9th %ile |
| XMLHttpRequest / Ethernet | 2227 | 108.0 ms | 113.0 ms | 119.0 ms | 120.0 ms | 326.0 ms | 706.0 ms |
| XMLHttpRequest / Wifi | 2069 | 108.0 ms | 113.0 ms | 124.0 ms | 124.0 ms | 397.0 ms | 516.0 ms |
| Websocket / Ethernet | 2226 | 90.4 ms | 97.9 ms | 102.0 ms | 106.0 ms | 198.0 ms | 894.0 ms |
| Websocket / Wifi | 2069 | 92.7 ms | 101.0 ms | 110.0 ms | 107.0 ms | 385.0 ms | 540.0 ms |
There's a difference, but it's fairly minor, at least through the 90th percentile (and this sample is too small to place much weight in the 99th percentile figures). Clearly, WiFi does not explain the speed difference between a smartphone and a conventional computer when both are connecting to a DSL line.
The conclusion seems to be that phones are just plain fundamentally slower. This could be a property of raw processing speed, the browser implementation, competition from background tasks, or all of the above; but it does not seem to relate to factors I can control in an experiment. Encouragingly, the newer phone (Nexus One) and newer network (3G) both perform better than the original iPhone and 2G, so perhaps we can look forward to a general trend of improvement.
Appendix: Histograms
There are no surprises in any of the histograms for these benchmarks, but as histograms seem to be a theme for this blog, I'll toss in a few. Here's the "Chrome / DSL" XMLHttpRequest benchmark:
Many of the histograms look like that -- a sharp spike around 100ms, and a tail out to 1 second or so, with not much structure in the tail. (More structure might be visible with a larger sample count.) The iPhone and Nexus DSL tests had a broader peak -- here's the iPhone:
The block of timings in the 200-300ms range don't seem to be explained by WiFi or other network issues, so perhaps they reflect garbage collection, CPU contention, or some other environmental factor on the phone.
The cellular links had much broader peaks; here's the iPhone again (this is the same AT&T 2G test reported above):
Other cellular links have histograms similar to this one, though not as wide.
