Most of the graphs look pretty uneventful. This computational benchmark from EC2 (measuring Math.sin) is typical:
The Math.sin benchmark running on App Engine shows a little more variation:
Shared storage services show much more variation. Here is the latency to read a single record from an Amazon SimpleDB database (without consistency):
App Engine Datastore read and write latencies wander around in a similar manner -- here's the graph for (non-transactional) reads:
EBS read and write latencies are comparatively stable, with some variation at the 99th percentile. EC2 local-disk latencies are even more stable. Amazon RDS latency is another story -- look at this graph for reads (of a single record from a pool of 1M records):
Here is the graph to write one record to RDS:
Of course, I have no way of knowing what caused any of these variations. I haven't gone near these servers in quite some time, so the RDS latency improvements in June and July can't be anything I did. Just another reminder that you can't take the performance of a shared service for granted. I did receive a communication in late June from AWS support that "we are currently working on adjustments that would reduce latency in the db.m1.small instances"; this may explain the drop on July 9th.
I'll revisit this topic every few months. Until then, back to the logging project...