First, for comparison, here’s the November 2009 version at the same scale:

And this is how the list looks now:

For one, given the number of recent announcements around it, I was expecting for 10G-Ethernet to gain some more adoption by now. Yet (assuming those details in the TOP500 list – and my parsing of it – are correct), only two new systems use 10G-Eth while those that were in last November’s list have now fallen out.

But something much more interesting seems to be happening at the lower right corner of the graph (look at the three blue (Inifiniband) dots nearest to the legend, interestingly, all three systems are Chinese). The Tianhe-1 hybrid Intel Xeon + ATI Radeon cluster (now at #7) got some company from two more GPGPU clusters, Mole-8.5 at #19 and even the new #2, Nebulae. They are characterized by a rather low efficiency – especially for this part of the ranking. Note that no-one else in the right half fools around with mere Gigabit Ethernet or otherwise has an efficiency that is lower than 70%. Yet the number two of this list only manages a 43% efficiency, and needs 2984 GFLOPS of raw computing power (28% more than the #1 system, Jaguar) to get at a LINPACK score of 28% less than Jaguar’s.

The explanation for this lies in two numbers that are not shown in this graph: power and cost. Just like the commodity-based clusters started to take over from the hard-core custom-built supercomputers some 10 years ago, the GPGPU-based system may very well be on its way to take over the charts. They follow the same basic recipe as those Beowulf-inspired clusters: a not-so-great efficiency, which is cured with loads of cheap, low-power processing power.

The efficiency of the cluster improved drastically over time, using better interconnect such as Infiniband. The cluster idea is now so prevalent that over 80% of the systems in the Top500 today should be categorized as clusters. The next question is how GPGPU will evolve into something that can combine its advantages of low cost and low power with increased efficiency. Nvidia has some clear ideas about HPC being the future for their products (although their first step, Fermi, got executed rather strangely…). ATI/AMD are also hard at work with FireStream and OpenCL. And coming from the other side of the arena, we have the general purpose processors moving towards simpler, but many more cores per chip, an idea embodied in the form of (among many others) Tilera’s TILE family or Intel’s SCC.

Interesting times again loom ahead. Let the games, eh computations, begin!

My preliminary findings: Google AppEngine is really cool (obviously), and using Python again after almost OD-ing on PHP was very pleasant. I’m still learning to properly use the datastore though, setting a multi-column primary key to guarantee unique (date, currency) records wasn’t very straightforward. Also the import of the historical data was a bit of a hassle with the import script timing out, until I found how the BulkLoader can automatically do this in multiple HTTP requests. Finally, getting this to run on my own domain, currencies.apps.grandtrunk.net, took some time until I found out the right DNS magic to set in DreamHost‘s panel (if you’re interested: I’m now fully hosting apps.grandtrunk.net, which allows me to set the domain validation code using a normal file uploaded to DreamHost; apps.grandtrunk.net is also the domain I told Google Apps to use, while at DreamHost I needed to set a CNAME (alias) record for currencies.apps.grandtrunk.net that points to ghs.google.com). The cron job is also humming along nicely now downloading daily updates, so convert away while I watch the dashboard seeing my quota trickle down…

Using the StackOverflow public data set (the October 2009 one, which I still had lying around from last time), I set about plotting the number of questions versus the number of answers contributed by each user. This graph shows the raw results:

Clearly, several very active users tend to ‘specialize’ towards either the Questions or the Answers axis. One user is even up to almost 800 questions, but barely gave any answers. Can you earn reputation points that way? Sure, if they are good questions, other people will pass by that have hit the same dead-end and express their eternal gratitude of seeing their problem already solved by voting the question up. Here’s the same graph from before (except for the log-log axes) but including colors to show reputation classes:

The next graph plots each user’s reputation versus their “Q-to-A ratio” (their number of questions divided by their number of answers). For users without any answers I just plotted their number of questions (red dots).

The vast majority of users give much more answers than they have asked questions (so they have a Q/A-ratio of less than one) and inhabit the region near the horizontal axis. Most high-reputation users also tend to answer more than they ask. In the lower reputation regions (<1000) much higher Q/A-ratios can be seen (including those with an ‘infinite’ ratio, i.e. with no answers).

(As I found out later, someone else had already computed the top Q/A-ratios on an older SO dump, although I always prefer looking at things with some graphs…)

Finally, here are some Q/A-ratio distributions: the first one shows the number of users in each Q/A-ratio class. (Never mind the fact that the 1/1 class seems extraordinarily large in comparison to it’s neighbours, I’ve used a pretty strange class grouping function (an arctangens, so it can show both 0 and infinity symmetrically around 0) to make it fit properly on the graph.) The distribution is clearly weighted towards the right, again showing that most users have contributed more answers than questions.

We can also weigh users by their reputation, so we get the fraction of total reputation that is represented by the users with a certain Q/A-ratio. The already large group on the left side of the Q/A scale becomes even more important now, meaning that most reputation is owned by users with a low (<1/10) Q-to-A ratio. This is confirmed by the next graph which shows the average reputation of all users in a given Q/A class.

Here are some nice results from my last trip:

I also re-did some older ones, including my 360° Empire State building one:

Comparing these Hugin results with my previous panorama’s, both stitching and overall image quality have been hugely improved. The Empire State one still has some rather bad stitches though, but that’s probably because I was still using my Canon 18-55mm kit lens at the time, and set it to its extreme end of 18mm which results in some pretty bad distortions. I’ve learned to use an intermediate zoom settings since, which greatly improves results.

It’s easy to do this using tags: some sort of clustering should be applied according to how often each pair of tags shows up at the same question (a user that knows both ASP and ASP.net shouldn’t be considered a ‘diverse’ person, so this should be factored out first), next we can count in how many different clusters that this user has contributed a good answer.

I’ve had a stab at trying this on the last (October 2009) StackOverflow public data set. I’ve ignored the SU and SF parts of the dump. The idea is to count how many of SO’s questions you could conceivably answer, given your proficiency in each of the tags of that question.

First I’ve scored all answers each user has given. 20 points go to it being the accepted answer, another 80 points are distributed over all answers to a question in relation to the answer’s votes. This is not exactly the same as the reputation earned for that answer, since popular questions get a lot more up-votes to their answers, but in this analysis this isn’t really worth much more than a good answer to an unpopular question.

The points for all your answers are distributed over the tags with which the question is tagged. This gives each user a number of points for all tags. These points are converted into a proficiency that user has for this tag: proficiency = 1 - exp(-points / 500). So 1000 points (10 very good answers) gets you to 86%, more points will asymptotically get you to 100%.

At this point we can compute the average proficiency over all tags for each user, which is plotted in this graph (average tag proficiency versus reputation, for users with reputation > 1000):

The next step is to compute the question proficiency. This is done, for each question and each user, by taking the geometric average of this user’s tag proficiencies over all tags that this question is tagged with (I’ve choosen the geometric average rather than the arithmetic one, since you’re supposed to be knowledgable on all components of a question in order to answer it). These per-question proficiencies can again be averaged (arithmetically), yielding the user’s average question proficiency which is a measure for how many of the site’s questions he/she could answer. This graph plots average question proficiency versus reputation:

Note that this last one is a fairly heavy query (~1 second per user), so it could more feasible to base an actual implementation on tag proficiency only (see also this graph of tag vs. question proficiency), although this would overvalue knowledge of topics SO doesn’t care much about.

Analysing the question proficiency vs. reputation graph, we see that both are clearly related (if you answer enough questions, you’re bound to have covered most of the tags). Still, some users reach a possible threshold value of 20% question proficiency at a reputation of only 10,000; while others couldn’t get there even at 60,000 reps.

I must say the result looks pretty professional. Much faster, and even cheaper, than soldering them ourselves!

The funny little ‘ergonomic’ programmer that Michiel designed also looks rather cute

So, if you’re interested in learning about microcontrollers, have a look at our site. We’re still working on some of the tutorials, but once those are done, nothing should be left in the way to really get you started with microcontrollers!

For the following set of graphs, I downloaded the XML file for the current TOP500 list (since updated to the November 2009 list). I decided to look at each system’s LINPACK performance (Rmax) compared with its peak performance (Rpeak) which is what you get when adding the theoretical performance of all processor cores in the system. The result should be a measure of how efficiently the system’s total computational power can be used, which is mostly a function of the interconnect (also software, but for a TOP500 listing we would expect that to be rather good!).

So here is the graph:

Here’s the same data but this time efficiency (Rmax/Rpeak) is plotted directly:

For top-of-the-list machines, Infiniband and proprietary interconnects (mainly BlueGene, Cray and SGI’s NUMAlink) are the most common. Their efficiency is rather similar at 75-80% for the large machines, and up to 95% for smaller ones. The main alternative interconnection technology, Ethernet, only starts at to be in common use from system #98 onwards. Clearly, the efficiency of these systems is significantly less, at some 55%.

Two main outliers to this theory can be seen. System #5 uses Infiniband but manages to get only a 46% efficiency. This is the Tianhe-1 cluster located in Tianjin, China. It consists of 5120 ATI Radeon HD 4870 video cards, in addition to a number of Intel Xeon CPUs. I would guess that the software for this rather radical new architecture (especially at this scale!) is not yet fully up-to-date, and that this system should get a boost in Rmax by the time of the next TOP500 list next June.

The other exception is #486, again a Chinese system, which uses a 10-Gbit Ethernet network (rather than 1-Gb Ethernet for the others – at least, as far as I could tell from parsing the list). Clearly, the faster Ethernet technology significantly improves efficiency: at 70% it comes much closer to the Inifiband and proprietary-interconnect powered systems. When compared to 1GbE, the difference between 10GbE and Infiniband is minimal though, and also the price comes pretty close. So I guess, in the end, you do get what you pay for…

I couldn’t resist, and bought another lens. This time a real wide-angle lens, the Tokina AF 11-16mm f/2.8 AT-X Pro DX. Yes, it’s a zoom lens, and on paper the range fitted nicely to my Tamron 17-50mm f/2.8. But the difference between 11 and 16mm turned out to be so tiny (and the image quality so good that cropping an 11mm image down to 16mm would be more than adequate), that I might as well have gotten a prime 11mm one. But still, a very nice lens this is!

So what exactly did I ‘need’ this lens for? My brother wanted some shots of his studio. Our attic isn’t that big, but on these pictures it looks pretty impressive

Of course, it’s also a nice holiday lens:

Instead I’ll be focusing on very well known, established compressors that are easily obtained (I only use precompiled packages and won’t build from source) and have reasonable run times. Also I won’t try every combination of compression options but limit the test to one general option (-1 to -9 for gzip and bzip2, -m1 to -m5 for RAR, …).

Updates

• 2007/04/25: Updated 7-zip to version 4.45: small improvements in speed and compression ratio
• 2006/06/08: Reran all tests on a Pentium D 830 (dual-core, 3 GHz, 2x 1MiB L2) and 2 GiB RAM
• 2006/02/13: Updated 7-zip to version 4.33: both faster compression (up to 20%) and better compression ratios (a few %)
• 2005/11/27: Updated 7-zip to version 4.30 beta: high compression (-mx=7 … -mx=9) is now up to 40% faster
• 2005/11/03: Updated 7-zip to version 4.29 beta: no changes
• 2005/04/12: Updated 7-zip to version 4.16: the ratio is the same, but compression and decompression are now 30% to 40% faster!
• 2004/11/10: Added the 7-zip compressor
• 2004/09/30: Added the lzop compression algorithm

The Test

Just like Johan De Bock’s excellent GIMP Source Compression Test, I’ll be using the GIMP 2.0 Source tarfile:

• The test file: GIMP 2.0.0 Sources as one TAR
• MD5-Hash: d2a1c33317fb57bbed3641671b2da163
• Total size: 78,745,600 Bytes

Each program is used to compress and decompress the file with each of the selected command line switches. The time required for both compression and decompression is measured, as is the size of the resulting archive. To have an idea of the accuracy of the timing measurements, the test is repeated three times and the minimum runtime (= the one with the least disturbances) is reported. All this is done on a Pentium D 830 (3.0 GHz) processor with 2 GiB RAM, running Fedora Core 5 Linux. Only one CPU is used for the tests (using -mmt=off for 7-zip, default for other compressors).

The Contenders

The versions aren’t always the latest and the greatest, but they are – again in the spirit of being ‘practical’ – the most recent ones installed by the Fedora Core 5 distribution.

Results – graphs

These graphs show the results for the various compressors and their switches. Compression and decompression time, respectively, are in the horizontal axis, compression efficiency (compressed size / original size, so smaller is better) is on the vertical axis.

Conclusions

• The best compressor can be found near the lower (efficient) left (fast) corner. You’re right: there isn’t any! All real compressors are either inefficient (gzip), slow (lzma/7-zip), or moderate on both counts (zip). This means your ideal compressor will depend on how you value speed against efficiency.
• gzip is consistently faster but less efficient than bzip2. RAR is on all counts better than bzip2, so that’s probably why you have to pay for it
• The simple switches used give you a decent choice between speed and efficiency, however, the effects are usually smaller than the differences between different compression algorithms. gzip and zip use the same algorithm so compression ratios are similar, but the timing depends on the optimizations used in the actual implementation.
• Between the GNU compressors, the sequence lzop – gzip – bzip2 – 7zip gives you a wide range of speed / ratio trade-off. compress and zoo are obviously outdated, they are at the same time slower and less efficient than gzip.

Ranking

As we have already seen, your ideal compressor will depend on what you use it for. This ranking can be customized to what you want to do. It will compute the time required for 1 compression, a number of downloads over a network and the following decompressions.

total time = compression time + n * (compressed file size / network speed + decompression time)

For instance, if you compress a file to send it over a network once, n equals one and compression time will have a big influence. If you want to post a file to be downloaded many times, n is big so long compression times will weigh less in the final decision. Finally, slow networks will do best with a slow but efficient algorithm, while for fast networks a speedy, possibly less efficient algorithm is needed.

• One compression
• Network speed: kbps or 0 for infinite speed (i.e. no network transmission time is counted)
• download/decompression cycle(s), or -1 for only download and decompression (i.e. no compression time is counted)

Network speed: 1000 kbps    Downloads/decompressions: 1

Compression time    Transmission time    Decompression time

Optimal compressors

For any bandwidth / #downloads combination we can now determine the optimal compressor.

Related sites

• Ultimate Command Line Compressors, comparing almost all existing algorithms on efficiency, by Johan De Bock.
• Compression Links Info, a good all-round site about everything related to compression.
• The Wikipedia article on data compression

Results – table