Using the StackOverflow public data set (the October 2009 one, which I still had lying around from last time), I set about plotting the number of questions versus the number of answers contributed by each user. This graph shows the raw results:

Clearly, several very active users tend to ‘specialize’ towards either the Questions or the Answers axis. One user is even up to almost 800 questions, but barely gave any answers. Can you earn reputation points that way? Sure, if they are good questions, other people will pass by that have hit the same dead-end and express their eternal gratitude of seeing their problem already solved by voting the question up. Here’s the same graph from before (except for the log-log axes) but including colors to show reputation classes:

The next graph plots each user’s reputation versus their “Q-to-A ratio” (their number of questions divided by their number of answers). For users without any answers I just plotted their number of questions (red dots).

The vast majority of users give much more answers than they have asked questions (so they have a Q/A-ratio of less than one) and inhabit the region near the horizontal axis. Most high-reputation users also tend to answer more than they ask. In the lower reputation regions (<1000) much higher Q/A-ratios can be seen (including those with an ‘infinite’ ratio, i.e. with no answers).

(As I found out later, someone else had already computed the top Q/A-ratios on an older SO dump, although I always prefer looking at things with some graphs…)

Finally, here are some Q/A-ratio distributions: the first one shows the number of users in each Q/A-ratio class. (Never mind the fact that the 1/1 class seems extraordinarily large in comparison to it’s neighbours, I’ve used a pretty strange class grouping function (an arctangens, so it can show both 0 and infinity symmetrically around 0) to make it fit properly on the graph.) The distribution is clearly weighted towards the right, again showing that most users have contributed more answers than questions.

We can also weigh users by their reputation, so we get the fraction of total reputation that is represented by the users with a certain Q/A-ratio. The already large group on the left side of the Q/A scale becomes even more important now, meaning that most reputation is owned by users with a low (<1/10) Q-to-A ratio. This is confirmed by the next graph which shows the average reputation of all users in a given Q/A class.

It’s easy to do this using tags: some sort of clustering should be applied according to how often each pair of tags shows up at the same question (a user that knows both ASP and ASP.net shouldn’t be considered a ‘diverse’ person, so this should be factored out first), next we can count in how many different clusters that this user has contributed a good answer.

I’ve had a stab at trying this on the last (October 2009) StackOverflow public data set. I’ve ignored the SU and SF parts of the dump. The idea is to count how many of SO’s questions you could conceivably answer, given your proficiency in each of the tags of that question.

First I’ve scored all answers each user has given. 20 points go to it being the accepted answer, another 80 points are distributed over all answers to a question in relation to the answer’s votes. This is not exactly the same as the reputation earned for that answer, since popular questions get a lot more up-votes to their answers, but in this analysis this isn’t really worth much more than a good answer to an unpopular question.

The points for all your answers are distributed over the tags with which the question is tagged. This gives each user a number of points for all tags. These points are converted into a proficiency that user has for this tag: proficiency = 1 - exp(-points / 500). So 1000 points (10 very good answers) gets you to 86%, more points will asymptotically get you to 100%.

At this point we can compute the average proficiency over all tags for each user, which is plotted in this graph (average tag proficiency versus reputation, for users with reputation > 1000):

The next step is to compute the question proficiency. This is done, for each question and each user, by taking the geometric average of this user’s tag proficiencies over all tags that this question is tagged with (I’ve choosen the geometric average rather than the arithmetic one, since you’re supposed to be knowledgable on all components of a question in order to answer it). These per-question proficiencies can again be averaged (arithmetically), yielding the user’s average question proficiency which is a measure for how many of the site’s questions he/she could answer. This graph plots average question proficiency versus reputation:

Note that this last one is a fairly heavy query (~1 second per user), so it could more feasible to base an actual implementation on tag proficiency only (see also this graph of tag vs. question proficiency), although this would overvalue knowledge of topics SO doesn’t care much about.

Analysing the question proficiency vs. reputation graph, we see that both are clearly related (if you answer enough questions, you’re bound to have covered most of the tags). Still, some users reach a possible threshold value of 20% question proficiency at a reputation of only 10,000; while others couldn’t get there even at 60,000 reps.

For the following set of graphs, I downloaded the XML file for the current TOP500 list (since updated to the November 2009 list). I decided to look at each system’s LINPACK performance (Rmax) compared with its peak performance (Rpeak) which is what you get when adding the theoretical performance of all processor cores in the system. The result should be a measure of how efficiently the system’s total computational power can be used, which is mostly a function of the interconnect (also software, but for a TOP500 listing we would expect that to be rather good!).

So here is the graph:

Here’s the same data but this time efficiency (Rmax/Rpeak) is plotted directly:

For top-of-the-list machines, Infiniband and proprietary interconnects (mainly BlueGene, Cray and SGI’s NUMAlink) are the most common. Their efficiency is rather similar at 75-80% for the large machines, and up to 95% for smaller ones. The main alternative interconnection technology, Ethernet, only starts at to be in common use from system #98 onwards. Clearly, the efficiency of these systems is significantly less, at some 55%.

Two main outliers to this theory can be seen. System #5 uses Infiniband but manages to get only a 46% efficiency. This is the Tianhe-1 cluster located in Tianjin, China. It consists of 5120 ATI Radeon HD 4870 video cards, in addition to a number of Intel Xeon CPUs. I would guess that the software for this rather radical new architecture (especially at this scale!) is not yet fully up-to-date, and that this system should get a boost in Rmax by the time of the next TOP500 list next June.

The other exception is #486, again a Chinese system, which uses a 10-Gbit Ethernet network (rather than 1-Gb Ethernet for the others – at least, as far as I could tell from parsing the list). Clearly, the faster Ethernet technology significantly improves efficiency: at 70% it comes much closer to the Inifiband and proprietary-interconnect powered systems. When compared to 1GbE, the difference between 10GbE and Infiniband is minimal though, and also the price comes pretty close. So I guess, in the end, you do get what you pay for…

Instead I’ll be focusing on very well known, established compressors that are easily obtained (I only use precompiled packages and won’t build from source) and have reasonable run times. Also I won’t try every combination of compression options but limit the test to one general option (-1 to -9 for gzip and bzip2, -m1 to -m5 for RAR, …).

Updates

• 2007/04/25: Updated 7-zip to version 4.45: small improvements in speed and compression ratio
• 2006/06/08: Reran all tests on a Pentium D 830 (dual-core, 3 GHz, 2x 1MiB L2) and 2 GiB RAM
• 2006/02/13: Updated 7-zip to version 4.33: both faster compression (up to 20%) and better compression ratios (a few %)
• 2005/11/27: Updated 7-zip to version 4.30 beta: high compression (-mx=7 … -mx=9) is now up to 40% faster
• 2005/11/03: Updated 7-zip to version 4.29 beta: no changes
• 2005/04/12: Updated 7-zip to version 4.16: the ratio is the same, but compression and decompression are now 30% to 40% faster!
• 2004/11/10: Added the 7-zip compressor
• 2004/09/30: Added the lzop compression algorithm

The Test

Just like Johan De Bock’s excellent GIMP Source Compression Test, I’ll be using the GIMP 2.0 Source tarfile:

• The test file: GIMP 2.0.0 Sources as one TAR
• MD5-Hash: d2a1c33317fb57bbed3641671b2da163
• Total size: 78,745,600 Bytes

Each program is used to compress and decompress the file with each of the selected command line switches. The time required for both compression and decompression is measured, as is the size of the resulting archive. To have an idea of the accuracy of the timing measurements, the test is repeated three times and the minimum runtime (= the one with the least disturbances) is reported. All this is done on a Pentium D 830 (3.0 GHz) processor with 2 GiB RAM, running Fedora Core 5 Linux. Only one CPU is used for the tests (using -mmt=off for 7-zip, default for other compressors).

The Contenders

The versions aren’t always the latest and the greatest, but they are – again in the spirit of being ‘practical’ – the most recent ones installed by the Fedora Core 5 distribution.

Results – graphs

These graphs show the results for the various compressors and their switches. Compression and decompression time, respectively, are in the horizontal axis, compression efficiency (compressed size / original size, so smaller is better) is on the vertical axis.

Conclusions

• The best compressor can be found near the lower (efficient) left (fast) corner. You’re right: there isn’t any! All real compressors are either inefficient (gzip), slow (lzma/7-zip), or moderate on both counts (zip). This means your ideal compressor will depend on how you value speed against efficiency.
• gzip is consistently faster but less efficient than bzip2. RAR is on all counts better than bzip2, so that’s probably why you have to pay for it
• The simple switches used give you a decent choice between speed and efficiency, however, the effects are usually smaller than the differences between different compression algorithms. gzip and zip use the same algorithm so compression ratios are similar, but the timing depends on the optimizations used in the actual implementation.
• Between the GNU compressors, the sequence lzop – gzip – bzip2 – 7zip gives you a wide range of speed / ratio trade-off. compress and zoo are obviously outdated, they are at the same time slower and less efficient than gzip.

Ranking

As we have already seen, your ideal compressor will depend on what you use it for. This ranking can be customized to what you want to do. It will compute the time required for 1 compression, a number of downloads over a network and the following decompressions.

total time = compression time + n * (compressed file size / network speed + decompression time)

For instance, if you compress a file to send it over a network once, n equals one and compression time will have a big influence. If you want to post a file to be downloaded many times, n is big so long compression times will weigh less in the final decision. Finally, slow networks will do best with a slow but efficient algorithm, while for fast networks a speedy, possibly less efficient algorithm is needed.

• One compression
• Network speed: kbps or 0 for infinite speed (i.e. no network transmission time is counted)
• download/decompression cycle(s), or -1 for only download and decompression (i.e. no compression time is counted)

Network speed: 1000 kbps    Downloads/decompressions: 1

Compression time    Transmission time    Decompression time

Optimal compressors

For any bandwidth / #downloads combination we can now determine the optimal compressor.

Related sites

• Ultimate Command Line Compressors, comparing almost all existing algorithms on efficiency, by Johan De Bock.
• Compression Links Info, a good all-round site about everything related to compression.
• The Wikipedia article on data compression

Results – table