"I thought it only fair to do an Open Benchmark giving them everything they ask for," says Weiner -- whose company has come under sustained and virulent criticism from the Linux community in recent weeks. "At that point, they cannot reasonably claim that the benchmark was biased."

But the immediate response from leaders in the Linux world can be summed up succinctly: No way in hell. As far as Linux creator Linus Torvalds, leading Linux hacker Alan Cox and Red Hat CEO Bob Young are concerned, there's no advantage to be gained from participating in what they consider to be a game that's still rigged in Microsoft's favor.

"The entire test is engineered to favor NT," says Alan Cox, who laid out his technical objections in a recent LinuxToday article. "His 'open test' is simply tying the open software community's hands while providing a facade of openness."

On April 13, Mindcraft announced the results of its first test comparing NT and Linux. NT, Mindcraft said, performed significantly better in key areas such as file- and Web-serving. The Linux community immediately erupted -- and with some justification. As was eventually discovered, the test had been commissioned by Microsoft and conducted in a Microsoft computer lab with the help of at least one Microsoft independent contractor.

Within weeks, Mindcraft announced that it was redoing the test, this time with the help of the "top minds" of the Linux community. But, as Salon reported on April 27, Torvalds and Cox both expressed extreme dissatisfaction with the amount of direct input they were allowed. No outside observers would be permitted at the test, and they said they weren't being given enough information to make a difference.

Microsoft, says Weiner, is withholding the results of the second test until a third test can be carried out. This time, he says, Linux representatives will be encouraged to be physically present at a neutral site.

"Mindcraft wants to show we really did run a fair and accurate benchmark the second time," says Weiner. "We wanted to do that the first time but we unfortunately did make some mistakes tuning Linux, Apache and Samba. Despite the flames from many Linux proponents, we really do want to show Linux, Apache and Samba at their best on an enterprise-class system."

But Torvalds and Cox say that there are still some serious problems with how the new test is set up. One of the conditions set by Mindcraft is that no new patches or fixes can be incorporated in Linux that weren't already publicly available on April 20. This isn't fair, says Torvalds: For one thing, one of the main strengths of Linux is the ability of its open-source community to respond with new patches and fixes for every problem that arises. Even worse, adds Torvalds, Microsoft had the opportunity to "tune" NT for the particular hardware configuration that will be used in the test "for years."

Next page | Torvalds: "It's simply the same thing all over"