Open Source Quality

From: Douglas W. Jones <jones_at_cs_dot_uiowa_dot_edu>
Date: Sun May 09 2004 - 11:57:32 CDT

On May 7, 2004, at 10:25 PM, Edward Cherlin wrote:

>> Doug told me about a Wisconsin study of commercial Unix v.
>> open source workalikes.

Here is the original study, thanks to Google:

An Empirical Study of the Reliability of UNIX Utilities
   Barton P. Miller
   Lars Fredriksen
   Bryan So

Communications of the ACM, 33, 12 (Dec 1990) 32-44

Summary
   Operating system facilities, such as the kernel and utility
   programs, are typically assumed to be reliable. In our recent
   experiments, we have been able to crash 25-33% of the utility
   programs on any version of UNIX that was tested. This report
   describes these tests and an analysis of the program bugs
   that caused the crashes.

And, here is the followup study:

http://www.opensource.org/advocacy/fuzz-revisited.pdf

Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities
and Services
   Barton P. Miller
   David Koski
   Cjin Pheow Lee
   Vivekananda Maganty
   Ravi Murthy
   Ajitkumar Natarajan
   Jeff Steidl
     2000

Abstract
   We have tested the reliability of a large collection of
   basic UNIX utility programs, X-Window applications and
   servers, and network services. We used a simple testing
   method of subjecting these programs to a random input
   stream. Our testing methods and tools are largely
   automatic and simple to use. We tested programs on nine
   versions of the UNIX operating system, including seven
   commercial systems and the freely-available GNU utilities
   and Linux. We report which programs failed on which
   systems, and identify and categorize the causes of these
   failures.

   The result of our testing is that we can crash (with core
   dump) or hang (infinite loop) over 40% (in the worst case)
   of the basic programs and over 25% of the X-Window
   applications. We were not able to crash any of the network
   services that we tested nor any of X-Window servers. This
   study parallels our 1990 study (that tested only the basic
   UNIX utilities); all systems that we compared between 1990
   and 1995 noticeably improved in reliability, but still had
   significant rates of failure. The reliability of the basic
   utilities from GNU and Linux were noticeably better than
   those of the commercial systems.

   We also tested how utility programs checked their return
   codes from the memory allocation library routines by
   simulating the unavailability of virtual memory. We could
   crash almost half of the programs that we tested in this way.
==================================================================
= The content of this message, with the exception of any external
= quotations under fair use, are released to the Public Domain
==================================================================
Received on Mon May 31 23:17:27 2004

This archive was generated by hypermail 2.1.8 : Mon May 31 2004 - 23:18:16 CDT