Re: Open Source Quality

From: james_in_denver <james_in_denver_at_hotpop_dot_com>
Date: Sun May 09 2004 - 13:52:41 CDT

Doug,

Interesting read. I am not sure how much weight to put behind that study
though as it was published, as far as I can see four years ago, and was
based on software that is now nine years old?.

FWIW,

James

PS: Hopefully, Linux, and the other OS's have fixed many of these
problems?.

On Sun, 2004-05-09 at 12:57, Douglas W. Jones wrote:
> On May 7, 2004, at 10:25 PM, Edward Cherlin wrote:
>
> >> Doug told me about a Wisconsin study of commercial Unix v.
> >> open source workalikes.
>
> Here is the original study, thanks to Google:
>
> An Empirical Study of the Reliability of UNIX Utilities
> Barton P. Miller
> Lars Fredriksen
> Bryan So
>
> Communications of the ACM, 33, 12 (Dec 1990) 32-44
>
> Summary
> Operating system facilities, such as the kernel and utility
> programs, are typically assumed to be reliable. In our recent
> experiments, we have been able to crash 25-33% of the utility
> programs on any version of UNIX that was tested. This report
> describes these tests and an analysis of the program bugs
> that caused the crashes.
>
> And, here is the followup study:
>
> http://www.opensource.org/advocacy/fuzz-revisited.pdf
>
> Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities
> and Services
> Barton P. Miller
> David Koski
> Cjin Pheow Lee
> Vivekananda Maganty
> Ravi Murthy
> Ajitkumar Natarajan
> Jeff Steidl
> 2000
>
>
> Abstract
> We have tested the reliability of a large collection of
> basic UNIX utility programs, X-Window applications and
> servers, and network services. We used a simple testing
> method of subjecting these programs to a random input
> stream. Our testing methods and tools are largely
> automatic and simple to use. We tested programs on nine
> versions of the UNIX operating system, including seven
> commercial systems and the freely-available GNU utilities
> and Linux. We report which programs failed on which
> systems, and identify and categorize the causes of these
> failures.
>
> The result of our testing is that we can crash (with core
> dump) or hang (infinite loop) over 40% (in the worst case)
> of the basic programs and over 25% of the X-Window
> applications. We were not able to crash any of the network
> services that we tested nor any of X-Window servers. This
> study parallels our 1990 study (that tested only the basic
> UNIX utilities); all systems that we compared between 1990
> and 1995 noticeably improved in reliability, but still had
> significant rates of failure. The reliability of the basic
> utilities from GNU and Linux were noticeably better than
> those of the commercial systems.
>
> We also tested how utility programs checked their return
> codes from the memory allocation library routines by
> simulating the unavailability of virtual memory. We could
> crash almost half of the programs that we tested in this way.
>

==================================================================
= The content of this message, with the exception of any external
= quotations under fair use, are released to the Public Domain
==================================================================
Received on Mon May 31 23:17:27 2004

This archive was generated by hypermail 2.1.8 : Mon May 31 2004 - 23:18:16 CDT