Re: EVM2003 Status Report(s)

From: Alan Dechert <alan_at_openvotingconsortium_dot_org>
Date: Fri Nov 14 2003 - 01:58:46 CST


> Qa is kind of not there. I only have Jan's code to work with and
> zero documentation. I don't even know how to properly use the
> existing code. Without documentation, I don't know what to test.
This reminds me of an interview I had a few years ago (apologies in advance
if I already told this story): I had been a QA engineer at Borland and then
Intel. Later I interviewed with Netscape. They asked a lot of good
questions about my testing philosophy. I explained how the user
documentation, the specs, and the product have to all match, and I discussed
strategies for keeping them in synch. After a while, they asked. "what if
the product has already shipped but the specifications are not done?"

Even with software expected to be used in a production environment, the
documentation often lags seriously. For demo software produced by an
all-volunteer team, it's a bit much to expect really good documentation. I
started with some high hopes in this area but those hopes were quickly
dashed. Let's say the EVM2003 is a bit informal in this regard.

But it's not quite true to say there is zero documentation -- even if there
is zero documentation by some reasonable standards. We've probably overused
the email list and not given enough attention to organizing the things that
have been decided in the course of these discussions. But, in fact, a lot
of design decisions have been taken in the course of these discussions. So
the voting-project list archive itself represents a good part of EVM2003

Besides that, David wrote an architecture document -- while not complete
(and some of it has since changed) -- containing a lot of information. I
wrote a kind of requirements document that has been generally accepted with
a few minor changes:

Again, this document is not that great but there is a lot of criteria
spelled out to test against.

In short, you will need to use some creativity to develop some test plan.
Some of it is just common sense. For example, the BVA app has a bunch of
wav files. Can we look at the way this works and figure out if each wav
file is the right one? They are numbered sel001, sel002, ... sel116 where
each one represents, one after the other, one of the 116 possible selection
on the test ballot. Jan found one mistake. Are there others? Jan knew it
was a mistake not from any spec that was written, but because it was obvious
what should be expected given the scheme I was using. I promise to make
some corrections and post the new BVA app in the next day or so.

> For an overall plan I would like the following:
> (1) Some documentation on:
> a. What code should be used (filenames)
You may need to dig for this. We expect to have five PCs at the demo.

- Two of them will be nearly identical voting machines (one with mouse, one
with touchscreen -- they should work exactly the same way).

- One scanning station (BVA app)

- Tabulation app

- Vision-impaired voting machine

This amounts to four different standalone applications. You should be able
to start writing down some things to test with each of these apps based on
what you already know about them. We've heard very few specifics about the
tabulation demo so we should ask Arnie and Chris about it.

In addition to the four standalone applications, there is the web-based
demo. This is pretty much what we see here: The ballot GUI will be
different but the functionality is basically there.

> b. What's the expected behavior
This is a fair question. When in doubt, write down what you think is
reasonable. If others disagree, you'll find out pretty quickly.

> c. hardware/software needs
> i. What hardware/software does the code require (OS, cuecat,
I think the voting machines will be running Linux/Python -- of course we
need to get more specific there. I don't think it matters much what is used
for the BVA app or the tabulation app. I expect to use Windows (98, NT,
2000, XP, whatever) for BVA. Please ask Arnie and/or Chris what they will
be using for tabulation app. Only BVA uses the CueCat.

> (2) A set of acceptance critera
> a. This means what is necessary for passing QA. -- I'm willing to work
> Alan or someone else on this.
I will be glad to work with you on this.

> (3) I need to know who should get assigned what bugs.
Vision impaired voting machine -- Jan

Voting machine with touchscreen or mouse -- David/Anand (or whomever is
coding the ballot GUI) or Jan if it has to do with printing. Now there
could be a seam between GUI and printing that you might have to guess on.

BVA -- me

Tabulation -- Chris

On line demo -- Jan and/or whomever is helping Jan with the ballot GUI.

> This doesn't have to be extensive; these are the basic things I need to
> complete my task.
I will help you fill this out but you can do a lot of it by just typing
away. You can't always expect development to say how things are supposed to
work. It's normal for QA to have a lot to say about it.

--Alan D.

= The content of this message, with the exception of any external
= quotations under fair use, are released to the Public Domain
Received on Sun Nov 30 23:17:03 2003

This archive was generated by hypermail 2.1.8 : Sun Nov 30 2003 - 23:17:12 CST