Product Specification and QA

November 30th, 2008

Product development follows a very well defined procedure that is the same for any type of product being built:

1. Define
2. Develop
3. Test
4. Release

When defining a product, typically marketing and engineering define a product specification and go through the a long, drawn out process of locking down the definition before development actually starts.  There are a number of problems in this process though in modern product development:

1. Marketing typically has a very high level product requirement, but lacks much of the finer details that actually are required to get a product built and that defines a majority of the actual work being done
2. Engineering does not want to spend the time documenting requirements and technical plans/documents
3. Engineering will start to work prior to requirements being finalized due to both rapid development schedules and to pipeline the process while marketing requirements are still being defined
4. Test and QA are typically an afterthought.  It also is typically “reactionary” or based on previous failures found, instead of predicting where failures will occur

What is QA (quality assurance) and test really though?  Yes, it defines a set of tests that ensure a product meets a defined quality level, but what is the quality level definition?  Aren’t these procedures and quality definitions actually defining the product?  If so, why do we even bother with the process of writing requirements documents?  Why don’t we just specify a series of tests that define the requirement and guarantee it actually meets it?

I bring this up because of two examples I’ve seen in working on product development:

1) I’ve defined a high level requirement such as “product must play an mp3 song”, to later find out that what was delivered was a product that did play an mp3, except that it could not play many of the different bitrates or had no volume control.  You could argue this was due to lack of “requirements”, but the funny part is that it might pass through a QA process that was also defined based on “product must play an mp3 song” and did not factor in corner cases based on bitrates because the QA team had no idea on what bitrates were possible.

2) Certifications are a painful process that everyone hates, yet I’ve realized their value is establishing a base level functionality and guaranteeing the product meets it.  If there is no certification process, the only way to ensure a product actually performs the way it should is to have some sort of test process.  Either way, the test or certification now defines how the product should work.  To build the product, someone must take in to account all tests and ensure that they build something that will pass all of them.

This may seem very obvious, but I believe many companies still don’t actually understand what this truly means.  If you did, product definition should be altered to the following:

1. Requirements should not be written in a document form.  Requirements should actually be a series of tests that all result in either a pass or a failure.   The product should then be developed to ensure all tests pass.
2. To ensure the product meets marketing requirements, it means that marketing should own the definition and development of these tests.  A QA manager or QA plan definer should then be tightly linked to the product definer or should be the same person.
3. This also means no work can start unless test plans are in place

The main problem with this is that product definitions or requirements serve two purposes: to define the product as well as communicate the definition.   If someone is asked to review a requirements doc that is actually a very long list of tests, they will not understand what the original purpose of the product is.   Unfortunately, if you break the requirements documentation from the test definition, they will become out of synch.  I believe this “readability” aspect of the product definition is a small price to pay for the efficiency gain of changing the development process as I described.

Entry Filed under: Quality