Excel as a Test Management Tool

Because you're going to use it anyway...

Ruby-Selenium Webdriver

In under 10 Minutes

%w or %W? Secrets revealed!

Delimited Input discussed in depth.

Managing Knowledge Transfer Sessions

Read the article, grab the templates

Ask questions

If you don't ask, you won't learn. You won't teach either.

Saturday, 8 November 2008

China - Software Testing Club

Over at http://www.softwaretestingclub.com/ I just set a club up for China.

I have a special interest in China (having family there) and in Software Testing and I have a problem - finding out about Software Testing in China is like haystacks and needles.

At a Client site recently I encountered a number of Chinese linguists doing translation work for software. It occurred to me I've never encountered a Chinese Software Tester in the UK. Indian sure, Chinese, they don't exist it seems.

I know off hand of two companies that are involved in offshoring test and development to China, CSC and Bleum. I know a tonne of them in India.

I know of Indian testing qualifications, universities that teach software testing to some level, I know a few Indian testing luminaries too. I have no idea about courses, conferences, institutions or guru's (chánshī) in China. (Slight lie, I know of this training provider but it's in Chinese so am working off dodgy Google translations http://www.beidaqingniao.org/)

Is ISTQB established in China yet? I've contacted the person marketing it over there, we'll see what response I get.

Friday, 17 October 2008

Test Estimation - Research and Thoughts

Test Estimation, what is it and how to do it.

I think this easily joins the list of "Top 10 Most Asked Questions" that would be put together by trawling any popular forum or testing community website. There's an idea...

Someone being able to estimate, a) How long it will take to write Test Cases and b) How long it will take to execute them; is a fundamental for any test team. This assumes the person doing the estimation is reasonable at the test analysis activities or has someone in the team who is, a Test Analyst maybe?

Estimation has a number of factors of course; complexity and scope of the testing needed and of the item under test, number and competency of testing staff, the impact of development on the testing effort or more accurately the impact on the quality of the test item, the list goes on.

In Q1 2008 I had a sit down with myself and drew together a collection of thinking I'd done around this topic. I'd hoped to formulate a model that was sophisticated enough to be of use but not so convoluted that it was unusable. I've seen a lot of vague discussion about estimation and a lot of nice but complex theory that would never be usable in the heat of a project.

One usable approach can be seen in Tony's Video and there's even a worksheet on his website.

From my side I did write a paper on Test Estimation though it contains several approaches and not one - I just knew it would the moment I started writing. That I believe reflects the main issue of why we keep talking about this topic, there's no one-size-fits-all and I suspect that the 'answer' is more likely a set of guidelines and approaches, not axioms or rules.

Apply these in context of your environment to make them relevant and augment them with additional guidelines that make your approach work for you, in a way that you understand.

My paper and a JavaScript/HTML paper is posted
on the site here. It contains an image of how to complete the HTML page too as it may not be overly clear.

One other approach is to do a decomposition of the work to be done. During test analysis we'll usually try and identify all of the individual peices of functionality to be tested. Creating a collection of scenarios or test objectives for each of these and then defning a set of Test Cases. From here we can work out the individual timings for writing the Test Cases and then Executing them. In total we'll have our estimate for test prep and running.

Here's a copy of a Test Estimation Workbook that could be used for this approach. Perhaps using this after the 'main' estimate is done might be interesting. I wonder if we'd have the time!

This approach means we can also track planned against actual effort to provide some clever Measures & Metrics reporting and use the data to feedback into our future estimates.

Mark 'roughly this big' Crowther

Monday, 29 September 2008

20th UK Test Management Forum

I just noticed details for the next UK:TMF have been posted:


This is the free and informal meeting that happens each quarter. Well worth a visit to enjoy a conference that's run in a way that encourages open debate and knowledge sharing.

Hope to see some of you there.


Monday, 22 September 2008

Communication Training or your life

MSB Executive http://www.msbexecutive.com/

I've just had one of the hardest days of training, heck one of the hardest days, I've ever had.

NMQA kindly set me up with Martyn "I will rebuild you" Barmby of MSB Executive for a day of Communication Training.

It started gently enough with some discussion about how I approach presentations, why I dislike sales (activity, not people) and moved into everything from how I stand and walk, enter a room, make small talk and greet people.

Later on we covered the SPIN technique (Situation, Problems, Implications, Needs) which can be used to structure presentation of products and services. Then came the role play, walking about while speaking to control breath and delivery, followed by a 'therapy session' on any mental blockers I might carry about with me that impede my performance.

This type of training is so hard because in order to benefit the most you need to set-aside what you think and know and think you know. You need to take on board every comment, engage in activity that makes you feel uncomfortable. It drains you physically and emotionally.

Has it helped?

Following the training I had a Client meeting where they opened with "You probably can't help us.." and ended with them agreeing product presentations, collaboration on developing testing standards for thier industry and sending a pilot job to us as a trial. Yes, it helped!

The next meeting with another Client I proposed a spend on testing equal to 10% of total annual revenue. Hey, testing's important dammit ;] I gave the best presentation of my life, addressed all needs and managed all issues. They agreed to the proposal and we're now two people on site.

I admit it, Martyn has taught me how I can enjoy Presentations AND Sales, wonders never cease.

Yes, the training works. This is the website you need:


Mark "I should have had this training 10 years ago" Crowther.

Thursday, 3 July 2008

The Cost of Quality

Read the Cost of Quality Discussion Paper

One of the key areas of Quality Management that I try to promote is the understanding of quality costs to the business. It's all too easy to know there are problems with the business operations, be those Software Test processes or Software Development, but never to really understand how much they affect the bottom line.

The paper provides a worked example based on public information available for EA Inc. As stated in the paper this applies to every business and the principles discussed can be applied to SMEs. In fact there are two good examples that I use time and again as to why SMEs must understand the Cost of Quality principles:

  • It's critical that SMEs see maximum benefit from often rarefied financial resources. They must remediate money wasting practices now and avoid wastage scenarios in the future. Addressing the Cost of Quality can provide both the remediation activities and provide an avoidance strategy.

  • An SME that is experiencing serious growth or is holding a strategic piece of the market may be a candidate for acquisition. They should understand the Cost of Quality to ensure investors are shown the management have these costs under control.

The actual experience however is that a business won't embark on a Cost of Quality programme, why? Many reasons such as fear of the unknown, can they face up to knowing and do they have the drive to complete the programme? If the business appears to manage costs enough to make profit they may feel a lack of motivation, not a world class attitude but very common.

What's discussed in the paper is a big step for any business to take. The rewards are great but the work is long and hard. However, the trick is in adopting the Cost of Quality mentality from day one. Then a programme never starts as something separate, it's just a natural part of the business practice as are reaping the benefits and avoiding the future issues.

Mark Crowther, Manager of Costs

Tuesday, 13 May 2008

QA Survey, Interview Questions, Awareness Campaign?

Some time ago myself and a colleague were asked to brainstorm a list of questions that might be used in an internal survey. The purpose of which was to assess how aware the organisation was of what the test team did.

It was one of those great 1 hour brainstorms that produced a peice of work that surprised both of us.

Having been searching my PC for something entirely different it was a pleasant surprise to rediscover it. So, before I lose it again I thought it might be wise to share it ;)

Enjoy using these and add any in the comments that you think of yourself.

1) Do you know what type of reporting QA provide during a development project?
2) Are you aware of any reporting or information that is available when QA are not engaged in a development project?
3) Do you know if QA publish any information such as what they are working on or who is working on it?
4) Are you aware of what could be provided using any reporting systems we might have?
5) Are you aware of any reporting systems QA might have?
6) Have you ever seen reporting for areas such as Bug Quantity found, % complete of the test cycle, % code coverage?
7) Are the above or similar of use to you? How?

8) Is there a process for requesting testing / test resource from QA?
9) Do you know what type of testing QA provide during a development project?
10) Are you aware of the testing that QA perform when not engaged in a development project?
11) What reasons do you know of for QA selecting a type of testing to deliver?
12) Can you name any dependencies QA have from other teams that help them deliver the testing?
13) How do these dependencies impact QA delivering the testing?
14) Is it clear why testing takes the time it does?
15) When a Test Engineer isn’t testing, do you know what they are doing?
16) Is it clear how early QA should be involved in a project to deliver effective testing?
17) Do you know if and how QA use Requirements and Design documents?
18) Do you know what items or documents QA create when delivering testing?

Bugs and Issues
19) When do QA submit bugs?
20) Is it clear why QA will submit bugs?
21) Against the above do you know of any reason when we wouldn’t submit a bug?
22) Is the process for managing bugs through QA and dev clear?
23) Do you know the process for deferring bugs?
24) And closing them?

QA and Testing Knowledge Share
25) Has it been made clear what the ‘Software Test Life Cycle’ is?
26) Are you aware of any other practices and processes QA have in place?
27) Is so, do you know where they are published, if at all?
28) Do you know what tools QA use to aid testing?
29) Do you know if QA, development and documentation use the same tools?
30) Do you know how the outputs from tools get used by different teams?
31) Do you know how many Test Engineers are in the team?
32) Are you aware of how they are assigned test work?
33) Do you know if QA have relationships with development teams?
34) If QA do, is it clear why?
35) If QA don’t, do you believe QA should?

36) Have you ever attempted to build a relationship with the QA team?
37) If you have, how did this go?
38) What would you suggest is the single most useful thing QA do?
39) What would you suggest is the single most useless thing QA do?
40) Do you believe QA add value to a project?
41) If not, why not?
42) If so, how?
43) Can you name a project that would be in a lesser state of quality without the input of QA?
44) Can you think of a project where QA didn’t help or perhaps were detrimental to the project?
45) If QA could make one change to what we do, what in your opinion should it be?
46) Do you think it’s clear to QA what your team do?
47) Come to think of it, what is it you do? :]
48) Do you believe QA should be proactive in chasing development or development should come to QA?
49) Do you see QA as a potential pool of resource for adding to development, a stepping stone into development?
50) Should QA provide guidance to the developments teams on how to test their own software?

Wednesday, 16 April 2008

Test Case execution priority

A question put to me recently:

"If you have Test Cases already prepared and you have to just excute those test cases in a very short time then how will you decide which test cases has to be excuted first. Is there any technique ?" - Mrityunjay

If you have more Test Cases than you have time to run then you clearly need to put them in some priority order to ensure you run the 'Most Important Tests'.

I think it's rare that the test team can do this as we don't have a complete view of the importance of all development features that are the Most Important.

Approach 1

As soon as you have them written get together with people that can decide their priority such as the Architect, Product Owner or Project Manager and Walkthrough the Test Cases. Ask them to put them in order for you. Ask them "If I test nothing else, if I show nothing else works - what would you want?". Then when you have that ask them "If that was done, what would be next?" until you're done. This meeting should be 30 minutes tops.

Approach 2

If the test team really have to choose then there's analysis needed to decide the selection of the cases:

  • Risk - Does the Test Cases cover an area where there's been a lot of change or where many issues are usually found?
  • Importance - Does the Test Case cover an area that absolutely must work and/or will always be used?
  • Breadth - Does the Test Case cover functionality broadly and is a high level test over a set of lowere level, alternate Test Cases?

If you have a Test Case that is covering critical functionality in a high risk area then you need to run it. If you have a Test Case that is covering an area of minor change across functionality hidden several menus deep that will be used by a small group, leave it to the last.

Of course now you need to make sure you're reasonably accurate in your estimation of how long test execution will take. This will ensure you can declare what test will be executed and which wont be.

Thursday, 27 March 2008

Test Estimation, take a step back

As test professionals we're expected to answer 'off the cuff' how long something will take to test. It must be the single most common dreaded moment for any test lead or manager. The projects starting and we're expected to know ho long all test tasks will take, including the identification, authoring and execution of tests. Sadly, 'How long is a piece of string...' doesn't go down well nor does sarcasm about how it's pointless estimating because after the project owner/customer/developers delay/change/fail to write documents, etc. the estimate will be worth poop. As true as it might be.

So, these 'challenges' aside how can the test professional at least arrive at a reasonable set of estimations for the test tasks? Firstly we need to know what tasks are needed. It's seems right that we should be able to break down a delivery of testing into a set of known or expected tasks. Having those identified we could then estimate how long each would take.

Taking a step back - it's essential that the test manager is clear on what tasks should be performed for a given project. That comes through designing a test methodology or several which can be applied to various project types.

A further step is to model the effort needed to deliver each task. For example, writing the Test Plan. Though it's impossible to estimate with perfect accuracy the test manager cannot make this up every time. They must have 'model' answers that can be refined in context of the project.
When I get folks who are convinced they can't possibly say how long I ask: 'How long does it take to write a test plan?' - 1 month? 1 hour? Well no. 1 week then? Too much. With all reviews, maybe 1 week start to end. The effort for this? By the test manager maybe 2 days tops. Great, so my model answer is a Test Plan takes 1 week to deliver and requires 2 days of effort by the test manager.

Test Case identification and authoring is more difficult as the complexity and number of cases needed is entirely context dependent. There's a greater reliance here on the project being run in a way that delvers such artefacts as good requirements documents. Ah yes, back to that old chestnut.

Next time I get a project I'll think about it's complexity, the competency of staff, etc. and provide a duration/effort in context of the project. But I have my model answer as a start point so I can arrive at reasonable estimation more quickly. Rinse and repeat for every other task in your methodology and you have a general estimation model(s).

Tuesday, 1 January 2008

Oh monkies, a Survey!

If you're part of the QA & QC Google Group you'll have noticed my posting about analysis techniques and practices I put up there some time ago. Actually, you probably didn't notice as no one replied. But it's OK, I'm not (that) bitter. To help folks I've put together an anonymous survey. Non, no, don't thank me.

A few months ago I read Lesson 32 (chapter 2 generally) and section I-8 of the NRC Fault Tree Handbook and I felt I was getting a real insight into analytical techniques. You know those things you do naturally to some degree, but you do them in a crude fashion and you're not actually sure if they have a name? That was me, heck still is!

Well here's the shimmy, I've been studying the topic of testing requirements analysis for a few months now. Part of what I'm trying to understand is what we 'really' do in this area. This really is one of those areas where we know what we 'should' do but rarely do or get the chance to do.

There's a link to the survey and pop-up on my website. The vocabulary I use in the survey is ambiguous on purpose as I'm trying not to 'lead' people in their answers. I'm no professional survey author so hopefully it wont confuse you too much! It’s only 10 questions but you’ll see you can add as much extra detail as you wish.

Drop me a mail with your critique of the survey too. I'm very keen to see how these can be used as a tool to get more insight on topics of interest.

Remember the survey is anonymous so I can’t thank you individually, you can always email me of course! So, thanks in advance and I'll make sure I share the information here once I have something meaningful together.

Mark "and just another 50 questions" Crowther