Advise for bug hunters

We organized a bug hunt for our latest release, inviting developers, PMs, pre-sales, tech writers and IT people to test with us in short fun sessions.

I wrote these tips for the non-testers - see if you want to add something:

A few tips on how to be successful in bug hunting, collected from veteran hunters:

Be pessimistic, but energetic

you know that everything that can go wrong WILL go wrong, but you do it anyway (and report it)

2. Don’t look for bugs under the street-light

go for the less travelled paths

3. Think negative

The positive paths are working

4. Size matters

Go for big inputs (files, compliance content, number of jobs in workflow, etc.)

5. Do it again, and again

Jobs will run the first time, but the second time?

Create, Delete, Re-create – BOOM!

6. Be quick, but thorough

Use existing resources (user guide, SDD) for quick checklists

Good luck

Picture by http://www.flickr.com/photos/slmatthews

A bit about ITIL

The company I work for creates software that help organizations with huge IT infrastructures manage their IT and tie it very strongly to their business needs.

As IT organizations grow mature and stressed to improve and bring more value for money, we see they tend to look for "best practices" for managing their IT business. ITIL is one of the sources they go to, so we decided to learn ITIL in order to "speak" the language our customers are speaking and understand the way they see their domain.

ITIL stands for Information Technology Infrastructure Library. It is actually a collection of books, describing different aspects of managing an IT organization. It is organized into several parts, including:

• Service strategy - how to decide what services to provide to bring value to your customers
• Service design - how to design and implement the service
• Service transition - how to move the new or updated service from the lab to production
• Service operation - how to operate the service and support the users on an ongoing basis
• And...
• Continuous service improvement - how to keep improving these process and make them even better (maybe "bester"?)
  

BTW, "Best Practices" reminds me of dilbert, asking the pointed hair boss: if everybody is doing a "best practice", doesn't it become a "mediocre practice"?

I rather enjoyed the training - we had an excellent instructor with YEARS of tough industry experience, and excellent examples for everything.

To read more about ITIL go:
ITIL official site - http://www.itil-officialsite.com/AboutITIL/WhatisITIL.asp
Wikipedia - http://en.wikipedia.org/wiki/Information_Technology_Infrastructure_Library

ISTQB? Check. ITIL? Check

I am now ISTQB and ITIL foundation certified, and got an observation about these certifications. In both cases I studied and took the exam with people that are new to the field of IT and software testing. I saw that for the experienced pros, the study was focused on agreeing on words and terms to describe processes and objects they work with daily. These falks had a rather easy time when preparing and in the exam itself.

The other part of the crowd were inexperienced and tried to learn the glossaries by-heart, to memorize them the best they can for the exam. Some of them didn't manage to do so. I believe that learning glossaries by-heart, without an experience base to tie the concepts to is futile.

The test itself was hard for non-English speakers. It is a time-limited multiple-choice test, filled with semi-tricky questions. I happen to be quite good at this sort of tests, but others were dissapointed and said the ITIL test was "nasty".

Bottom-line: in the future, I will recommend 2 things to my team members and collegues:
1) go do the certification when you have enough experience to make it valuable
2) arrange study groups - this way you can share experience with others and enjoy the company while learning for the exams

P.S.
The ISTQB certification comes framed in a nice glass frame, looking very professional - a lot more "impressive" than the academic degree that took me years to complete. Feels more like PR than a certificate of acheivement...

More bugs in the ISTQB certification exam

In the last post I wrote about 2 bugs found in the ISTQB certification exam. Well, turns out there was another one, or actually 2:

1. In the ITCB site (Israeli Testing Certification Board), it said that results will be emailed after 2 weeks. The instructor that supervised the exam told us it would take 3 weeks. This sort of bug is called an inconsistency bug. The main effect on inconsistency bugs is user confusion and loss of trust.
2. The results were promised 3 weeks after the test. This date is behind us now. This is a bug, since a bug can be defined as a gap between user expectations to actual behavior.

So, at this point I am a bit dissapointed of not getting a score for a 40 question multiple selection exam in 3 weeks. The surprise for me is that I am actually waiting to get the score - not so cool :-P

ISTQB certification exam - done

I took the ISTQB foundation level last Thursday. I feel pretty confident about my answers, but must admit that I had to guess a few answers.

As you might have guessed, you cannot gather a bunch of testers in a room without them finding bugs and down-talking the product quality :-) Got 2 bugs in the test:
1. Medium (and resolved on the spot): 1 character ommited, but it was a significant one, differentiating between sentence II to sentence III...
2. High (and gave us a free correct answer): a question regarding a certain decision table did not include... the decision table! Well, we are going to get this one for free.

Some updates and promises for posts-to-come

Been away for a while - funny thing of broken habbits: our wireless router broke down, and I didn't manage to fix it (or get myself to get a new one), so I stopped connecting to the internet with my laptop when I am home. Immediate result: lost my home-internet habbits. Call it vacation. (BTW, I write this post from the kids PC).

Things that I need to update about in posts-to-come:

1. ISTQB self study group: certification exam is next week (July 10th). I feel ready, and got things to say about the ISTQB sylabus.
2. Ruby: had some progress (thanks for the reference to Aptana studio Rich!), need to share something I don't get in Ruby, as well as rcov - a nice (too?) simple code coverage tool.
3. SIGIST conference: been to the Israeli Special Interest Group In Software Testing (SIGIST) conference. I gave a presentation about how to extract test cases from UML.
4. Mind maps: I am hooked up on mind maps, using them as my new external brain and memory. Is this a passing thing?

Cool things in Ruby

I have been using Ruby for quite some time now, but only now I begin to see what I missed...

It started when we began using WATIR for shooting HTTP transactions to test our application. WATIR is based on Ruby, activating Internet Explorer's COM object - I recommend it for quick useful scripting.

What caught my attention was the Ruby under WATIR - the scripts were easy to write, readable, and contained clever "tricks" that saved a lot of scripting time (I'll give examples later).

Well, as they say: "I came for WATIR and stayed for Ruby".


cc Tom Goskar

Ruby is a cute scripting language that gives a much nicer feeling from a usability perspective than perl. For starters, it is OO. It is very flexible in regard to syntax. It is rich with functionality and extensibility options.

Don't worry, there are down sides too (remember that I am only starting to understand Ruby, so there is a high chance I am BSing here). For example, I still do not have a convenient working environment for scripting and debugging. The editor that comes packed with Ruby for Windows (SciTE) colors keywords and lets you run the script from the editor, and the interactive Ruby command line (irb) is a nice option that let's you script and run interactively, but this is not enough for me - I am used to the rich coding experience of Eclipse, so what I did now is to install the Ruby plug-in for Eclipse, and I'll update later about how convenient it is.

Well, I want to give a few examples for why I like Ruby and why it is cool.

I like Ruby's arrays. They come with many services built in as methods and let you use the array as stack or queue with no effort. Also, you can add arrays, subtract them, keep only unique elements, sort, etc. - ultra convenient.
myArray = [1,2,3,4,5]
myOtherArray = [2,4]
myDiffArray = myArray - myOtherArray # result will be [1,3,5]

I also like Ruby's built-in iterators, allowing you to split anything (more or less) into pieces and operate on each piece until you do them all:
myArray = [1,2,3,4,5]
myArray.each do |element| # element isa variable that gets each array element by turn
puts element # puts prints to the console
end

Another thing I like about Ruby is its rich core library and even richer extensions collection, covering (probably) everything I need. As in other user contributed libraries, the quality and completeness is not guaranteed and you are required to work hard to differentiate the good things from the bad.

I just started climbing the learning curve here, so I'll update later on things I find.

If you want to see Ruby in action in a few minutes, do the Ruby in 20 minutes tutorial.