Bug Reports: the good, the bad, the ugly

Lover of both oatmeal and coffee, Philip asks via the Suggestion Box for "Examples of bug reports spanning the spectrum of the best case to worse case, so that we have an idea of what you're looking for versus what users think you need."

Ooh, this could be a touchy subject... I don't like calling people out on bug reports because I appreciate that it takes effort to file them in the first place. So I don't want to be mean and mock people's good intentions. Instead, I'm going to try to point out good feedback report examples, and then discuss some general "don'ts" when it comes to filing feedback reports. I may pull a few examples out, if I think I can make them general enough so as not to offend the reporters. ;-)

This first example is what I consider to be a great bug report, and my heartfelt thanks goes out to Charles for reporting it (his reports generally rock, btw).

Product: REALbasic
Platform: Mac OS
Version: REALbasic 2008 r3
Description:
Dictionary lookup of 32-bit integer keys using a 64-bit integer returns different result in PPC v. Intel.

Steps to Reproduce:
dim X as Int32 = 43
dim Y as Int64 = 43

dim d as new Dictionary
d.Value(X) = true
dim b as Boolean = d.Lookup(Y, false)

b is true in Intel builds, but false in PPC builds.

< link to project goes here >

This is an awesome bug report for several reasons. 1) It's concise. There's not a lot of "fluff" in the report -- it just states the observations the reporter was seeing, a snippet of code that shows the issue, and the expected behavior (which is implied). 2) It has example code which demonstrates the issue, with as little extra noise as possible. 3) There was an attached example, with exactly the same code as what's in the report. I don't even have to copy and paste! 4) The report states what version of RB the reporter was using, as well as what OS, and came with a valid email address (which you can't see here) in case I need to communicate further. I love seeing reports like this because it gives me every reason imaginable to see if I can find and fix the bug, and no reason to say "I'll get to it later."

Let's peek at another excellent example of a bug report, only this time, it's going to be one of the automatic reports we get via the IDE's feedback mechanism.

[Auto] Compiler failed assertion in SemBase.cpp:440(2008r3)
File:SemBase.cpp
Line number: 440

Condition:code

Message:

IDE Version: 8.3.7130.4

Installed Plugins:
None

User Comments:
A poorly-written constructor of an XML subclass causes this assertion. Easy to replicate:

1 new project
2 new class
3 new constructor:

Sub Constructor

Super.XmlDocument

Exception xe as XmlException
break

Exception
break
End Sub

4 Try to compile: boom!
User Email:XXXXX@XXXX.com
XXXX@XXXX.com

Much of the information is auto-generated, but it's still very little to go off of. In fact, in this particular example, there's practically no indication of what the issue might be. I know what file it's in, and what line it's on, but that's about it. However, this kind soul did exactly the "right" thing: 1) Provided an example of how they got the assertion to happen, 2) Left a valid email address for me to contact them if I need more information (though I X'ed it out so they don't get any spam because of me). I can't stress this enough -- automated reports are great, but providing your email is better still. Quite often I need more information to be able to reproduce a bug, and if I don't have an email address to get that from, the best I can do is put up a "response message" that basically says "please email me an example that demonstrates the issue." Actually putting the example directly into the feedback report is even better still -- no email required! Thanks to this user, I was able to fix an assertion that was automatically reported by 16 people.

Now that you've seen some examples of good feedback reports, I want to discuss some things you should avoid doing, and perhaps see if I can scrounge up an ambiguous example for each. I do not intend to offend anyone, so if you see your report here, please don't be mad! I merely want to point out how to improve.

The first one has more to do with feature requests than bug reports, but since we get a lot more requests than bug reports, I want to point this out. We're smart people here at REAL. We design a complicated project, and we try to provide a very consistent API that appeals to as many people as possible. Please, don't tell us what to do -- tell us what problem you need solved. Instead of saying "we need XXX", it's usually better to say "I'm trying to do YYY, but can't. I'm imagining something like XXX." We're pretty literal when it comes to requests. If you say "I want XXX" and we don't want to implement it, we'll close the report. We do this because we're not mind readers, not because we're petty. Here would be an example of what I'm talking about.

I propose a new keyword such as "carray" for handling C style arrays in standard RB fashion:

carray myfloatArray() As single
or
carray myfloatarra(10) as single

Internally - each carray would hold two values:

1: a pointer to the C array
2: a ubound property holding the number of elements in the array. This can be CHANGED by code.

Nowhere in the report does it actually state *why* the reporter needs this functionality, what problems they're trying to solve, or what this will allow. It just states "do this, and this is how."

My biggest bug report pet peeve are reports that lack any actual steps to reproduce the issue. Quite often, we'll get a report that says "blah blah blah doesn't work." And that's about it. While that's sometimes enough to go off of, most of the time it really isn't. We need a code snippet, or better still would be an attached project file. A common variation on this theme are bug reports that include only a crash log -- that's better than nothing, but still not much to go on. Thankfully, if you have a valid email address, we can at least email you to ask for more information. Here is an example of what I'm talking about:

Product: REALbasic
Platform: Mac OS X 10.5.4 Intel
Version: RB2008r3.1
Description:
For more information please visit:
< link removed -- it was just a reproduction of the OS crash log >

Thread 0 Crashed:
0 ...ple.CoreServices.CarbonCore 0x914d3164 Plugin_GetCountAvailableTextEncodingConversions + 67
1 ...ple.CoreServices.CarbonCore 0x914d3001 FindConversionPlugin + 90
2 ...ple.CoreServices.CarbonCore 0x914d2858 TECCreateConverter + 247
3 rbframework.dylib 0x005e9d51 ConvertEncoding(string, unsigned long) + 529
4 rbframework.dylib 0x0067fde6 fileMacCreatorSetter + 126
5 XXX 0x0007bcf3 FolderItem.MacCreator.Set%%oi4s + 56
6 XXX 0x002d56f6 appPreferences.Save + 2357
7 XXX 0x002cba33 App.Event_Close%%o + 104
8 rbframework.dylib 0x0061edd0 RuntimeRun + 96
9 XXX 0x0000e96f REALbasic._RuntimeRun + 34
10 XXX 0x000057ad _Main + 73
11 XXX 0x00001514 % main + 36
12 XXX 0x0044e376 _start + 116
13 XXX 0x0044e2cc start + 43

All I can tell from this is that ConvertEncoding is crashing in the bowels of OS code for some reason, but no idea how to actually reproduce the crash. It's better than nothing (because sometimes there's an obvious issue in the function listed), but it's still not enough most of the time.

The final "please don't do this" item on my personal list is when you tell me how much I suck and I'm ruining your life, etc. We all hate bugs, and we all understand how incredibly annoying they are. There's nothing worse than being in the zone, and then having to come out of it for some stupid bug. But that's no excuse to take it out on the person reading your bug report -- they want the issue fixed just as badly as you do. We're all on the same side! I'm not going to post some of the abuse I've seen over the years, but suffice it to say that there are some reports which are better left unsent.

So my tips for making good feedback reports are:

• Be concise -- no one wants to read a book


• Have examples -- if a picture is worth a thousand words, a simple example project is worth a photo album


• Use a valid email address -- That way, we can contact you if we need more information


• Don't be mean -- no one likes bugs, and no one likes being yelled at


• Tell us your problems, not the solutions -- we're smart enough to handle it, I promise (and this rule applies mostly to feature requests)

I hope this helps (and I hope our totally awesome tester, Kyle, doesn't disagree with anything I've said), and I hope this doesn't scare you. ;-) The majority of the reports we get are "good enough", there's just a few gems and a few...not...gems that are worth pointing out. Thanks for your feedback though, as it certainly helps make REALbasic a better product!