Who here is a beta tester?

| | Comments (81)

I'm just curious to know how many people reading my blog are beta testers? Is it a large number or a small number?

The reason I ask is because I feel let-down by our current beta testers, and I want to push them to do a better job. I sat down last night to use the latest beta for some "for-fun" programming and I ended up filing about 10 reports. The release before that, I filed about the same number of reports. So within a week, I've filed 20 reports, which is by far the most number of reports I've seen come in for any of our alpha releases.

My disappointment comes from the fact that many users complain about quality issues with the product, and yet they join the betas program and don't report as many bugs as they should. I get the impression that the vast majority (though certainly not all!) of our beta testers are not testers at all; they're only interested in a sneak-peek at what's coming up in the next release. We assume our beta testers are doing what they promised they'd do -- we figure that they're testing the new features of the product. And when we hear silence, we figure that we must have done an alright job; after all, no reports are coming in. So it's quite sad when I can stumble upon some very fundamental issues that tell me no one is testing the new features at all. I'd like the issues I'm talking about, but since the product isn't released, I don't want to discuss new potential features. But trust me, I found some that made me stop and go "why didn't anyone say boo about this?"

So, I'm issuing a challenge to all the beta testers who do read my blog: see if you can report more bugs than me. Come on guys, it shouldn't be that hard to do. Each new release for this version, let's have a little contest to see who can enter the most useful reports. For this "contest", please report only on new features (I don't want a bunch of reports on old stuff -- we can cover that ground later). Also, while I want to make sure every bug gets fixed (hence the reason for the contest in the first place), no one is allowed to say "Aaron said file this, and then you ignored me by not fixing it." You won't be ignored. :-)

The more good reports we get from beta testers, the more solid the product will be. You guys find new and creative ways to use features that we don't always envision. When you do this, you find bugs that we wouldn't otherwise find (since we have a natural bias for using the features "right") and we can make the product even better. Better beta testing = less bugs in a release.

81 Comments

I am not a beta tester (though my boss is on the beta tester mailing list).

Our beta testers are the same way. We have about 20 or 40, and only about 5 of them do much of anything. I think it's just hard to get beta testers to do anything.

Often I'll find a bug in RB (stable) and then not file a report because of the website or because I don't care to take the time to make a sample project. I'm under the impression that bug reports without example projects get pretty well ignored. Also, with our rather large framework, it's hard to separate out the source of the problem into a separate project to begin with.

Besides, lately my biggest complaint is speed of compilation and interaction in the IDE, and that's not really a bug I can file.

In my humble geneology app, I have hundreds of downloads, but I could just as easily run seti at home and hope for an answer. I get absolutely no reply about features, bugs or what ever. So I am doing translations now, hope english speaking people will beta test. getting good beta testers is really hard

It's not that we ignore reports without example projects -- it's that we close reports we can't reproduce. But it's ALWAYS better to report something than to not report it, even if you can't attach a sample project. Some reports "just click" with some of us sometimes.

And you can certainly sign on to any reports about the speed of the compiler and interaction with the IDE.

The RS site has been telling me "Your beta application is awaiting approval." for a long time now, so I'm afraid there's not much I can do to help.

I am currently testing the new beta. I do have 2 things to report, but have been covering my tracks first to make sure its legit rather than give shoddy reports. But I could do much better. I have actually submitted a few directly to William in regards to the excel class he wrote. That is one that has actually been a sore one since my project couldnt be put into production here because of this issue. Ive asked for possible inclusion into the next release but who knows. I think you guys are sometimes sitting with too full of a plate and not enough of you to go around that alot cant always be implemented(features(old and new) for new releases). Unfortunately some of this newer stuff that just came out, I really am not sure how to use it to test it out for you guys, else I would. Mars is really digging up some cool stuff. Im just not sure how to implement it. But you have my word I WILL try. Promise;)

@Tom -- William is on vacation right now though -- so you really should be reporting them via the feedback system (granted, I'm pretty sure William would be the one to work on them). This way he doesn't accidentally forget about them, as people on vacation tend to do. ;-)

And don't worry -- trying it a damned good start.

@Steve -- I'm trying to see if we're going to be letting a new batch of beta testers in. I'd be happiest if we kicked the beta testers out who aren't reporting anything and replace them with people who will help.

I surfed around the realsoftware.com website for about 5 minutes before finding any mention whatsoever of the beta program, and I couldn't visit the page because it required a login for the feedback system. Grah. Oh, and when you search the site using the search field, typing in "beta," one of the results looks like the right page, but it's actually a 404 error.

In short, maybe you should tell your web people to give the program a little more visibility. I'm only casually interested in it, and I'd like to know what it entails and such, but I'm not going to sign up for an account just to read about something I probably won't (or won't be allowed to) participate in anyway.

@James -- then it sounds like our entry barrier is working just fine. If you can't be bothered to make a feedback account, then you obviously can't be bothered to file bug reports (since you need a feedback account to be able to file a report). And if you can't be bothered to file reports, you should not be a beta tester.

I've placed 4 or 5 reports for the most recent betas, but I've given up on reporting non-crashing IDE glitches 'cause they seem to be overlooked entirely. The big problem for a lot of testers is that the new IDE is still a pain to use for real work, so testing is limited to simple little hacks to try out new features, or bulk compiles of existing 5.5 projects. I posted a bunch of IDE problems early on but they've received little or no attention. Some are serious workflow issues - like not knowing where or what an undo just undid (ncawrgaq) and the IDE only restoring the most basic project state (hxgsjusx). There's also a bunch of situations were copy or paste commands just don't work :/

I've also got at least 2 bugs that I simply can't reproduce outside a certain large project, and previous experience has taught me these *will* be ignored/closed so, nothing reported yet (though they've been posted on the NUG).

@Frank -- we do pay attention to the usability issue. The Undo and Copy/Paste issues bug the hell out of me as well, and it's on our list of things to take care of.

However, I don't see how failing to report these issues is going to ever improve the situation for you. If you don't report them, then there's a 100% promise that it won't ever be looked at. If you report it, then SOMEONE reads it and prioritizes it and you have a chance it's going to get fixed.

@Aaron: if what you want is participation, creating barriers probably isn't the way to go about it, no matter how reasonable they seem. :)

@Daniel -- I disagree. We have a closed beta program to be able to weed out the tire kickers from the people providing us with testing. And I know I'm picking on James (and I know he can handle it and won't take it personally) when I say this, but if you're too lazy to make a feedback account, you don't have any reason to be a beta tester since you won't file any reports.

I would be much more likely to work with the beta versions regularly if I were permitted to send applications built with beta versions to a few of my users to test. In my case, this is like two people (my application has a very limited user base), but since I can't do that I don't bring my project up the the new version until it is about to be released.

@Mike -- you can send your applications out to be tested, we don't disallow that. We just don't want you giving away beta copies of RB or talking about what new features will be in it.

Aaron, I'm a beta tester, but I haven't contributed much lately due to lingering health problems (prolonged recovery from my vestibular surgery -- I'm still pretty nauseous much of the time and more migraines are being triggered). This past couple of weeks has been the first time in a long time that I've been able to spend more than a couple of hours programming at one sitting. I asked Geoff whether or not I should leave the beta tester list until I recovered, but he said it was OK to remain. When I am able to program, I use the latest beta version, and try to use the new language features if possible, so I can see if any problems arise.

@Scott -- the purpose of my rant wasn't to make you feel guilty. ;-) It was more to incise those who can report bugs to actually file as many reports as they can. If you can only test for a few hours a week, that's fine by me -- so long as you report any issues you find. The fact that you're trying out the new language features is awesome!

I agree with Frank. The biggest issues for me are now IDE-usability-related. There are a few bugs with built apps and such, but not many.

Now, not reported you say? Think again.

Bugs:
dvbvfagv
rbajqxvp
jukorywk
gdnprdei
rekczhnj
jesimvio
cavujgnq
kvjupaxx
anjdtktr
unhzyzpd
gkxjdily
mtpyzgse
kucvdbos
ifvdgjnv
pukelwxh
qdzdqeze
umtexqwn
vzyrfkhv
cluoshzb
fgfztfrp (Unable to reproduce? Try again w/ OS X.)

Important feature requests:
jsdktbry
frkqtiak
oguvdufr
wnombhua
bgeuviup
dsvwfpuu
hjvyxmdf
qlpwlmmb
qnpnhxzd
spvjtzpe
khapcgqz (Better handling of missing external items, in general)


I used to report up a storm during the early stages of RB 2005's development. Most of my reports are just left open, so I have the impression that no one even reads them. So why bother? I'm still very much into beta testing, and I'll gladly contribute reports. But I'm discouraged when what seem like important IDE bugs aren't reviewed or verified, let alone fixed. And I can work around the IDE bugs, so I don't have a strong incentive to waste a QuickFix or waste your time campaigning for them.

I would be glad if the bug database was more closely policed. Instead of letting 40+ of the reports on my list stay open, at least review them. Close reports if you don't understand them, or if they violate a reporting rule in any way, or if you just feel like it! I'd rather have reports closed unnecessarily than ignored; at least I get the impression that something is being done about them.

@Adam -- you're right. There's no good excuse for not at least marking a report as reviewed. I know that when I'm browsing the bug base, I mark everything I read as reviewed, or verify it (and fix it!) if I can. But not everyone does this, and there's no regularity to it.

It's something we should be working on a lot more diligently. However, we *do* see every report that comes in, and I know that I read the majority of them. Heck, I get an email any time someone reports a new bug. So they aren't ignored. It's just that sometimes I'm on vacation. ;-)

Great, glad to hear that the reports are indeed being read. I do believe in the bug database, and it works quite well for most of the issues I have. There aren't any "showstopper" major bugs that are preventing me from releasing an application or anything like that.

The IDE is another matter. I think it's improving, but slowly. (I'll be happy if we're just allowed to use Cmd-L to jump to the LR Location Bar on Mac!)

OK OK. I've been so busy with work that I haven't had the time to play with the Alphas/now Beta. I'll file at least one bug report tonight that's been "broken" since RB 5.5. :D

OK, I filed a few. I'm curious now, too--what are the 20 you filed? I'd like to review them & sign on, or at least see which bugs you found.

Yay! I noticed a handful of reports just came in. Excellent!

As for which ones are mine -- I can't list them because they're new features in an unreleased version of RB. However, there's a bunch of them. ;-)

OK. I added 5 new ones even a crash maker or two! :D

Just found a compiler bug, reported under vkivvxdb. I have no idea where to start tearing apart - could Mars perhaps enlighten us as to what the assertion in FunctionCalls.cpp line 816 is for?

Mars probably could, but I probably could as well. You'll get that assert while we look for legal method variants (picking which overloaded method to use). If you have an extension method or an instance method (one which has the self parameter passed as the implicit first parameter) but no base expression, then you'll get this assertion.

I was in the beta at the beginning of the Rb2005 development but couldn't compile my (largish) project (because database support wasn't in IIRC). Then I got dumped for lack of reports. Hmmm.

When things slowed down here I applied for "reinstatement" since I was having some significant issues that I wanted to see resolved (this was with R2 or more likely R3) and I was willing to spend time testing to help clean them up.

I suspect others are in the same boat, though, that there are times when they can spend time doing testing and times when it's difficult at best. I certainly want to see Rb continue to improve and succeed.

Dave

@Aaron,
What do you mean by "base expression"?

And thanks for the quick reply - I didn't realize you were familiar with that part of the code as well. :-)

And while I'm here and thinking about it, I'm having trouble with a declare that returns a const char*. I've tried telling REALbasic that it returns a Ptr, but the resulting memoryblock has a size of -1. I know the data is longer than that because another declare reports its size to be >1. Any ideas? If it's any help, this is a declare from the Lua library.

The base expression would be the part before the ., if I'm to understand the logic of this properly.

A const char * is usually a CString. Any time you get a Ptr back from a declare, it's size is set to -1 to denote that we have no idea what its true size is. So if it's really a CString, then just call CString( 0 ) on the MemoryBlock and you'll have you String. But it may be more clear to declare it as returning a CString directly.

@Aaron: OK, I'll see what I can dig up on the failed assertion. Thanks for the info.

WIth regard to the declare, calling CString(0) worked perfectly - muchas gracias! As far as returning a CString directly, though, the compiler doesn't like that.

Hmm, that's interesting... I've never tried returning a CString directly, but I'm not certain why it shouldn't be allowed... I'll have to ask Mars about that one.

Glad the .CString worked for you though!

I'm going to continue the declare discussion on e-mail, so we can get this thread back on topic. :-)

Well, when a status to join a beta program says, for many months, that "Your beta application is awaiting approval", you can't blame anyone else than RS.

Beta testers might have time or not to test new features - maybe many of them don't even know how to test them and I'm sure if I were a beta tester, I wouldn't know or have the knowledge to test some of the new features. Maybe many of beta testers are not motivated to test or report anything due to the way bug reports have been treated (read bellow). Maybe some of the beta testers need to be recycled ;)

I also had (before I read you message above) the impression that most of bug reports were just being ignored - there's no change on the status for many months, no email, nothing!

I also think that some bugs don't get the attention and priority they deserve - maybe because no one else on RS needs them solved so fast as I (and many users) need them because we *use RB 2005 to develop our applications* - i.e. Copy/Paste bug fmlbhpur, Alt Gr key bug crbtxfgd and others. If these bugs have priority and RS can't solve them, JUST LET US KNOW!

The lack of communication between RS and users that submit reports is a BIG problem! When IMPORTANT bugs are submitted (like the above) and after 6 months there's no communication from RS and the bug status still remains "Open" then something is wrong.

If RS needs feedback, we also need some feedback about the progress made on solving the bugs - if this is not done, then users might think that nothing is being done to solve the bugs... so maybe some users think "why care to report new ones?"

When I see a bug I usually report it. But I must say that my motivation is not as it was before when I got some feedback and major bugs were solved in time. After 6 months, let's see if any of the above bugs will be fixed on next 2006 r1 release.

"We have a closed beta program..."
I think RS would benefit a lot if the beta program were not SO closed!

After many months waiting to be approved for the beta program, who knows if one of these days I'm approved if I will have the same motivation and time than I had when I submitted my application. Maybe I even have to refuse if it takes much more time :)

Ok, I'm sorry for being a little OT - I know this was about beta testers not reporting as expected, but I felt my feedback could help in any way.

I feel guilty for not filing many reports. To be honest, I've been so busy with non-RB work that I don't have time to play with the alpha/beta before the next one comes out! When I am working with RB, I make sure to file reports on even the smallest of problems I find.

Also, I don't particularly care how well the IDE is polished off (so long as it does work). I get mostly pissed when I find a show-stopper in the framework. My most recent one is olfayulb - but I have a suspicion that's an Apple Bug. :-(

@Carlos -- I can understand your frustration as a user. Now imagine mine. Let's take the copy/paste bug for a quick example. I've stated more times than I can count that we know it's a bug, we're working on it, but it's a very difficult problem to solve without breaking absolutely everyone's menu code. However, instead of properly using the feedback system and signing on to an existing report, people just simply file A NEW REPORT. I've already verified about the first 5-10 reports on this topic. Now I just stopped looking at them when I see that in the subject line. I would venture to guess there's another 25 reports all about the same topic. Kinda frustrating.

As for AltGr, I was very surprised to hear about that one seeing as how it wasn't until r2 or r3 that anyone even mentioned it to us. And I think that one's been fixed now.

I agree that communication channels need to be improved though, so don't get me wrong.

"So it’s quite sad when I can stumble upon some very fundamental issues that tell me no one is testing the new features at all."

I think that may be how your users (even your beta testers) feel. I get the feeling that REAL uses its beta pool as if it was a QA department. Remember, the beta testers are not paid for their work, receiving no form of compensation, not even a free or discounted product. Yes, REAL is lucky to have a dedicated community who often go out of their way to test the product, and fan-based volunteers may cut it in the world of open source, but it's not a great strategy with closed source, for-profit products. Simply put, beta testers are *not* Quality Assurance, period. You don't release a product with 20 easy-to-find and fundamental bugs to over 300 paying users, even if they are in a beta group. Beta users expect a mostly-stable, tested product that they can begin using in every-day work. That doesn't mean it has to be good enough that they can just start using it instead of the old version for everything, but it should be stable enough to hold up to at least casual use.

To turn this around, why weren't these bugs found internally? Was it reviewed by QA? What does the person/people in QA say about this? How did such fundamental flaws in new features sneak past? Seeing that they did implies that QA at REAL is simply broken. I don't know how it is broken - probably just a lack of personnel, but it is clearly broken and it's entirely unfair to blame the beta testers for that. This is really something you guys have to take up - you just will not get the same level of testing from a beta group you will get from a paid staff.

Sorry if this felt rant-ish. It's not meant to be an attack on you or the company, both of which are pretty cool. But I see this come up again and again with the beta releases. I really think this is something you guys need to address internally. With the new rapid-release schedule we will see even more problems with a dysfunctional QA dept.

@Brady -- I disagree with you. If we had a normal beta program, then I could totally agree. However, we don't; we give access to our ALPHAS (and call it a beta program, go figure). Alphas are never stable, no matter how much QA they get. Ever. Ever. So if you think a lot of bugs make it into the released alphas and betas, you should see the number that get caught internally and fixed before you ever see them.

Granted, some stupid ones make it out at the last minute, usually driven by the need to get beta testers something to work with before the previous release expires. Example: the variantToFoobar functions causing problems left and right in the latest release. However, we try to make that the exception, not the rule.

I think I'm well within my rights to expect beta testers to do what they said they would do when they signed up. Because we expose our alphas to users, we expect a certain level of participation from them in finding bugs and reporting them.

Basically, and not to sound contrite (because I do value your opinions): our game, our rules. I'm not saying our rules are smart or attainable... ;-) But we do get to pick the rules. We've always told people what to expect when we let them into the beta program; and lately, I've felt like there's more tire kickers than there are helpful people reporting bugs. When there's over 300 people in the program, and a new ALPHA RELEASE only generates maybe 10-20 reports -- I tend to think the beta program is broken. Hell, I'd be *happy* to get 100 reports (only a third of people reporting!), and I'm *sure* that no amount of internal QA will find every nuance that bugs people or strange use-case that we haven't thought of.

+1 Brady.

Users join the beta program to validate whether the bugs they filed are going to be fixed in forthcoming RB releases, and whether they're exisitng software will work on said forthcoming release. Anything regarding the privilege of testing RB for RS is gravy for RS.

If anyone files a bug report, it may or may not be fixed. If a customer wants it fixed, the resolve is to sponsor RB to fix it through the Paid Support program. Does the beta program change this in any way?

I'll leave my rants on organizational infrastructure for the rest of the posters, but suggest that if you are truly interested in a thriving test process, start a collaborative RBUnit project and open source test library.

Agile2006 is in MN this year. Going to be there?

There's a point in every developer's career where they whine about the Test department, sometime near the whines about the Marketing dept., then later they realize that all they get from these folks are gifts to be received graciously.

I can certainly understand that you want more input from your beta testers. But then you ship ALPHAS to them, versions you know aren't very stable. This makes a big difference: your users pay to use RealBasic, which means that most of them are using RB in a production environment, working on real projects. It seems difficult to me to use an Alpha version for production.

Which means -- most of your testers will only give a quick try to your Alphas. Anything that strikes immediately will probably get reported. If your alpha crashes, your testers will revert to the last stable version (probably the released one) and wait for the next alpha / beta.

I used to try out every build of Mac OS X Apple sent to developers -- until one day, a beta scraps my hard drive. I'm a developer and I can't afford my production machine going down, so I stopped trying out the OS betas.

My own beta testers act the same way. Actually, very few of them are active and you need a large pool of them to get useful reports. The only user really banging the apps is himself a QA guy at work, and I can see the huge difference it makes in reports quality / volume.

So the take out is -- hire more QA people, external beta testing is just not reliable enough to ensure your product is of acceptable quality.

@DeanG -- the beta program and paid support are two separate processes. And FWIW, I agree that the beta testers should check to make sure their existing projects continue to work; that's one of the most important things to test. And I do think our beta testers do a great job of that. However, they should also be trying out new features, which is something I think our beta testers aren't doing a good job on, and that's the stuff we need the most help testing.

The thing that people seem to be misunderstanding the most is that no amount of internal QA is going to catch the strange bugs. For instance, I had a user report (a long time ago) that dim n as new NetworkInterface didn't work. The thought that you could create a NIC in software never even crossed my mind; so it's not much of a surprise that it never got tested. You guys come up with some very creative ways to use features that we never even dream of, so the really good testing that stuff really can only happen by you guys. We write unit tests for all the new features we can (some things a very hard to unit test too -- such as "does the debugger show this variable properly"), but since we're the ones designing the features (QA included), we have a bias when writing unit tests. We *try* to get all the crazy cases too.. but we're human and fail at it.

As for Agile2006, no, I won't be there. And I'll admit, this is my whining about a lack of good testing all around (both internal and external). While I agree that internally, we can make changes to make things better -- I don't think I'm out of line in asking for external testing help as well.

Aaron, I didn't take this post as an attempt to make me feel guilty. Hey, I'm Jewish. You could never guilt me as much as my parents -- it's a genetic gift!

I'm just tired of being sick. I started work on Reality Check right after Real World and made tremendous progress on it until the vestibular problems hit (I can force myself to work with the migraines -- I've had them daily for 10 years). Unfortunately, since the surgery, there's lingering nausea, and it triggers the migraines more. The additional pain, nausea, fatigue and cloudy thinking keeps me from getting a much programming done. It's frustrating, because I was supposed to have fully recovered a while ago, and when you brought up the beta testing, it reminded me of how much I could have accomplished.

@Florent -- Since I'm not in charge of hiring, I can't really hire more QA myself. ;-) However, I can certainly prod people via my blog. :-P Plus, I think the discourse with users that we're having is a good thing. I get a lot of new points of view that are external to the company which I can then take to people higher up than me as ideas.

Hopefully no one takes any of this as being an attack or just blind "screw you guys" comments. It's neither; I am listening, taking notes and looking at it with an open mind.

@Scott -- I really hope you start feeling better. Not because I want you testing the product, but because having medical issues sucks. Hopefully the delayed recovery isn't a sign of other medical issues!

Feel better, that's an order from Crazy Dr Aaron. :-P

Goddamned this thread is long.

Aaron, the point isn't that I'm too lazy to file a feedback report *if* I were to participate in the program, the point is I can't even GET ENOUGH INFORMATION ABOUT THE PROGRAM to decide whether I want to participate or not.

What are the advantages? Is there any kind of 'gift' to testers? (I know some companies provide a free license for testing purposes, and some companies provide a free product license if you submit a good number of fixed bugs.)

I'm not going to sign up for something, put my name on YET ANOTHER spam list, just to find out the answers to these questions. I've signed up on every goddamned website on the internet, I'm sick of signing up for things.

Sure, put the application for the beta program behind the sign-in. But at least give me an OUTLINE of what it's about outside so I can make my decision without having to give my personal information to yet another .com.

I gotta rant about this a bit more after reading more of the thread.

I can't even answer the most fundamental question about the beta program using your website without signing up for an account: "Does it require owning a RB license?" Knowing the answer to that question would instantly qualify or disqualify me, and I'd know whether it was worth signing up for a feedback account.

So yeah, you're trying to "screen" me or whatever, but why not just give me a little info and I can save you the trouble? Right now your website has nothing. And searching for it returns a 404.

> the beta program and paid support are two separate processes.

A paid bug fix gets rolled out first via the beta program. It's a moot point in this "all bugs are shallow" discussion searching for "enough eyeballs", but still an important cog in the process, especially in regards to customer feedback.

> However, they [beta testers] should also be trying out new features,

Why do you expect this? Your 'should' count is pretty high..

Would you post a link to the rules/agreement of the beta program?

@DeanG -- Because that's their "job" as a beta tester. They are supposed to test (hence the word "tester") anything and everything they can get their hands on.

And I'd love to post a link to the rules, but James alerted me to the fact that *it's gone!* It's supposed to be at

but when I go there, it's a page not found. So at this point, I give up. If we don't even have an active page telling beta users what's expected of them (that's easy for them to locate, which searching for is NOT), then I really shouldn't be bitching.

:: shakes head ::

I win teh internets.

I'm going to make just one more comment on this thread (famous last words). I think what we see here is the beta program itself is broken as a beta program. Part of that is the size - a smaller group of 20 dedicate beta testers would probably find and report more bugs. And a contest like this, while it may generate a temporary spike, can't fix the problems. My personal feeling is an increased commitment to QA and a tighter (real) beta program would help to find more bugs. On the flip side, the current beta program does an outstanding job at creating community and promotes user feedback on the actual features, which would be a terrible thing to lose. I really do enjoy the product, the employees and the community surrounding it, and would hate to see any of those things diminished. So, any "fix" to the testing program (both the beta group and QA) should take those current strengths into account.

So, to sum up, I know there are problems that should be fixed, but I don't want to lose what we already have, and no, I have no simple answer to how it should be fixed, but I'm happy there are some people at the company paying attention, and now on my seventh comma (make that eight), I really should end this sentence.

"As for AltGr, I was very surprised to hear about that one seeing as how it wasn’t until r2 or r3 that anyone even mentioned it to us."

Not correct! It was reported on June 26th (report crbtxfgd) for r1. I also sent a message to the NUG list for this bug and report id on July 11th with the tile "Windows Alt Gr Key Not Working on RB 2005 IDE".

"And I think that one’s been fixed now."
Great! Thanks ;)

LoL you bring up good points and lots of commas Brady. I can see why the community aspect is important, and I agree -- I think the community is what makes REALbasic so much fun. You get stuck? Check the mailing lists, check the forums, hell -- ask a RS engineer in a personal email! Our strong community is a great asset.

Begin Rant

But you also bring up a nit I need to pick -- commenting on new features. This release seems to be the "worst" that I recall with people nay-saying a feature without ever trying it out. A number of the threads could have been eradicated by 1) reading the readmes, and 2) trying the feature out. Instead, people started a gigantic thread on how the feature "should" behave and speculated to no end. While user feedback is certainly important to us, we would prefer you try the features out, try to use them in situations you'd think are realistic, and THEN tell us what you find you cannot do. Griping about theoretical problems without ever trying the feature out is frustrating; we spend a lot of time designing how these features work, how it fits into our roadmap, etc. So to be told that we've done something "wrong" by users who have yet to actually *use* the feature is demeaning and frustrating.

End rant. :-P

@Carlos -- I stand corrected! And I don't usually deal with this sort of issue (William tends to). So if I wanted to test to make sure this is working, what language, keyboard layout and keys should I be pressing to test this out?

Aaron:
Using a Portuguese (Portugal) keyboard:
[ - Alt Gr + 8 (key with number 8)
] - Alt Gr + 9
{ - Alt Gr + 7
} - Alt Gr + 0
€ (euro symbol) - Alt Gr + E
£ (pound) - Alt Gr + 3
@ - Alt Gr + 2

Unfortunately, it looks like it's still not working. I can enter those characters in an EditField, but when I try in the code editor itself, I get nothing. Rawr! I'll be sure to pass this information along.

How about in windows installing a United States INTERNATIONAL keyboard? For me this solved some simular problems with accented characters. Does that help?

@Andre: I already tried that - to add on the Regional Options a new Input Local with a US keyboard layout. Then I just needed to switch between the keyboards from the taskbar.

The problem is to memorize the place of the new keys on my keyboard when programming with RB - many important keys like = ( ) " - + changes their places - a mess when my fingers are faster than my brain ;)

I found I was wasting a lot of time trying find the correct keys. I had also to remember to change again the keyboard layout when I was using other tools that needs my default keyboard.

It's a workaround, but not a good one. Nice to know it worked for you.

One more thing... One of my "bugs" was probably more of a "feature request" (however, from a consistancy POV, it was a bug). However, I no longer see a "feature request" option for bug reports. Tonight the status of the bug was changed to "not a bug"... well, inconsistancy is a bug. How do I change the report from a bug to a feature request?

Good question -- we no longer differentiate between bugs and feature requests (that I'm aware of), so I think the best thing to do is just to say "this is a feature request" or something.

What's the bug ID (you can email me if you want) and I can re-open the report and mark it as a request.

@Aaron - Sent via e-mail.

*screams* There's a bug in that corner over there! And it's ID is siuoward! :-)

I am also a BETA Tester.
Last Bug Report: fdiculgf a few Minutes ago.

Aaron: You're a RB programmer and you know where to test RB for possible bugs: that's why you can find 20 bugs in one shot :-)

The rest of us mostly put our attention on classes that our project uses, so it's difficult to find a QuickTime bug when you never use that class in your projects.

Everytime a new alpha/beta is coming out, I download it and test with my apps. Most of the time the bugs I found are already reported. When I hit something that is not reported, I rapidly report it.

However, as you're disappointed with betatesters not posting bugs, many RB developers feel the same when they see their bugs never fixed (checkout the NUG).

I can comment more on that from my own experience, privatly, if you want.

I'd like to be able to provide feedback on the beta versions, but unfortunately i've requested to be a beta tester about 3 weeks ago, and my request to be a beta tester is still pending approval. Our company will start converting a 300000 line vba program to rb in februari. That means there will be 3 programmers at our company that will be using rb fulltime from then on. I've been using rb since june and have a tendency to hammer at thing till they work. I would love to be able to help improve the software (realbasic), since i was also responsable for the decision to use rb in our company. This means that the better rb becomes, the more my boss will be happy with my decision to use rb. As a professional programmer i reallly expect my programming environment to work as documented, since anything else will waste my time when i'm actually programming for our company. Therefore i benefit directly if rb works as documented. It's a win-win situation. I get a product i'm happy with, and rb works the way it should, so realsoftware has a product people want to use professionaly. Hopefully i will be able to betatest soon. And no my programming doesn't stop when i go home, since i don't always have time to test stuff during working hours, i tend to do that at home. Trying out new stuff is what makes working with computers fun for me. It sure is a lot more fun than programming the same old stuff over and over again, plus on top of it a new platform always gives new possibilities, so just converting the vba to run in rb would be pointless, since we wouldn't be using the possibilities of the new platform. There, i hope i wasn't rambling too much ;)

@Celeenwerck Dirk -- Once I return from my vacation, I'm going to find out what's happening with new beta applications. Right now, all the applicants are just sitting in a pool, and we've yet to remove "unhelpful" people from the betas program. So I'd like to see us make changes so that people like you (and others who've commented on here) have the chance to contribute while people who are stagnant on the betas program are removed.

@Aaron: I agree you should remove unproductive beta testers. But make sure you guys take into account past performance, not just the last month or so. And also, it would be nice if beta testers were given a warning or two before they're kicked out.

At this point, I think it's safe to say if you haven't reported a bug since the program's inception, you're safe to punt. ;-) But yes, a warning may not be a bad idea.

@Aaron-- Great. And while I was filing all these new beta reports, I noticed some shortcomings in the beta system. So I filed a report, and dropped a link to the NUG. Lo and behold--a couple of hours later it has 11 votes already!

Check it out yourself at buihofay. I know you're not behind the feedback system (considering your well-advertised hate of web programming!) but perhaps you could nudge the person who is in the right direction. After all, if the feedback system was a nicer place, I could find myself spending even more time there! ;-)

Well before I became a beta tester I reported bugs and on several occasions chatted with those at realsoftware about a issue I was having....so sometimes I think it really should'nt matter whether your beta testing or not...just report what you find and in the long haul it should make RB that much more better. One other thing that strikes me as odd is not having a mulitple file selection in the open dialog. Having come from VB it was always a use on windows to select a bunch of files for processing, so I can see where RB has its roots in the mac....well I can use the wfs for this or even prior to that a snippet jon j gave me to do the same....but when I want to try it on linux...Im stuck. Not knowing the scope of the code nor the depth, how hard is that to implement into the open dialog?

@Tom -- I agree, everyone should report bugs (beta tester or not)! As for the multiple file selection, that was a shortcoming of the old compiler (no way to return arrays from any methods), but there's no reason we can't implement it in the framework. Feature request it (if it's not already in the system) -- it's very reasonable to ask for.

ID-bgtsgbcy

Says its been reviewed, so hopefully.....

Holy crap. This is a lot of responses.

I reported a couple hundred reports in the first few releases, but many months later, I am sad to see that some of my show stoppers have not been fixed yet. I don't want to report even more bugs, since that will mean that my original bugs will take even longer to get fixed.

@Jeff--you said it best. That's how I feel.

@Aaron--You said you get an email every time a new report is filed. Do you also get an email every time someone signs on to an existing report, or requests a status change, or modifies an existing report?

Those would be beneficial since they would let you better keep up with existing reports in addition to the new, breaking-issues ones.

@Adam -- no, I only get emails when the report is first submitted. However, in the web interface that I use, all reports which have been modified percolate to the top of the list. So any time a change is made, it gets noticed (unless there's more than 100 reports changed since the last time I checked).

If I got an email every time a report changed... the email would lose its use due to swamping me.

Aaron,

I am in the beta program, but lately have not been filing much. My main problem is I use 2005 at work, but at home where I do most of my programming I use 5.5.5. Mainly because of the speed difference and compiling time difference. I do like the 2005 interface, but some of the bugs just kill me (well mainly the copy/paste issue, which I am glad to see your comments on). I reported that back in June (sqlrikjx), but still run into it. For me it kills my productivity, but that is another thread. On windows the 2005 IDE is a huge improvement and it seems to run quickly. On my mac though is a different, it seems to run much slower (this might be my hardware also). For me, there are three reasons I have not moved. First because I will need faster hardware (mini and old powerbook) and second because when I did test using the first few releases, my mac application's size went way up (I need to retest and to compare the compressed size). The third and main reason is time. Right now I am finishing up my MBA and don't have time to retest all of my programs. I have one class to go and then might make the switch (hopefully rb will be able to compile for the new macintels by then). I just wonder how many others in the beta group fall in the same category of using 5.5.5 for the bulk of work and just some 2005.

Right now I am at work, working on a new project. I started out testing on the betas, but switched back to r4 due to a weird IDE bug where the window gets all messed up and it crashing. I had not filed a report on what I was seeing, but just did. One other comment is it is hard to see in the web feedback when something or what version something was reported on. For instance, I am guessing that you saw the same bug as me (using windows), but it is just in the latest beta version, so how do you search for bugs for the latest beta? What I was seeing is the scrollbars just go nuts and disappear or show wrong. How do you search for that?

I really love using RB and want to help any way possible, maybe some base guidelines need to be sent out to the beta testers with each release. Such as these are the known issue already reported for this release. It might even be good to weekly give a tally of reported-fixed-closed.

Thanks for the great work.

My disappointment comes from the fact that many users complain about quality issues with the product, and yet they join the betas program and don’t report as many bugs as they should.

Well geez, from the looks of it lately, I though the purpose of the beta list was to audition for new general manager of RS. I'm sorry if I offend anyone by reacting to it, and I hope you guys weigh the peanuts in the peanut gallery appropriately.

My concerns and focus in beta testing is compiling and the runtime. If external items are working right, and my projects compile, then you haven't broken anything too badly, and that's great by me. I also try the interesting new features, and usually find that they work as advertised. The IDE doesn't get a lot of attention from me, as it's quite serviceable for me, and improvements are welcome but not my top concern.

Here's another BIG problem with the current system:

Only those on the beta program can help make sure that the new version did not break things from older versions!

Now think what happens if such a breaking bug would not be spotted by the SMALL amount of beta testers - RS will then release a final version to everyone, and if that contains a bug seriously breaking old app, it'll be another wait for a new release in 3-4 months for all of them!

Before RS introduced the "rapid release" thing, we could have dealt better with this: A version 5.5.0 came out and when the full amount of users got their hands on it, all the breaking bugs would be discovered and (rahter) soon a 5.5.1 would get out. Not so any more!

The new rapid release model needs a new way of beta testing therefore.

Here's my suggestion: Make it a TWO-STAGE testing process!

First, add new features in the alphas, and let a closed group have access to them, to play with and discuss them.

Then, in the BETA phase, i.e. the phase when you only fix bugs, let EVERYONE get a hand at it so that you can make sure that everyone can at least test their old apps and make sure you've not broken anything.

Doesn't that make sense?

The idea makes sense, but the logic is flawed. You didn't get a rather soon point release before (it was frequently 90 days or more, unless the bug was MAJOR), and you had no promises of ever getting any point releases. And with the rapid release model, we don't sit on things until the 90 days are up -- if there's a major bug, there'll be a release quite quickly with the fix in it.

We had an open beta program, and it didn't work too well. It confused a lot of people, and it wasn't effective. So we closed it. It's become slightly more effective, and the confusion level has gone WAY down (I don't recall the last time I've seen people on the NUG complaining about bugs in the beta version -- saying they can't release their app because RBXa4 is 'unstable').

All I can say is that when I start the new beta to see that it has not broken my existing code for customers projects, if I hit a bug in the IDE I report it.

If the project won't compile these projects the beta DOES get put back on the shelf until the next release. I suspect that there are a number in this category that try their work in the releases and when things dont work they revert to the last version that did and continue working / earning a living with that.

Unfortunately with 2005 and 2006 this has happened in just about every release and only 1 of about 12 projects has been compiled in anything newer than 5.5.5

I also try to go through and find the bugs that I've reported already and see if they are fixed as the release notes indicate. Some are easy to track down and verify, and some are not (the whole network issue where the remote app was connecting back to th wrong IP address you and I found I still can't reproduce)

What I suspect you're seeing is that folks are using the versions that "work" and continue to work for them and are only trying the latest beta releases to see if the next version may be ready for them to move up to yet.

A follow up

Assuming that "And when we hear silence, we figure that we must have done an alright job; after all, no reports are coming in." may be exactly wrong

There are several interpretations :
1) the one you suggest
2) that no one is testing (for whatever reason)
3) that it's not working very well but people can't reproduce the problem so cant report a bug
4) any one of several hundred other interpretations of "no one reporting any thing so ....."

2006 has been crashing on me for any one of several reasons but most of which I seem to not be able to reproduce reliably. Rather than submit a random "it crashed" report that can't be reproduced, as this is a useless report, I've been trying to work with copies of projects and reproduce them (with little success)

Leave a comment

Disclaimer

I'm currently an employee of REAL Software. My blog is mine. The opinions represented in this blog are mine as well and may not represent my employer's opinions. All original material is copyrighted and property of the author.

REALbasic® is a registered trademark of REAL Software, Inc. REAL SQL Server™ and Lingua™ are pending trademarks of REAL Software, Inc. All rights reserved.