Zugg Software :: View topic

Posted: Wed Sep 20, 2006 9:55 pm

Today is Wednesday "chore" day and one of the things I decided to do was evaluate an automated testing tool for Delphi called TestComplete (from www.automatedqa.com). It works for a lot of other languages other than Delphi, but seems to be the recognized leader in testing GUI applications.

It's expensive ($499 for the standard, $999 for Enterprise), so it needs to be perfect.

But so far, after just an hour, I'm a bit disappointed.

The basic way it works is that it records a script of you using the application. Along the way you can capture images and when it later runs the test itself, it can compare and see if the images match.

But this resulted in the first problems. First, the default image format is 16-color BMP. But when it captures the image at test run-time, the run-time image is full color. So obviously when it compares the full-color real-time image with the saved 16-color BMP, it finds lots of differences.

OK, so when saving the images at recording time, I have to remember to select a different file format, like PNG. I didn't find anywhere to change the default, so this is something I have to change each time I save a file. In fact, the number of steps involved in capturing an image at record time is rather annoying.

Even after this I ran into problems. Turns out, when you capture an image at record-time, the keyboard focus shifts to the TestComplete Record dialog. The application being run looses focus. This means that the captured image doesn't have the keyboard focus indicators. Lots of controls in CMUD change their appearence when they get keyboard focus.

So again, when running the automated tests, it flags all sorts of problems because the images captured at record time are not the same as the image at test run-time.

Also, the tools to capture an image seem limited to only capturing a "component", like a frame or window or button. You can't just select an arbitrary part of the screen. For example, if I just want to see if the caption of the window has changed properly to match the name of the CMUD session, I can't just select the caption, I end up having to select the entire frame. And then this compares a bunch of stuff that I'm not interested in.

Not very impressed after only a couple of hours playing with it. I've sent them some email on this, and their support is usually very good, so I'll see what they say about it.

But given how long it took to create some simple tests, I'm not optimistic about this idea of automated testing. Certainly not for the $499 price at this time.

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

Yeah I downloaded the trial of that a year ago after reading some blog or other, maybe a Joel on Software one, not sure. Anyway I gave it all of about 10 minutes before I decided that it was way too hard to use and went back to my usual method of testing (give it a quick bash, send it off to the customer and wait for the phone calls *halo*).

SubAdmin Joined: 18 Nov 2001 Posts: 5182

My testing concept is pretty much as weak. Record a script of mouse and keyboard events in a basic macro program and then after changes let it run that script 100 times and see if it produces bad results at the end.

I have to say automated testing is a relatively new concept, and it will take longer for those attempting to implement it to realize what is actually needed then it will for the way we interface with the computer to change. I would posit that if the support is there for a person who states they are considering a purchase then perhaps they may actually be worth the price. It shows they have a concern at being a solid contender in this developing field and will likely value you and your wants for a long time to come. If they respond that way then they are working in the same lines as the Zuggsoft model, and that is the best there is.

Wizard Joined: 14 Aug 2004 Posts: 1269

Posted: Thu Sep 21, 2006 4:23 pm

Wizard Joined: 14 Oct 2000 Posts: 1368

I am surprised someone sell software at $499 when its functionality pretty much only seem worth $49 :P. Capturing images of the desktop and compare them doesn't sound too advanced.

There is a lot of more things that probably could be done to find bugs in code. E.g static checking(stricter parser/semantic analysis).

Posted: Thu Sep 21, 2006 7:38 pm

Rorso: Oh, don't get me wrong, TestComplete is *very* advanced. In fact, it's so advanced that I have no clue how to use most of it. It's geared more for high-end corporate developers using Visual Studio I think. The GUI testing was only a small part that I understood and tried to get working. Take a look at the www.automatedqa.com web site and select the TestComplete product and you'll see what I mean.

But I already have a lot of unit testing stuff, and parser testing stuff. It's really the GUI that remains to be tested. That's where a large percentage of the current bugs are. In other words, the underlying database might be working fine, but it's only when you add the visual controls and click on things in a certain order that starts to cause problems. The events and messages that get passed between the database and sometime like a TreeView or Grid can get very complicated and very difficult to test in any sort of automated fashion.

Posted: Sat Sep 23, 2006 4:25 am

Well, I'm not impressed with the email I got back. Mostly just said that my issues were added to their tracking system. He gave a potential kludge for getting around the keyboard focus issue that I might try. But honestly, the more I have played with it, the more I think it's complete overkill for MicroISV's developing desktop applications. I mentioned the high cost to him and he just replied with "its one of the lowest prices in the industry". Well, that might be true, but that doesn't make it affordable for MicroISVs.

Once again, MicroISVs get ignored by the tool developers. At $499, TestComplete would cost more than any other development tool I have purchased, including the Delphi 7 Pro environment itself! On an absolute scale, that's too expensive.

While the idea of automated GUI testing was nice, I just can't justify the cost at this time.

Too bad because I think AutomatedQA might be a MicroISV or might have been at one time too. The guy who responded to my email is the same guy who responds to my questions about the window docking library.

I haven't found any other GUI testers for Delphi like this, but if anyone runs across anything, let me know. The software developers in Russia really seem gung-ho with Delphi development these days and are producing a lot of nice free tools, so maybe someone overseas will fill this niche for small developers. Everyone in the US seems content to gouge the big Microsoft VS development companies.

Newbie Joined: 03 Apr 2019 Posts: 1

As the technology is upgrading day by day. Growth as well as demand in software testing is increasing.
and also there are lot of free as well as paid tool which are available in market which makes testing more easy as compare to earlier.