Today is Wednesday "chore" day and one of the things I decided to do was evaluate an automated testing tool for Delphi called TestComplete (from www.automatedqa.com). It works for a lot of other languages other than Delphi, but seems to be the recognized leader in testing GUI applications.
It's expensive ($499 for the standard, $999 for Enterprise), so it needs to be perfect.
But so far, after just an hour, I'm a bit disappointed.
The basic way it works is that it records a script of you using the application. Along the way you can capture images and when it later runs the test itself, it can compare and see if the images match.
But this resulted in the first problems. First, the default image format is 16-color BMP. But when it captures the image at test run-time, the run-time image is full color. So obviously when it compares the full-color real-time image with the saved 16-color BMP, it finds lots of differences.
OK, so when saving the images at recording time, I have to remember to select a different file format, like PNG. I didn't find anywhere to change the default, so this is something I have to change each time I save a file. In fact, the number of steps involved in capturing an image at record time is rather annoying.
Even after this I ran into problems. Turns out, when you capture an image at record-time, the keyboard focus shifts to the TestComplete Record dialog. The application being run looses focus. This means that the captured image doesn't have the keyboard focus indicators. Lots of controls in CMUD change their appearence when they get keyboard focus.
So again, when running the automated tests, it flags all sorts of problems because the images captured at record time are not the same as the image at test run-time.
Also, the tools to capture an image seem limited to only capturing a "component", like a frame or window or button. You can't just select an arbitrary part of the screen. For example, if I just want to see if the caption of the window has changed properly to match the name of the CMUD session, I can't just select the caption, I end up having to select the entire frame. And then this compares a bunch of stuff that I'm not interested in.
Not very impressed after only a couple of hours playing with it. I've sent them some email on this, and their support is usually very good, so I'll see what they say about it.
But given how long it took to create some simple tests, I'm not optimistic about this idea of automated testing. Certainly not for the $499 price at this time.
Joined: 10 Oct 2000 Posts: 1551 Location: Australia
Posted: Wed Sep 20, 2006 10:15 pm
Yeah I downloaded the trial of that a year ago after reading some blog or other, maybe a Joel on Software one, not sure. Anyway I gave it all of about 10 minutes before I decided that it was way too hard to use and went back to my usual method of testing (give it a quick bash, send it off to the customer and wait for the phone calls *halo*).
My testing concept is pretty much as weak. Record a script of mouse and keyboard events in a basic macro program and then after changes let it run that script 100 times and see if it produces bad results at the end.
I have to say automated testing is a relatively new concept, and it will take longer for those attempting to implement it to realize what is actually needed then it will for the way we interface with the computer to change. I would posit that if the support is there for a person who states they are considering a purchase then perhaps they may actually be worth the price. It shows they have a concern at being a solid contender in this developing field and will likely value you and your wants for a long time to come. If they respond that way then they are working in the same lines as the Zuggsoft model, and that is the best there is.
_________________ The only good questions are the ones we have never answered before.
Search the Forums
Record a script of mouse and keyboard events in a basic macro program and then after changes let it run that script 100 times and see if it produces bad results at the end.
Although this may not be the ideal way of testing an application, I can certainly see this approach working quite well with CMUD. As CMUD is so scriptable itself, using an outside macro program to interface with the GUI can produce lasting effects in CMUD settings that can be compared against the known good result (which one can check be analysing the pkg database file or exporting to XML and then doing a diff). For example, you record a macro of you clicking to open the Editor, typing #var myvar {myval}, followed by the shortcut key combination to send that command. Then at the end when you are looking at the results you would see that in CMUD v1.06 myvar would not have been set, since the macro program was not able to get focus in the Editor. So now you know that there is a problem with the Editor (although you don't know from the xml result that it is a focus problem, it would have been easy to narrow down manually).
I am surprised someone sell software at $499 when its functionality pretty much only seem worth $49 :P. Capturing images of the desktop and compare them doesn't sound too advanced.
There is a lot of more things that probably could be done to find bugs in code. E.g static checking(stricter parser/semantic analysis).
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Thu Sep 21, 2006 7:38 pm
Rorso: Oh, don't get me wrong, TestComplete is *very* advanced. In fact, it's so advanced that I have no clue how to use most of it. It's geared more for high-end corporate developers using Visual Studio I think. The GUI testing was only a small part that I understood and tried to get working. Take a look at the www.automatedqa.com web site and select the TestComplete product and you'll see what I mean.
But I already have a lot of unit testing stuff, and parser testing stuff. It's really the GUI that remains to be tested. That's where a large percentage of the current bugs are. In other words, the underlying database might be working fine, but it's only when you add the visual controls and click on things in a certain order that starts to cause problems. The events and messages that get passed between the database and sometime like a TreeView or Grid can get very complicated and very difficult to test in any sort of automated fashion.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Sat Sep 23, 2006 4:25 am
Well, I'm not impressed with the email I got back. Mostly just said that my issues were added to their tracking system. He gave a potential kludge for getting around the keyboard focus issue that I might try. But honestly, the more I have played with it, the more I think it's complete overkill for MicroISV's developing desktop applications. I mentioned the high cost to him and he just replied with "its one of the lowest prices in the industry". Well, that might be true, but that doesn't make it affordable for MicroISVs.
Once again, MicroISVs get ignored by the tool developers. At $499, TestComplete would cost more than any other development tool I have purchased, including the Delphi 7 Pro environment itself! On an absolute scale, that's too expensive.
While the idea of automated GUI testing was nice, I just can't justify the cost at this time.
Too bad because I think AutomatedQA might be a MicroISV or might have been at one time too. The guy who responded to my email is the same guy who responds to my questions about the window docking library.
I haven't found any other GUI testers for Delphi like this, but if anyone runs across anything, let me know. The software developers in Russia really seem gung-ho with Delphi development these days and are producing a lot of nice free tools, so maybe someone overseas will fill this niche for small developers. Everyone in the US seems content to gouge the big Microsoft VS development companies.
As the technology is upgrading day by day. Growth as well as demand in software testing is increasing.
and also there are lot of free as well as paid tool which are available in market which makes testing more easy as compare to earlier.
You cannot post new entries in this Blog You cannot reply to entries in this Blog You cannot edit your posts in this Blog You cannot delete your posts in this Blog
