Zugg Software :: View topic - Delphi Tip #1

Posted: Tue Mar 14, 2006 6:48 pm

I thought I'd post a positive blog entry for a change. I wanted to tell you about something really good that I found for Delphi a few weeks ago.

It's called FASTMM (http://sourceforge.net/projects/fastmm)

FASTMM is an open-source memory manager replacement for Delphi. It both replaces the BorlndMM.dll file that is used by the Delphi IDE, making it faster, but also compiles into your code to replace the default memory manager.

It has several important features:

1) It handles memory allocations and de-allocations more efficiently, making code faster. It also deals with small block allocations more efficiently to prevent memory from becomming as fragmented. This is a big deal for zMUD and CMUD.

2) It allows memory to be shared between the main application and DLL programs, much like the FastShareMem program that I found last year when I was working on zApp.

3) The best feature is the built-in memory leak detection. With full debug mode enabled, it performs real-time memory leak detection and writes detailed debug information to a log file when the program exits notifying you of exactly where the leak was initially allocated, along with the Delphi line numbers in source code and the Delphi type of the variable that was allocated. In addition, it can test for stack and heap corruption.

This has made a huge difference in CMUD. When I first compiled CMUD using FastMM, it found all sorts of problems that tools like AQTime had missed, including an obscure memory leak in Delphi 7 that apparently has never been fixed even in Delphi 2006.

Because this works in real-time as long as the debug flag is enabled, every time I run CMUD I'm getting full memory leak detection. So as soon as I add any code that has a problem, I learn about it immediately when I run CMUD the next time. This makes it a lot easier to fix problems since the new code is still fresh in my mind.

Getting this detailed information about problems every time you compile and run instead of having to run an external tool like AQTime is incredible. This should mark the end of all memory leaks and memory problems in my software.

I can't say enough good things about FastMM. Every Delphi programmer should really use this. Stable, fast, and one of the best memory debugging tools I've ever used.

Posted: Tue Mar 14, 2006 9:01 pm

Yayyyy Zugg!!! The program sounds so cool it's almost enough to make me want to go out and learn Delphi. Laughing

(My Pascal is quite dated.)

I'm also happy because this means more stable and efficient code as you release it which should be good for your already stellar reputation, and help you release code faster since it's easier to identify potential issues.

Posted: Sun Mar 19, 2006 1:36 am

Of course, the problem with something cool like FastMM is that you end up having to fix *all* of the memory leaks...even the ones in Delphi.

Today I ran into an obscure memory leak in the Delphi docking code (have I mentioned before how much I hate the Delphi docking code? I think I have). Anyway, Borland does some really stupid stuff using global variables. There is something called DragObject which is the global object being dragged.

They play some games with freeing this global object and saving stuff to another global variable called DragSave. The problem is that the TDragDropObject constructor and destructors set DragSave to nil without checking to see if it should be freed first.

Internally, Delphi creates a second class called TDragDropObjectEx that overrides this behavior to avoid losing the value of DragSave. But they don't use this new class in one of their own routines (ManualDock) which causes a memory leak.

So I had to write my *own* ManualDock routine that creates TDragDropObjectEx instead of the normal TDragDropObject. This stuff is buried in the controls.pas file, which is one of the files that you can't modify and recompile without breaking Delphi.

Another four hours of debugging Borland code down the tubes. Man, if only I could charge them for all of the time I have ever spent fixing their own code!

Enchanter Joined: 10 Oct 2000 Posts: 698

I'm trying to feel your pain, Zugg, but all I can accomplish is a chuckle - sorry. Do you get any satisfaction in being resourceful and competent enough to fix stuff like that? I mean, you've got to know what you're doing to be able to debug, fix and then rip on other people's code like that. I'd find satisfaction in having my own application support my family, but I'd think that fixing somebody else's screwy code once in awhile would be even more satisfying because of the frustration it caused. Satisfaction like beating up the bully that's picking on your little brother. Maybe I'm wrong.

I'm really glad that this program works so well for you - FastMM, not Delphi. Great tools make tedius tasks tolerable.

Posted: Sun Mar 19, 2006 7:10 pm

Sometimes it brings satisfaction. But most of the time it just brings frustration. I get frustrated when I have paid a lot of money for buggy code. If I had lots of extra time then it might be more satisfying. But I'm so swamped with my own code to write that when I have to spend time fixing other people's code that I've paid hundreds of dollars for it just stresses me out as my release date gets pushed later and later.

As I've said in some other blogs, I'm a quality nut. I get really jazzed and excited when I get something that is high quality and works well regardless of the price (stuff like FastMM). When I get something that is buggy and crashes all of the time and causes me to waste valuable programming time, then I get unhappy and depressed.

But maybe your comments will help me the next time I get frustrated. I use your "beating up the bully" analogy and see if that makes me feel better.

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

Hooooo boy, I know this is off the topic of Delphi Memory Management - but it is Memory Management none the less.

A preface... our work has decided that we will be developing in C#.NET from now on. Now I don't necessarily mind that, it is an easy language to pick up and run with but CRIKEY! You can run into some interesting problems.

My problem (which took a full week of development to diagnose then fix) was as follows: We have to sync a large (well 4 meg) buffer of XML from the PC to the Pocket PC, then whack it into a database and stuff. So the first thing you think (because C# lets you be lazy) is use strings eg:
InputBuffer += TheStuffWeJustGotFromTheSocket;

However, the way C# implements that particular bit of code is:
NewInputBuffer = new string( OldInputBuffer, TheStuffWeJustGotFromTheSocket );
and assigns OldInputBuffer and TheStuffWeJustGotFromTheSocket to be disposed of 'later'

This means if you have 2 megs of input and you append another 20kb to it, it creates a new buffer of 2 megs 20 kb and copies the data. Now that's an inefficient use of processor, and a little chunky on the memory - though you would assume if you did a "Garbage Collection" immediately afterward it would free the memory. Not so much.

The garbage collector has a few levels of operation which is all to technical to get into, but basically small items (items less than 20kb in size) will get cleaned up pretty quickly, however large items (items over 20kb in size) don't appear to be cleaned up at all. Sure they're de-referenced, but the RAM isn't allocated back to the OS and it seems to become fragmented. What this was meaning is if you only have 32 megs of RAM because it's a PocketPC, and these large buffers don't get de-allocated you end up getting an out of memory exception after you've got about 1 meg of the file.

So, then you look a bit deeper. Microsoft suggest using a StringBuilder class since it can pre-allocate an amount of RAM. Well that's ok, I figure I'm going to get around a 4 meg file so if I create a string builder class with a 5 meg buffer then that should do the trick. Well, firstly strings in .NET are all double-byte, so the 5 meg buffer turns into 10 megs of memory, so while is borderline on the PocketPC, it's still do-able. The trouble comes from the whole bit about large buffers not being returned to the memory pool. When you have finished with the StringBuilder and it falls out of scope (or you set it's length to 0 in order to clear the buffer), then you run into this whole 'large buffer (this time 10 megs worth) not getting de-allocated' thing so that brings on the out of memory exception even faster.

Same deal with a MemoryStream - which does approximately the same thing as the StringBuilder, but only uses single byte characters instead of double byte.

So in the end, you say to yourself "Well, I got to stick to using buffers that sit under the 20k threshold". So you end up having to create an Array/List/LinkedList of buffers each at 16kb in size and writing a class similar to MemoryStream to wrap them, but instead of having one great big buffer you have many smaller buffers which don't face the de-allocation problem. It's nasty, but at least you no longer get out of memory exceptions.

At least now I know and now you know, so if you start wondering why 20k+ buffers are behaving like they're leaking memory, you'll have some idea of how to fix it.

Blargggh. Well I've vented. Time to go home and relax.

Posted: Thu Apr 06, 2006 11:56 am

Sometimes .NET really does suck.

I've also had issues with COM objects. It just doesn't clear them. I've followed instructions on the MS website that supposedly clear up the issue but to no avail.

So I have written a program that creates an Excel spreadsheet from scratch but the more you run it the more EXCEL.EXE processes are left running. It makes me look as if I can't program even though I've done everything I can to fix it Mad

Posted: Thu Apr 06, 2006 4:49 pm

Taz, have you tried getting the "active" Excel COM object rather than creating a new one? At least in Delphi there are two ways to create a COM object: CreateNew and CreateActive. I always try CreateActive first to see if the object is already in memory. If that fails then I create a new object. Not sure if this would help in your case or not. To clear them I usually just do:

Object := unassigned

(again in Delphi).

Rainchild, you raise a really good point. All of the new development stuff (.NET, etc) seems to work on the idea that programmers don't have to care about low-level details anymore. These days programmers don't need to worry about memory management, or disk space, or any of those things. Memory is cheap, disk space is cheap, and the developement tools are getting to a higher and higher level of abstraction with more layers between your code and the actual hardware.

This might have some advantages for desktop applications (where you have lots of memory and disk space), but it's horrible for PocketPC developers where memory still really matters. Same thing with development for cell-phones. Back in the "old days" I coded on a machine that only had 64 KB of memory and it had to share that with the UCSD Pascal operating system. So we had about 48 KB left. And we wrote some amazing code in that space, all with the "high-level" Pascal language.

Fewer and fewer developers are gaining the skills in things like memory management. It's a sad trend. So all of us who still care about memory and code efficiency are becoming a dying breed.

Magician Joined: 09 Oct 2003 Posts: 459 Location: USA

I ran into a similar problem while babysitting someone else's code in ASP.NET and Crystal Reports. After viewing any report multiple times in a short period, it would start to cause errors upon loading the report. I didn't write this and didn't know much about Crystal Reports at the time, so I did a wee bit of research, it appears that reports don't unload when the page unloads from the server's memory (this, btw, is an undocumented 'feature' of Crystal Reports and I only found the solution by running across a forum post where someone had figured out that trick... <3 google). So in the unload settings for the page, I had to set VB.NET to dispose of the crystal reports object. Babysitting code is never fun, but luckily that programmer is back now.

Posted: Fri Apr 07, 2006 1:02 pm

The so called fix that is posted on MS website is basically to call the dispose method for every COM object you created and then do a garbage collection. Doesn't work tho!!

I'll look into getting the active object rather than create a new object tho, thanks Smile