Zugg Software :: View topic

Posted: Thu May 11, 2006 1:28 am

OK, rather than posting a bunch of stuff into the CMUD ETA thread, I've decided to start this Blog thread on the CMUD Development Status. I'll try to post to this often and give the details of what I'm working on, what problems I'm dealing with, and what cool stuff I'm solving. My hope is that I'll feel better by "venting" in this blog, rather than in the "official" thread in the CMUD forum.

It will give those people who are really interested in nitty-gritty development details something to read. It probably won't be very interesting, but it might help to see "inside the developer's head". Also, it might help give some people a better idea of how close CMUD is to release.

Right now it's very depressing for me to post updates in the official ETA thread because it seems like CMUD is in a perpetual "done in 2 weeks" phase. On one hand, I know this is a common problem towards the end of a development project, but on the other hand it's still depressing. Looking on the positive side: CMUD is getting better and better every day. Looking on the negative side, it's still amazing how many bugs I'm finding, especially in code that I wrote months ago and supposedly already had working.

It's the typical problem with integration. Several modules, like the new settings/package editor, and the shared package library, were written as seperate modules, and integrating these into CMUD results in finding problems that I never found before.

Also, as I go through all of the zMUD features to make sure they are working in CMUD, it seems like I'm finding problems everywhere. Mostly they are little. But today I actually found a memory leak that has been in the old zMUD code for about 6 years. It's very obscure and unlikely to be triggered in many cases. It was good to find, but I still get annoyed when I find bugs in old code that should have been working a long time ago.

Today I cleaned up some of the code that interfaces the new compiler with the user interface. The idea is to compile aliases, triggers, etc as needed and then cache the results. So the next time an alias is used it doesn't need to be recompiled. I already had code that was doing this, but it was specific to the scripts. Now I'm using the compiler for a lot more functions, including any time variables are expanded.

For example, a Button can have a normal script that is executed when it is pressed, but the Caption of the button can also contain variables. So the Caption also gets compiled. So I needed to generalize the routines that handled the caching of compiled code to also handle strings that could contain variables, such as button captions, trigger patterns, menu captions, etc. I finally got all of that working, but it was a bit frustrating since it seemed to break everything for a few hours until I got the new classes debugged. I seem to have a hard time these days writing good code to start with. Probably because I'm so brain-fried these days.

Once I got the generalized compiler caching working well, I then tackled the trigger priority order. Fortunately, that was quick and easy. My new design *finally* paid off and made something simple. Basically, I created a new database index (for the in-memory database, not the on-disk version). This index filtered for just triggers and put them in priority order. Using this in-memory index I can then perform a direct database access to the resulting trigger records. So simply looping through the index gives me the triggers in proper priority order. A simple change to my old trigger loop to use this index and it all worked.

The current speed results are pretty impressive. I turned off the memory leak detector which slows down the code a bit just to see how fast CMUD currently is. It now runs my test script in less than 1 second. That's almost how much time it simply takes to scroll that much text onto the screen. By comparison, zMUD takes about 3 seconds to run the script.

My next job is to implement a smarter update system for buttons and status bars and trigger patterns. Using the new class that I created for caching compiled code, I plan to add a list of variables that a script depends upon. Since the script is compiled, it's trivial to gather a list of variable references within the script. Using this list, I should be able to keep track of exactly what needs to be updated when a particular variable is changed. I'm hoping that it's just a one-day job to get this working, but stay tuned.

Feel free to comment on this thread, or just ignore it. I'm going to try and keep it pretty active and post daily.

Magician Joined: 29 Nov 2005 Posts: 482

Just realize zugg that this is alpha/beta process so not everything has to be perfect first time around and you might find that some of your fixes on "your development system" may not have the same problem on another machine as they will be setup differently.

Its common and i do it in my own programs to try to perfect everything and i find myself always finding bugs. Some really are some arent. But you will get more feedback and a better understanding of operation on systems outside of your own once you feel comfortable enough to release it and begin beta testing. You will also recieve more informed feedback regarding some of these changes in parsing you discuss because we will be able to do real time testing and script conversions to offer better informed feedback.

Just a couple of thoughts for you to take with ya. Thats exactly why i make sure one of the first things that works in any software i write is the auto update feature. Then i can release it and any changes i make are immediately updated in the software.

Posted: Thu May 11, 2006 2:28 am

This is one of the main reasons I come back to read your posts. The chance to "get into your head" is fascinating and quite entertaining all by itself. I have a few RSS feeds that I read, but I find yours the most enjoyable. I especially enjoy reading your rants about things go wrong, because I can vicarously throw a fit, through you. Twisted Evil

I find small business owners, and especially software small business owners, entralling.

The stuff in the context of programming everyday and seeing CMUD grow is something that pulls me in again and again.

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

I was kinda missing reading posts, it's one of the things I do at work to kill boredom from time to time hehe so when the forums aren't active, I'ma saaad panda hehe.

I wonder if it is worth doing a closed beta with just a handful of invited people in order to get a fresh set of eyes on it - maybe like edb6377 said, it's close enough and some of the things you're doing don't really need doing.

At least what you said with the caching of the variable references - is that really necessary for the first release? If we're already 3x faster than zMUD, you don't really need to spend an extra day to make it 3.2x faster for first impression?

Just a thought :)

Edit: btw I'm not saying I should be one of the handful of invited (if you even go that way) in fact I'm probably not the right type of person since I just use zMUD for adminning nowdays, no triggers or anything complex really required.

Enchanter Joined: 10 Oct 2000 Posts: 698

Posted: Thu May 11, 2006 2:38 pm

All I can say is that I get excited everytime i read the forum posts... I've been on the edge of my seats waiting for it's release. I've not even done that much new scripting in zMUD, because CMUD is where it's at. I understand the fustrations that you encounter, but we all appreciate you're hard work, and it will be rewarded.

We're all waiting for the release, but enjoying being in the developers head while it's in progress. Especially since you can whisper in his ear. Very Happy

Out of curiosity how is Chiara responding to CMUD. In terms of use, new user experience and intuitiveness. How else has her MUDding experience affected development. Is she still MUDding? What's her experience like?

Site Admin Joined: 29 Sep 2000 Posts: 389 Location: USA

Chiara's mudding has been set aside for a while. To enable Zugg to spend all his time programming, I am handling everything else for the household, plus my email duties, so I'm a little pressed for time.

CMUD is not yet stable enough for me to use. But I do see all the new screens, and we've discussed all the newbie features he's adding, and I'm looking forward to it as much as anyone.

Plus, I'd like my husband back. ;)

Since this is the venting thread, I'd just like everyone to know that when I say Zugg is spending ALL his time programming, that is not the exaggeration of a fond wife. I mean he sits in his chair at 10 am and gets up for restroom breaks. Lunch, an afternoon snack, and more often than not dinner, are delivered to his desk. He doesn't take phone calls, go to meetings, have watercooler discussions, or have friends drop by his cube to chat. 6 days a week. For what seems like forever. Lately he quits between 9 and 11. Which is about when I go to bed.

NO ONE will be happier to see this release than me, except possibly him.

Posted: Thu May 11, 2006 5:14 pm

Thanks for the supportive comments! I'll try to keep posting more "inside my head" reponses to this blog entry.

Just a quick note this morning though: as far as the bugs being fixed, they really are necessary. I understand, as edb6377, that not every bug I might find is severe, and I understand that I'm a perfectionist with high standards for software. But even with those high standards, there are still plenty of bugs left for beta testers to find. Even though I'm programming better than I was a few years ago, the first release *is* going to still be buggy and anything I can find and fix myself will help. And people who have been around for a while to remember the height of zMUD beta releases will remember how buggy they can still be sometimes.

The obscure bug that was in zMUD for the past 6 years was in the COM system. In zMUD, if you assign a COM variable like this:

Posted: Thu May 11, 2006 5:58 pm

I for one will appreciate the changes to #MESSAGE. I used it a few times but got fustrated with how it worked. I could see it was one of the older parts of zMud.

Chiara, when the first beta is released I, for one, promise not to submit any bugs for a week to give you guys a chance to actually spend tme together again. Smile

Wizard Joined: 14 Aug 2004 Posts: 1269

Zugg, you're a far more developed programmer than I, and also everyone is different, but I think it's a good idea to take breaks from work from time to time, e.g. lunch, dinner, other short breaks for a drink or a snack and whenever you get firmly stuck on something. I find sometimes that if I'm stuck on something, and go and make myself a drink and have a piece of toast, I can get a fresh perspective on the problem while I'm away from my desk or when I return and start looking at it again. It also might help the brain-fried factor, not to mention being appreciated by your wife, if you get away from your desk for meals. Wink

And if it reduces the brain-fried factor, maybe it will result in fewer bugs and less re-coding.

I like the updates on CMUD too. BTW, I assume that local variables don't go near the database though?

SubAdmin Joined: 18 Nov 2001 Posts: 5182

Chiara, hang in there. We will give Zugg back soon Exclamation

We care about your sanity and health just as much as Zugg's. We just forget to show it. If you happen to find a few spare moments maybe you could update us on all the other things like the low carb diet. I know I always like hearing from you.

Zugg, in many ways continous database updates really aren't needed. The time spent saving to disk is huge compared to memory allocation time. I would say, based on my own usage, that 3/4 of all variables will be replaced with temporary local variables, 3/16 would be temporary, but have to be 'global' and only 1/16th of all my variables actually need to be stored. Perhaps you really should think about how to let the user specify what needs to be saved to disk immediately, versus what just needs to be available to other portions of script.

I know it is sort of a kludge, but the reality is that variables in nearly all programming scenarios work close to the how the human brain operates. Only about 1 millionth of the data actually taken in, by a given person in a given day, is committed to long term storage. Humans of course take in and process a much larger amount of data then a task specific program will, but the pattern tends to be the same. Think about allowing a user to flag something about saving the variable, maybe save on close, or not at all (default of save on change). We still need some global temporary variables, without them new users would never be able to do certain things.

Screen updates are definitely the bane of speed. I am glad to hear that your going to make an index capable to reducing the freqency of updates so that the same information is not just being repainted.

Wizard Joined: 14 Aug 2004 Posts: 1269

I was thinking along the same lines as Vijilante but missed it from my previous post: some of my variable assignments, perhaps also about half could be moved to local variables, and probably half of the rest do not need to be saved between sessions. There is already a flag in zMUD - the Use Default flag - which effectively means the contents of a variable are not stored between sessions. So really only the Default needs to be saved to the database - the actual value can just always be in memory (for variables wih Use Default)...

Posted: Fri May 12, 2006 2:04 am

I think it's time to explain some of the new internal guts of CMUD. Don't worry, I've already considered the issue of database updates. Obviously if I'm always saving variables to the database then CMUD would be slow slow slow! Here is the architecture that is already implemented:

Settings (aliases, triggers, variables, etc) come from a Package file. This is a SQLITE database file. This disk database is read entirely into memory and populates an "in memory" database. Specifically, I use a component called kbmMemTable which is a *very fast* in-memory TDataset descendant for Delphi (it's a great free 3rd party component btw). kbmMemTable can stream *any* dataset into memory, including the SQLITE stuff from the Zeos data components that I'm using for the disk database.

Once the disk database has been read into memory, the disk database is closed and no longer takes up memory space. So we just have a single copy of the data in memory at this point. When a second package is loaded, the disk database is read into the *same* in-memory database. So the in-memory database is a "collection" of packages. Each package remembers the name of the disk file that it came from.

OK, now, in zMUD, settings were stored as a linked-list of various Delphi classes. There is a class hierarchy of settings, all descended from a master "setting" class. Aliases, Triggers, Macros, Variables, etc all descend from this master parent class. In order to re-use as much zMUD code as possible, I need to recreate this linked-list of settings records. But I don't want to store the data in memory more than once. So, the properties of the setting classes actually store and retrieve data from the proper record of the master in-memory database.

I've written some low-level access routines that let me store pointers to the actual records in the in-memory database, so fetching fields is very quick. I don't need to search for a "key" field in the database or anything like that. So retrieving data is very fast. Storing is a bit slower because doing any low level *writing* to the in-memory records is dangerous, and would also bypass stuff like the database index updates. So storing a value to a setting performs a normal in-memory database update. This is still relatively fast since it doesn't need to write to the disk.

One of the really cool features of the kbmMemData in-memory database is that it has "versioning". Full versioning allows it to save multiple versions of the same record. Simple versioning (which is what CMUD uses) tells it to keep the original value and the current value. Thus, you can always tell which records of the in-memory database have changed (or been added, or deleted). A "resolver" can loop through the in-memory database and generate the proper SQL commands to update the disk file.

Currently, this update happens whenever you exit the settings editor (or click the Save button), or when you exit CMUD. Note that it only needs to perform the SQL updates for stuff that has changed.

In zMUD, the raw memory contents of the linked-list records were streamed to the *.MUD settings file as binary data. This is why settings could get easily corrupted. If any of the many pointers maintaining the linked list got messed up (like from an access violation), then zMUD would be saving invalid memory locations to the settings file. Also, zMUD always saved the *entire* file. In CMUD, we are using SQL database updates to save stuff back to disk, which goes a long way towards preventing settings corruption. And we are only saving those values that actually changed.

So far, this works *extremely* well. The save that is done when exiting the settings editor is hardly noticeable for reasonable setting file size. This is because it only needs to update stuff that changed. However, if you change a lot of stuff, then it gets noticeable.

What I want to try to implement this week is a "background" saving thread. This thread would loop through the in-memory database looking for stuff to save. It would run at a low priority to keep stuff updated without you noticing it. It would have a "forced" mode when CMUD exits to make sure everything has been written to disk before the program exits. I don't yet know how easy this will be...it depends upon how thread-safe the database code is. If I can't get this working, then I'll probably stick with the current system for a while.

As far as variables: local variables do not touch the database at all...not even the in-memory database. Local variables are part of the runtime stack of the compiler and executor modules in CMUD. They are created and destroyed dynamically as scripts run. That's why they are local and go away when your script is done. But they are *very* fast.

The idea of not saving any variable to disk that has the "Use Default" flag set is a really good idea. I hadn't thought of that. But I'll definitely implement that. This is also how "temporary" settings work. Things like temporary triggers are stored in the in-memory database, but not written to disk. Actually, this doesn't work yet, but it's on my to-do list for the first beta. Just one of those loose ends I haven't tied up yet. Also, it can't store variables to disk that contain COM objects or COM arrays (just like zMUD).

So that's how CMUD works. It's pretty complex stuff, but when it works, it works really well. Unfortunately when it doesn't work, all hell breaks loose. The worst kind of bug is when the linked-list settings get out of synch with the in-memory database. For example, all of the command line stuff uses the linked-list form of settings (because that's the code that was ported from zMUD). The package/settings editor is brand new and it only uses the in-memory database and ignores the linked-list. But obviously, if you add a new setting in the setting editor, then a new record needs to get added to the linked list so that your scripts will see it. And vise-versa, if you use a command line #ALIAS command to create a new alias, this creates a new linked-list record but also needs to add a new record to the in-memory database.

This synchronization can get really messy sometimes. Some of the old zMUD code wasn't very consistent about how it created and destroyed settings. I found some code that bypassed my class hierarchy and was creating a destroying settings records itself, which was bypassing the database. Going the other way, when you delete a setting in the in-memory database, it doesn't get rid of the record, it just sets a "deleted" flag. This is how it decides to issue an SQL DELETE request to the disk file when it's updated. However, even though the record is marked as deleted, the old data is still there. At once point, the linked-list structure was still returning valid data for these deleted records, and so the command line and scripts still thought that the setting was valid.

Anyway, it's taken a while to get the bugs out of this. Fortunately, CMUD depends so heavily on the proper synchronization between the database and the linked-list that when there's a bug, it's pretty obvious. It's not necessarily easy to fix, but it's really obvious that *something* isn't working properly.

Wow, lots of detail you probably didn't care about Razz

Anyway, don't worry...I've thought a lot about this. And that's one of the reasons it's taking a while. Even though CMUD looks similar to zMUD in some ways, it's really completely different when you get down to these internal guts. But that's what makes CMUD a better program that can do more and do it faster.

-----
Today was a pretty typical day: I actually got the update dependency stuff working very quickly. About an hour to write it and about another hour to get it debugged and working. Got the button captions updating correctly first. Then ran into a wierd problem. If I had a button caption set to something like "hp: @hp" and then started changing the @hp variable, I started getting multiple captions that were overwriting each other on the button, creating a mess.

First I thought this was caused by a problem with the button transparency that I had to add to support theming. The button component used for zMUD buttons is *very* old. It was originally a 3rd party component that I *heavily* modified over the years (to support stuff like gauges). While I'm doing some new stuff in CMUD that will be better, these old buttons still exist for compatibility. I know some people have spent a *lot* of time tweaking their button positions and I didn't want to break that. Anyway, I started looking at the transparency for this old button code and couldn't find any problem.

After several frustrating hours of trying to track this down, I finally discovered that it was actually creating multiple buttons on top of each other. Not CMUD buttons, but the actual 3rd party button control itself. When the button was updated, a new visual button component was being created, overwriting the old one, but the old one wasn't getting freed. It didn't show up as a memory leak because the visual button still was owned by the main CMUD form and was being removed properly when the main form was destroyed at exit time.

From what I can tell, I think this bug was actually in zMUD too. I couldn't figure out exactly how to trigger it in zMUD, but the statement where the button was set to NIL without destroying it first *was* in the zMUD code. So this might be yet another problem in zMUD that I've fixed which would cause memory to grow as buttons were changed, but without being a technical memory leak.

So that killed a few hours. After I got the buttons fixed I then worked on the status bar and status window items. I *finally* split these two things up. It was a real kludge in zMUD to have a setting where the "name" was a status *bar* item and the "value" was a status *window* item. These are now split. There are check boxes at the bottom to decide whether you want a specific status item to be displayed on the status bar, or in the status window. Or leave them unchecked and it won't be displayed in either place (but will be available in other ways that I'm not ready to reveal yet Cool

)

This worked great for the status bar. For the status window I had to do some more work getting it integrated with the docking system. For now I've turned the status window into a normal "tool window" that has a caption bar when undocked. Since the single status window will eventually be going away anyway (replaced with zApp windows), I didn't want to put a lot of time into it.

Then, I worked on the trigger patterns and getting them working with the new dependency update. Each trigger already kept a cache of Perl RegEx stuff for speed, so it was easy to integrate with this. In zMUD, everytime a line was received, each trigger that contained a variable reference had to be re-parsed, converted to RegEx, and then pattern matched. Now in CMUD, this only happens when a variable that is used in the pattern is changed. Otherwise everything is cached from line to line, including the internal state of the RegEx engine for each trigger (and condition). If you have used Regular Expressions, you know that they get "compiled" in a way. So what I'm saying is that this compiled status also gets saved.

This change to triggers has made another big speed improvement. Using a trigger with variables is now as fast as a literal string trigger. This will be most noticeable when using big string lists, such as {{@friends}} and patterns like that. In zMUD, these kind of triggers can be really slow due to the parsing that is happening on each line. In CMUD they are really fast.

So, all in all it was a pretty typical day...the stuff that I thought was going to be hard worked pretty easily, and I got bogged down in some wierd bug that I never expected.

Oh yeah, and if anyone is still reading...(Seb) Yes, I do take breaks. Chiara was exaggerating a bit. We tend to go out for lunch and dinner a lot. When I get frustrated and need a break, we get out of the house and go eat. Runs up the food bill, but it helps me clear my mind. That's one drawback to working at home...when you need to get out of the office, you have to leave home too :) But when I'm in "the zone" then Chiara brings me my lunch and dinner as she mentioned. If you have ever done a lot of programming, you will know how important "the zone" is. It's a nebulous state that you can't force yourself into. But once you are in "the zone" then you can write some amazing code. I was in "the zone" for a while today, until I ran into that button bug. It's great when it happens, and Chiara has done a good job learning to detect when I'm in "the zone" and not interrupt me.

Chiara is amazingly tolerant of all of this. Our schedule totally revolves around me right now. If I'm in the zone, then she brings me food at proper intervals. If I suddenly get frustrated and decide I need to get out, then she drops whatever she is doing and we go out for dinner. CMUD wouldn't be possible without her, which is why it's CHIARA'S Mud Client.

OK, that's plenty for tonight. I enjoy reading your feedback so keep it coming if you still find this interesting.

Posted: Fri May 12, 2006 3:11 am

Interesting? More like trying to put down one of those Terry Goodkind, Tom Clancy, or Dan Brown books. Can't stop reading it until the end. Very Happy

Enchanter Joined: 10 Oct 2000 Posts: 698

Beginner Joined: 24 Nov 2002 Posts: 20 Location: Australia

One question - ever thought of taking up polyphasic sleeping? That way you can code 21 hours a day and possibly squeeze in 20 minutes or so for Chiara. Twisted Evil

Posted: Fri May 12, 2006 6:28 am

The in-memory database is *always* up to date. The background updating is for the disk file. And yes, we need that. One of the whole points of this new design was to eliminate the corrupted settings file problem -- it's the biggest support issue in zMUD. I don't want people to need to save their settings. I would like them to be always saved, just like how a normal database works. The CMUD design allows this, and the background saving would be the final piece of this design.

The likelyhood of a crash goes up tremendously during program exit. It's during program exit that Windows tries to reclaim it's memory and resources and problems earlier in the program can cause access violations to occur then. So saving settings at program exit isn't a very good idea...it's one of the problems zMUD had. I don't know how easy it is to corrupt an SQLITE database file, but I don't really want to find out.

But don't worry. If it takes more than a day or two to get background updates working then I'll leave it for another time. But I'd really like to finish the core design and have it be nice and stable, and having a reliable settings database is part of this.

Magician Joined: 09 Oct 2003 Posts: 459 Location: USA

Magician Joined: 29 Nov 2005 Posts: 482

Wizard Joined: 14 Aug 2004 Posts: 1269

Yes, I know about "the zone", and I can understand not wanting to leave it when you're in it. And deciding now is a good time for dinner when frustration builds up.

Sounds like good progress - I found your post very interesting and I'm excited by the idea that CMUD is nearly ready for beta release.

Wizard Joined: 25 Mar 2003 Posts: 1113 Location: USA

Even though that was the longest (non-log) post I've ever seen, I read every single word of it. I found all the technical stuff extremely interesting, and I'm looking forward to getting my hands on CMUD now more than ever. Keep up the great work, Zugg!

Posted: Fri May 12, 2006 1:50 pm

Posted: Fri May 12, 2006 3:42 pm

Zugg it was great to hear the design behind the CMUD settings. It is clearly well thought out, shows the experience you have, and is quite frankly fascinating. But perhaps, that's just the coder in me. On more than on occasion, I've been tempted to dust off my dated Pascal knowledge, and try my hands at Delphi, based solely on all the cool things you are doing with it.

I can hardly wait for the next one update. Very Happy

Posted: Fri May 12, 2006 4:45 pm

Posted: Sat May 13, 2006 2:40 am

Today was a pretty good day. Challenging, but productive and with a good ending.

Today's job was the background database saving. I was a bit nervous about this because dealing with background threads is always a challenge. But last year I did a lot of work with threads with zApp. One of the lessons I learned was to use the Indy Internet Components TidThread rather than the direct Delphi TThread. The INDY TidThread has a better execution method that is designed for threads that sit in the background and do stuff periodically (like network servers). The INDY code is well debugged and the TidThread object worked very well for me last year with zApp.

The real question was whether the database stuff was going to be "thread safe". I won't get into a whole discussion of threads, but what you need to keep in mind is that your thread code can be executed at *any* time, interrupting other processing that is happening. So you need to be careful with what you do in your thread so that you don't cause problems when interrupting normal processing.

This also means that threads are hard to debug because there might just be one small place somewhere in the code that would cause a problem if your thread executed right at that exact place. Causing this to happen on demand to ensure there are no problems can be tricky, resulting in thread problems that show up much later in the debugging.

The kbmMemTable in-memory database component already had some code to loop through all records, decide which records have been changed, inserted, or deleted, and handle updates. I had already written code for zApp to handle SQL updates, generating the correct SQL commands for each of these cases. So writing the thread code was pretty easy. I mainly just copied the code from the kbmMemTable Resolve routine, and from my zApp routines and put it all into a TidThread component.

The main execution loop of this thread looks at each record in the database to see what has changed. As each record is updated, the SQL commands are enclosed in SQL transactions (BEGIN TRANSACTION, COMMIT TRANSACTION) to ensure database integrity. After the SQL command has been successfully sent, the record in the database is marked as "resolved". After looping through all records, the thread goes to sleep.

I added a routine to resume the thread, and also another routine that would force two loops through the thread to ensure that the full database is saved (it might be triggered when the loop is already in progress, so the second pass ensures that nothing was missed). I call the first routine everytime anything in the database is changed (in the AfterPost, AfterDelete Delphi events). I call the second routine when you exit CMUD.

Now, this was just the first test, so I *wanted* to resume the thread on *every* change to the database. I basically wanted to flood this thread with calls so that it was almost always running. The more the thread was running, the more chance I had to catch thread problems.

It actually worked pretty well the first time. But after a few executions of my test script, it crashed. My test script does a *lot* of complicated scripting, including deleting settings and then checking to see if they were gone. Something related to deleting settings was causing a problem.

A couple of months ago, I fixed a memory leak in kbmMemTable related to deleting records. It wasn't properly freeing up the previous "versions" of the record (remember, it keeps track of the original record and the current record to determine what needs to be changed). When I removed my memory leak fix, then I didn't get any crashes from the thread (but the memory leak came back), so I knew it was somehow all related. Perhaps I wasn't fixing the memory leak properly.

Well, several hours later after learning more about how the kbmMemTable code worked and how they implemented versioning, I finally found the solution. They were not marking a copy of a record properly so that it would get freed when the database was closed. That fixed the memory leak and made things better. But then I'd still get some thread crashes.

I *finally* realized what was happening. The database has an internal list of records that contain pointers to the original version of each record. But each database table also has an "active buffer" which is a *copy* of one of these records, but still pointing to the same original version fields. When a record was deleted, the background thread would send the SQL DELETE command, and then remove the original version of the field (since the record was now resolved). But the ActiveBuffer copy still pointed to the original version data that we just freed, and when it tried to access that memory, it would give an access violation.

So the solution was to check all of the ActiveBuffers to see if they matched the current record being resolved, and it so, clear the pointer to the original data in the ActiveBuffer.

It worked!! I ran my test script continuously for about 100 interations and had no crashes. And the database file on disk was being kept up-to-date instantly, with about 20% reduction in speed.

So, now that I trusted the thread, I changed it so that it only runs every 10 seconds if needed. When something in the in-memory database is changed, a flag is set (NeedsUpdating), and every 10 seconds CMUD checks this flag and if it is set, then it wakes up the background thread.

The resulting performance hit for the database saving is not even noticeable now. My test scripts run at full speed, and the disk file is at most 10 seconds out of date. When you exit CMUD, it forces a full save before exiting. We now have all of the benefits of database integrity, without any performance hits. Woohoo!!! Very Happy

-----
My next job was to work on the Exception handling routines for CMUD. I wanted to set up a system where I could have the end-user send me stack dumps when CMUD crashes. I wanted to figure out how to capture a traceback of the stack where the crash occured, preferably with line numbers and routine names. And I wanted to either have the user send the crash report via email, or have it posted to our web server.

So first I needed to figure out how to determine the call-back trace and line numbers and routine names. So, I started doing a Google search. After about an hour of loose ends, I was about to give up and do it all myself, when I ran across a real Gem called madExcept ( http://www.madshi.net/madExceptDescription.htm ).

Wow, here is a Delphi component (also for C++Builder) that does it *all*. And this guy really knows what he is doing. He doesn't even use the Delphi VCL for his buttons and other components since the VCL isn't threadsafe. He does everything with the WinAPI directly. He uses the *.MAP linker debug file from Delphi, which contains stuff like routine names and line numbers, but not as much stuff as the T32 debug files. He also encrypts this data to make it a bit harder on the hackers. He includes a fully customizeable user interface, and includes full email sending and/or web posting. It is truely an amazing tool. One of those rare finds that brings a smile to my face...finding someone else who cares about software quality is always nice.

After trying the demo and playing with it a bit, I immediately bought a copy. It's going to be a huge nice addition to the CMUD beta testing and will make bug reports easier to send by customers, and easier to read and interpret by myself. I don't know how I lived without this for so long. Gee, can you tell I'm happy?? Mr. Green

So, today was a great day. I expected the background saving to take two days, and it took less than one day. I expected to spend several days on the exception handling and bug report sending code, and it took only 3 hours. I love it when I have days like this!