Zugg Software :: View topic

SubAdmin Joined: 18 Nov 2001 Posts: 5182

I have been working for quite the bunch of hours on spliting up my main settings into appropiate packages. Anyhow, drag and drop move operations between packages work well until trying to handle a large class.

Having tried this repeatedly and finally with the Auto Update off I can say that there are 2 different problems at work.

With the Auto Update off only a portion of the settings are copied, however the deletion seems to get issued for the top class. I think that if the current bug about settings within a class not getting deleted properly didn't exist then many of those settings I was trying to move would have become lost.

When Auto Update is on the problem becomes really nasty. In most attempts the portions of the settings that do not get moved are duplicated in the original source file and location.

Kludge warning:
I think a quick fix for a portion of this would be to disable the auto updating during copy, move, and delete operations. That will at least take care of part of the problem until after the public release when the real problem can be addressed in further betas.

Posted: Wed Nov 29, 2006 1:58 am

How large is a "large" class? If it's in a pkg file that you've sent me before, let me know which package and which class you are dragging. I just dragged a large class and didn't see any problems with it at all.

Auto update actually *is* turned off during deletes and move/copy. That's why I wanted you to check it. I'm not sure what it causing it to get turned on again in your situation. And I have no idea how it could possible not move all of the settings. Did you try clicking the Refresh button in the toolbar to force the tree to refresh to see if they were really still there but just not being displayed.

I verified one of your bug reports about the tabs getting confused (and showing Default settings), but I haven't been able to find any other problem with moving or drag/drop. Maybe my computer is too fast or something?

I'm really getting frustrated that I can't reproduce so many of the bugs people are reporting.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

I am actually looking to move the ACP class from the achaea_def.mud file I sent you after combining in a small portion from the ACPCustom class that I wanted to keep. The source text to create it for zMud totals about 186Kb.

After completing the creation of the new package by way of copying the full import package to a new file and deleting out the settings I don't want. I then closed CMud and relaunched, reopened my new package, and then used Save Package as... That last step being necessary to clean out the database of all the deleted stuff; the resultant package is 1.59Mb. If you want I could feedback a screen shot from a failed copy. I can also send you the good resultant package I have.

I only tested with Auto Update off only once starting from a fresh import, making sure the tabs were right, then attempting the move before posting here. The difference in behavior was enough for me to believe that the Auto Update wasn't getting shut off. In the cases of every failed copy or move attempt I tried closing CMud, relaunching, and opening the involved packages; most were ruined in many various ways. In most cases badly enough that I deleted the files, got a new import and tried to see if I could get a better result by a different method. I still have one package in my recycle bin that resulted in some sort of triplicate for the class, let me know if you want me to send you that one.

Looking around on the internet this seems like it might be a decent 'slowdown' software http://www.hpaa.com/moslo/4biz.asp

Posted: Wed Nov 29, 2006 5:50 pm

Actually, I think what I'll do is increase the auto-update frequency timer to make it happen more often. That should make it fail more easily and help me track down the problem. I'm already using critical sections to keep the auto-update from happening during something else, but something is probably firing off a Windows message that is doing something outside of the critical section.

I'll be working on this most of the day, so I'll let you know what I find.

Posted: Thu Nov 30, 2006 12:04 am

Well, I was able to reproduce the problem. I cranked up the AutoUpdate interval to 0.1 sec instead of 1 sec. Then I loaded your achaea_def file and tried dragging the ACP class to a new package. And it only moved a partial amount...the rest seem to be lost. So now that I've reproduced it, I should hopefully be able to finally fix it!

More later.

Posted: Thu Nov 30, 2006 12:14 am

Btw, what's with your alarm that does this:

#alarm *.501 {#WINDOW Channels;#WINDOW Room;#MENU {Settings|Edit Settings};#MENU {Windows|Automapper}}

This really drives CMUD crazy since it causes the window focus to move around every half second. This was converted from your *.MUD file and I can't even see why you would ever do this in zMUD. Was it some sort of testing?

Wizard Joined: 03 Mar 2001 Posts: 1127 Location: London

I'd hazard a guess that it was to stop the zmud icon in the taskbar turning green on trivial mud output
I used to have something similar to switch focus quickly when I got tells, because they were rarely important enough to need to distract me.

Speaking of which, is there any chance of being able to set certain output to not change the taskbar icon green? It was something requested a few times during the pre-release suggestions requests

Posted: Thu Nov 30, 2006 1:19 am

Well, I find the problem. It's a wierd technical one with Delphi databases.

The MoveToPackage routine looked something like this:

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

To me that looks like you're doing a linear search of every record in the database... can't you do a filter on parentid = x or some kind of "SELECT * FROM settings WHERE parentid = 'x'" and use the indexed searches that will be a ton quicker?

It gets a bit harder if it's gotta swap files, I guess since you will have to do an insert and a delete, rather than just an update.

Wizard Joined: 14 Aug 2004 Posts: 1269

Rainchild, the problem with that is that some of that selection will also have child settings that you need to move. That's why Zugg is doing it recursively.

Another approach though that we use at work to support a system of accounts, sub-accounts, sub-sub-accounts, etc. is to have an account "path" field in the MS-SQL database, in addition to what is basically a parentid (MasterAccountID). So, if you have ID 1, with childid 4, that has a childid of 23, id 23 will have an idpath of "/1/4/23/" (leading and trailing slashes are important). Then if you want to move id 1 and all its decendents, you can simply

Posted: Thu Nov 30, 2006 3:17 am

First, keep in mind that the *internal* database is NOT an SQL database. It's a Delphi Dataset, so I can do filters (which is what the settings editor already does when you search or click one of the filter buttons), but I can't just issue SQL update commands to it. Using a full SQL database for the internal database proved to be incredibly slow (that was something I played with over a year ago in a feasibility study). The internal database is a kbmMemTable 3rd party component, which is a *very fast* Delphi in-memory dataset.

OK, with that said, the trouble with stuff like putting a Path field into each record is that the ID field is only unique within an individual package. Once you start loading multiple packages into memory, then the ID fields are changed to have the package ID in the upper 10 bits to keep each ID unique.

In other words, Package A and Package B might both have records with ID values of 1. So you can't load both packages into a single internal database because the ID field is not unique. So when a package is read into memory, the ID values are changed to stuff the package index into them to make them unique.

Doing this would invalidate all of your Paths. Because you never know which package number you will be loaded into. And you might even be loading the same package more than once. You can see why this all gets *very* complicated.

To store the Path, then CMUD would also need to modify the Path field when it was read into memory. But then you have another problem...if you just move one class to a different location, now you have to go through the entire database to change all of the Path fields for every setting contained in that class (and all subclasses). So you haven't actually solved anything. Sure, you can update the Path field using the SQL statement that you mentioned, but that's still slow, and doesn't work with the patched ID values.

The way CMUD works right now is that normal modifications are very quick...just a single Parent field needs to be changed. It's only when you move something to a new package that it gets slow, because changing the package means changing the upper 10 bits of the ID field. And once the ID field is changed, you need to update the Parent fields of all child settings recursively. I designed it this way because moving stuff between different packages happens less often than moving stuff between class folders within a single class.

In any case, I was able to improve the recursive algorithm so that it wasn't so slow. Instead of making it completely recursive (which is easier to code and understand, but slow), I was able to flatten it out so that it can process each "level" at once. So if your settings are 3 layers deep in class folders, then it only has to loop through the database 3 times (instead of N times where N is the number of settings). So this has made a big speed improvement.

I'm still running into some other problems related to the Parent fields not getting updated properly, so I'm still working on this. I'll probably take a break and work on it again when I'm fresh in the morning.

Wizard Joined: 14 Aug 2004 Posts: 1269

Rainchild and I figured it might be possible to decouple from the kbmMemTable and reload it from the SQLite database after finishing the transaction, I guess.

I remember something in your blog about this, but is there a reason that you don't keep the packageid in the internal database (in a separate field)? Can't kbmMemTable have a primary key made up of two columns (packageid and ID)? I suppose in the dataset that would be stored internally as a hash of the two values anyway, similar to what you are doing.

Anyway, sounds like you're getting there.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

That particular alarm was my way of making an onload event in zMud. It has a child state that keeps it from ever firing a second time until the session is closed and reopened. If it was causing major problems then its child state got lost in a copy from package to package. I have seen that happen for quite a bunch of triggers, but could never nail down a method to just produce that bug. I am glad to hear you were able to find the source of this one, and hopefully they are all related.

Posted: Thu Nov 30, 2006 6:34 pm

Vijilante: Ahh, ok, that makes sense now. I see the extra child state that is marked as manual. And yes, this seems to somehow get lost, so I'll add this as another bug to track down. Fortunately I can just click the NoTriggers icon in the command line to stop it from looping when I'm debugging.

Seb: The kbmMemTable doesn't handle primary keys that are made up of two columns very well. And as you said, the end result would be the same. I'd still have to recurse through the database to change the PkgId field, so that doesn't save anything. And decoupling from the kbmMemTable and reloading the SQL database would take a lot of time. It would take as long as it normally does to load your package files in the first place, which is pretty slow for large packages. And it would actually take even longer because I'd have to unload the package from memory first to get rid of the linked list cache, then reload the package and recreate the linked-list cache. And then I'd be tied to a particular set of SQL features needed on the back end. Right now CMUD could use *any* database server for it's back end (currently just uses SQLite) because CMUD doesn't need anything beyond the normal SELECT, UPDATE, INSERT and DELETE SQL statements.

Don't worry, I have spent a lot of time looking at database issues, and chose the current architecture very carefully. It's sort of like using the ADO database in the mapper...a database sounds like a really good idea until you try it and see the performance issues. Without the fast internal kbmMemTable, there is no way CMUD would ever work well enough for anyone to use. For example, if you have a normal external SQL database (even SQLite which is fast and mostly loaded into memory anyway), then do a query "SELECT * FROM Triggers WHERE Enabled=1 ORDER BY Priority" and then go through this list to test triggers, then you end up with a performance that is hundreds of times slower than CMUD is currently. So there is a huge difference between "traditional" database applications and what CMUD needs.

Posted: Fri Dec 01, 2006 12:21 am

Well, this set of bugs is literally killing me. I've been working on this for about 16 hours or so now (yesterday and today). I think I FINALLY have it working, but what a mess this all was.

And, unfortunately, there are still some kludges that will probably haunt me for a long time.

The problem was with trigger states. When trigger states were added to zMUD, it was a bit of a kludge. In the past there was a single TriggerRecord structure. It had properties such as Pattern and Script, etc. To implement trigger states, I added an array of TriggerStates and created a property called CurrentState. Then I just modified all of the other properties to return the value of the current state. For example, TriggerRecord.Pattern now returns TriggerStates[CurrentState].Pattern, etc.

This allowed most of the zMUD code that handled triggers to remain unchanged. To the rest of the program, the trigger states were hidden. The pattern and script and other properties of a trigger would just magically change when the CurrentState value was changed.

When this was converted into CMUD, I needed a way to store these triggers in the database. When a trigger only has one state, you still have the main TriggerRecord structure to create, plus one TriggerState structure to hold the single state. When you add additional states, then you only need the TriggerState structure for each subsequent state.

To synchonize the CMUD database with the internal linked list record structure, there is an internal database field that contains a pointer to the linked-list record. For the main trigger, this points to the TriggerRecord object. For a substate of a trigger, the database points to the TriggerState object.

This gets confusing when you are talking about a single-state trigger. The database points to the main TriggerRecord, but nobody points to the single TriggerState node.

So, if you loop through the database to update stuff, like the package id, the main TriggerRecord gets updated, but the first TriggerState node doesn't get updated.

This makes my head hurt just thinking about it again. PLEASE DONT post any "ideas" on how this could be better. I'm well aware of how it could be better, but I don't have the time to fix it right now. Let's just say that the synchronization between the database and the linked-list structures was getting screwed up by the MoveToPackage routine and that it was a real pain to track down all of the problems and fix them.

I've probably loaded and converted Vijilante's achaea_def.mud file hundreds of times now and know most of the ACP class by heart. It certainly provided a good test case, even though I really got sick of it after a while.

I still need to test it a bit more with multi-state buttons, but that's going to wait until tomorrow when I work on other button issues. And there still seems to be problems with copy/paste with large classes (like Vijilante's ACP class). So there is still lots of stuff left to fix.

It certainly doesn't feel like I'm getting close to the public release. Don't know what I'm going to do.

Wizard Joined: 14 Aug 2004 Posts: 1269

I feel for you Zugg. CMUD just has such an enormous number of (inter-connected) features for an initial public release that nailing the bugs is always going to be hard work. But you're getting there.

Just trying to get my head around this part though:

Posted: Fri Dec 01, 2006 6:32 am

The internal database is the kbmMemTable. The linked-list structure is the class structure used by zMUD. So in order to port some of the zMUD code for compatibility, I have to maintain this linked-list of class records. The class records contain pointers into the kbmMemTable database structure. So there isn't too much overlap in storage...it's really just two different ways of looking at the database. The MemTable database has a field for each record that points to the corresponding class record in the linked-list, and the class record has the pointer back to the database. Keeping these pointers in sync is what I mean when I talk about keeping the database in sync with the linked list.

I have database events that fire when something in the MemTable changes so that I can update the linked list as needed, and I have hooks into the Create and Destroy methods of the class records to create or delete the corresponding MemTable records.

All of this is separate from the actual external SQL database itself. The MemTable has routines for tracking versions of records and can update the external SQL database as desired using a background thread.

That's the essential parts of the architecture. Gives me the advantage of database integrity, while maintaining zMUD compatibility. The cost is keeping the stuff in sync.

Posted: Fri Dec 01, 2006 8:32 am

Wizard Joined: 25 Mar 2003 Posts: 1113 Location: USA

I wrote the ACP script, so you can blame me for that part. ;)

Glad to hear it's improving, and I hope you get the time after the public release to re-organize the trigger states structures, though you may run into compatibility issues then...

Posted: Fri Dec 01, 2006 6:22 pm

Well, what needs to be fixed with trigger states is to combine them into a single class type. Having the main TriggerRecord class, and then having each "state" be a TriggerState class is what causes the kludge. And the fact that the database points to the main TriggerRecord for the first state, but points to TriggerState classes for substates. It's just all horribly inconsistent.

Today I get to fix some more bugs relating to this. Like when you drag a trigger state from one trigger to another (needs to move the TriggerState class from one TriggerRecord list to another), or drag a trigger state to the main level to make it into a normal trigger (needs to create a new TriggerRecord parent), or drag a normal trigger into a substate of another trigger (needs to remove the previous TriggerRecord parent). etc. etc. It's just a mess.

And no, this doesn't have anything to do with the DevExpress toolbar mess...that's a totally different problem. The mess with Trigger states is my own mess that I created last year when I first implemented the new database structure and didn't fully think it out. Anyway, it's something I can fix later. I should be able to clean it up without breaking any compatibility.

Posted: Sat Dec 02, 2006 3:55 am

Bah...another 4 hours today dealing with moving Button with button states to packages. No, button states are not implemented the same way as trigger states. They are kludged differently. Yeah, I know, it's another mess :(

The problem with all of this is that there are so many cases to test: button without states, button with states, button without states within a class folder, button with states within a class folder, button nested multiple levels deep in a class folder, etc, etc. Just because one of these tests works doesn't mean that they will all work. Because of possible recursion, something that is nested within a class folder can fail to move to the new package, whereas something in the main selection works fine. So just when I think I have it working, I seem to find another case that fails. I hate code like that.

Anyway, I think it's working now.

The main setting corruption issue that remains is the Copy/Paste bugs. I've got it fixed for most cases, but it still fails when there are multiple subclasses within a class being copy/pasted. I'll leave that for tomorrow. And hopefully tomorrow I'll still have time to fix the problems with the button user interface. Otherwise I'll have to delay the release of "1.18 RC1" until Monday.

These last few posts contain a lot of whining, so they probably really belong in my Blog, but oh well. Sorry for hijacking your thread V. :)

Posted: Sat Dec 02, 2006 4:03 am

For some reason, the Blog threads haven't been showing up orange for me, so I haven't noticed new posts in there in a while. So it's nice to get updates where I'll notice them.

Regardless, see the "Congratulations" thread for some big-ups if you need heartening again. I honestly can't think of anything that CMUD needs for me to be able to use it at release other than the corruption bugs squashed - I haven't managed to get it to crash in ages, and that's what counts. All the funky script-breaking errors like the multiple localvars thing and the custom function spaces thing are gone too, so everything should work just fine for release :)

Posted: Sat Dec 02, 2006 4:16 am

Thanks, I really appreciated your "Congratulations" thread. But it's hard not to get depressed if you could see my bug list :( Yeah, most of them are probably minor to a lot of people, but there are still more bugs on the list than I'd like. Then again, that's probably always going to be true.

Posted: Sat Dec 02, 2006 4:24 am

You wouldn't be the developer that you are if you could see a bug list of any length and say "this is a satisfactory number of bugs". It's unrealistic to think that you'll ever reach a point where there isn't a single bug, but it's caring about the bugs on the list that gives you the drive to fix them. Can't be a bad thing.

From your blog, Chiara seems pretty understanding when you need to get away from it and when you don't, so hopefully it'll all be for the best.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

Wow. I didn't think it was going to turn out to be that nasty of a bug. I am glad I finally contributed something to the beta testing, I was feeling pretty useless.

If it is any consolation the last 2 days sucked for me as well. I have had a fan in my computer getting far too noisy for a while, and finally decided to do something about it. So Thursday I went out to get a new case fan and cpu fan, since I was too lazy to open the computer up and find out which it was first. I also decided that I would get another graphics card and second monitor since I wanted to be able to spread out the map and such down the line with CMud. The store I went to had no cpu fans, but they had everything else. I had a few other errands to run yesterday so nothing got done with it. This morning I start putting everything together and check the fans and of course it is actually the cpu fan that needed replacement. I decided to check on the forums and get my new monitor running before going out to get a new fan from another store, well the computer wouldn't turn back on. After a few hours I find out that the switch in the case is loose and had slid back too far for the button to hit.

Then I finally get to start playing with the new video card. First windows picked it up wrong, new I installed the drivers from the cd that came with it and they were wrong. In fact they virtually guaranteed a system crash. After about 50 reboots I was able to get newer drivers from the manufacture's website. 50 more reboots and I finally managed to get the adjustments that are supposed to automatically turn off if they won't to turn off and stay off. At this point I have a big problem with slow scrolling in everything. The zMud CTRL-Q test was reading 150 on my old monitor and 200 on the new one. A few more hours trying to tweak things around to get it right and I reached the solution...use really old drivers for the older video card. Both cards are ATI and the newer drivers even though they are listed as right for each of the cards were the problem. The newest drivers were needed for my new card, but for some reason they were horrible with the old one.

Any how, that little rant is all the thread hijacking I will do. I am just glad that bugs as bad as this one got found and fixed before the public release. Great job Zugg!

PS-You can't hijack a thread, it is your forum after all.