Zugg Software :: View topic

Posted: Fri May 19, 2006 4:52 am

bortaS, for crash reports I'm using the madExcept stuff that I mentioned last week. The crash reports need to work with Delphi to track the callback stack and retrieve line numbers from the Delphi *.map file, etc. Delphi handles stack calls a bit differently than C/C++ so a tool that works with Visual Studio, for example, won't work with Delphi. And generic Windows API stuff doesn't really work either. But the madExcept author did a *very* nice job of handling this. He creates a bug report that can be sent via email ot HTTP and I'm sure I can convert it into whatever form the bug tracking software needs, regardless of who I use.

Back to the programming today. My allergies went wild today and it got windy again (I hate wind). And Delphi crashed to the desktop about 6 times today! Thank goodness I installed that plugin to save my work every 2 minutes. It was a lifesaver today.

This stuff happens when I mess with 3rd party components. You see, Delphi compiles components into "packages", which are stored in *.BPL files. These are really just a special sort of DLL file. When Delphi first loads, it loads all of the installed package BPL files, just like loading DLLs. However, when you have the source code to 3rd party components (like I do) and you have the source code in your search path, then when you compile an application, such as CMUD, it will actually compile the most current source code files for the 3rd party components, rather than using the BPL packages.

This causes everything to be linked into a single EXE file and is the mode I like to work with. You can turn on something called "runtime packages" where instead of linking the packages you have to distribute the BPL files with your application. I stay away from this method.

In any case, let's say you modify a 3rd party component file. Now when you compile and run your program, you are running an updated version of the component. And this updated version doesn't match the version that Delphi loaded at startup from the BPL package. Most of the time this is OK, but when you start modifying the interface parts of the components, all hell seems to break loose and Delphi starts crashing a lot.

I have a "master build" script that recompiles *all* of my 3rd party and custom-written components in the proper order. If I quit Delphi and then run this script (which uses the Delphi command line compiler), then when I re-run the Delphi IDE, the loaded BPL packages are now properly in synch with the source code and the crashes go away.

I had a lot of crashes today because I spent the *whole day* messing with the Developer Express menu/toolbar system. The goal was to save and load the toolbar layouts as they have been customized by the end user. I've mentioned it before, and I'll mention it again...the architecture of the ExpressBars components is a mess. I don't know if they had a summer intern write this years ago or what. But it's a mess. It tries to be "elegant" and just ends up being a pain. It works for what it's meant for, but any attempts to extend the functionality or change it cause lots of trouble. Most of the code is all in a single unit with very little effort made to write clean interfaces for proper inheritence. Half the stuff you need to access is in protected methods, and sometimes in private fields.

The save/restore functions that come with ExpressBars are like this too. They can save/load to an INI file, the Registry, or a custom Stream. However, they access local fields so you can't just inherited from these to add other properties to the file if you want. You can't even copy the code and put it in your own routines because of the local field access stuff. So it's a pain.

At this point, however, I've changed and fixed so much stuff in ExpressBars that I think I'm past the critical point of no return. This means that it is very time consuming to apply new updates from DevExpress to ExpressBars. Which is probably ok since they apparently are rewriting ExpressBars from scratch this summer, so a new version is going to be completely different anyway. I'll probably wait at least a year after their first release before I look into changing to their new system.

So, in a way, I now treat ExpressBars as if it's my own code and just modify it directly however I want. So, to "fix" the save/load routines, I just went directly into their source code and changed it to do what I wanted. That worked pretty well. What it wasn't handling correctly were the "custom" toolbar items, like the command line. It was important to fix that. Can't have the command line disappearing when you load a toolbar layout!

But this led me down one of those rabbit holes. You know the holes that I mean...you look down into the dark hole and you know that you shouldn't go down there because it might take you a long time to find your way back out. And yet, something still pulls you down into the hole.

The "hole" in this case was the user-customization of the toolbars. When you click a little drop-down arrow at the right-edge of a toolbar, you can select a Customize option, which opens a window that contains all of the commands defined by the application and allows you to drag/drop stuff on the toolbar. This works almost exactly like it does in Microsoft Office, so people should already be familiar with this.

CMUD defines a toolbar/menu item for every command action. There are lots of these. Only some of them are placed on the toolbar to start with. Also, any custom buttons, status bar items, or menu items that you define in your settings are also available here and can be dragged to a toolbar. These toolbar items get updated just like any normal CMUD button or status bar item. This gives you a *huge* amount of flexibility in designing your own toolbar layouts and interface.

The problem was that this customization screen didn't use the themed controls, so it looked bad. I know, I shouldn't worry about looks. But I climbed down that rabbit hole anyway. And I had to continue and add new items to the customization menu to toggle toolbar items between icon-only and icon-plus-caption, and another menu item to toggle between icon-on-left and icon-on-top. This is where I got stuck in the rabbit hole.

You see, the architecture for ExpressBars defines something called an "Item" and something called a "Link". A toolbar consists of a set of "Links" that point to "Items". Each "Item" contains the code (Action) to execute that toolbar item. I've mentioned this before, but it's important to understand now. So, if there is a command "Item" called "Paste" which pastes text from the clipboard, you can have a "Link" to that Item in the Edit menu, and you can also have another "Link" to the Item on your toolbar. You can have as many Links to the same Item as you want.

Now, they were smart enough to realize that each Link might need to be customized a bit. Each Link can have it's own Caption, for example. When you are in Customization mode, you can right-click on any menu or toolbar item and rename it to something else. This just renames that single Link and not the underlying Item. You can also have a custom "paint style" for a Link. The choices are: "Standard", "CaptionInMenu", "Caption", and "CaptionAndGlyph". Forget about the details for a minute...notice something missing? What about "GlyphOnly"? Sigh...so I had to add that myself.

OK, now let's look at the position of the image and caption. The normal toolbar items in ExpressBars only support the icon on the left. You cannot make the large buttons like the default zMUD toolbar with the image on top of the text. I wanted this as an option. ExpressBars has an "extended" component called a "LargeButton" which has an alignment field for this. However, this is an extension item that is not supported by the custom Link fields. In other words, the ImageAlignment is stored in the *item* and not the *link*. So, if you change a toolbar to have images on top, then *every* link that points to that item, no matter what toolbar it's on, will get changed! So, you are trying to customize your toolbar to have images on top, and suddenly other toolbars also now have the big buttons.

So, for example, if you place the "Paste" item on your main toolbar, and also in your command line toolbar, BOTH links are effected. This is really bad and ugly when it happens to the command line.

Sigh...it's ok, I'm modifying their source code now, right. So I go in and add support for the GlyphLayout to the Links so that each link can have it's own layout. Now, FINALLY, it's working the way I want. At any time you can right-click a toolbar and select the "Show Caption" toggle, or the "Icon on Top" toggle. And each toolbar keeps it's own settings. And these settings can be stored and loaded from a file.

Oh yeah...I also fixed another memory leak and another crash/bug in their code that caused a crash everytime you right-clicked a toolbar item. What a mess.

So now it all works. It's cool....it's pretty...and it does what the user expects. Hopefully this is the end of the work on the toolbar system.

Posted: Sat May 20, 2006 3:20 am

Today I got a lot more important work done...I spent the day figuring out windows. No, not Windows...just windows. If you've been over in the CMUD Beta forum, you probably saw the post about child windows. This is a very important concept in both zMUD and CMUD and I wanted to get it right. After reading the responses and doing some design work, I finally had a plan for what I wanted to do.

In a way, I've reversed the entire assumption that both zMUD and CMUD were based upon. In zMUD, I had a major data structure called the PrefData. This was equal to a *.MUD file. A "window" had it's own PrefData. So, when you open a window in zMUD, it creates the window, then creates the PrefData, then loads the *.MUD file into the PrefData. Even a child window has it's own PrefData. When you use the "#WINDOW Tells" command to open the "Tells" window, the window is created, a PrefData is created, and the Tells.mud file is loaded into it.

In CMUD this has all changed. In CMUD, there is still a PrefData structure for compatibility. In contains the linked-list structure that I've mentioned before, along with the in-memory database of packages. CMUD allows multiple packages to be loaded into the same in-memory database, which is equivalent to loading multiple files into the PrefData structure.

So originally, CMUD worked like zMUD: You create a window, which creates the PrefData and the in-memory database, then you load the list of packages into the database (which populates the PrefData structure).

But over in the CMUD Beta discussion, we agreed to turn this around. Instead of creating a window first and then loading a package into it, a window would now be *part* of a package. In other words, you load the package, and the package contains a new "setting" called a window (just like an alias, trigger, macro, button, etc).

This makes a lot of sense for package development. You can create a "Tells" package that creates the Tells window and the triggers needed to capture text into it.

But man, what a big change this turned out to be. What this really means, is that we just have ONE PrefData structure in memory and ONE in-memory database. I'm ignoring multiple sessions for now...we'll get back to that later. So now, CMUD creates the "master" PrefData structure when it loads. *Then* it creates the window object within the main package. When you use the #WINDOW command, it checks to see if a package with the same name as the window exists, and if so it gets loaded. If that package doesn't already contain a window object, then a window is created within that package. If no package with that name exists, then a new window object is created within the current package.

Settings such as whether the command line is visible, or whether the status bar is visible are now properties of the Window object, rather than being preferences in the PrefData structure, since each window can have different settings for these options.

This has taken me a while to get my head around it all. It's really different than zMUD ever worked. But the more I get into it, the better I like it.

For example, the hostname and port now belong to a *window* object. You can assign different values to different windows, and this is how you end up playing more than one character. In fact, a really clean way to handle this is to create a "package" for each character, and when you want to play a character, just load it's package (using the #LOAD command). This would load all of the package aliases, triggers, etc, and then create a window. This wasn't possible in zMUD.

I am also adding some other properties to the Window object. For example, a Window object now has both a "Prefix" and "Suffix" string value. Whenever you send text to the MUD from a window, the Prefix is added to the beginning of your text and the Suffix is added to the end. For example, you can define a window called "Chat with Zugg" with a Prefix of "tell zugg ". Now, everything you type in that window's command line gets "tell zugg " added to the beginning, and you end up with a private chat channel window. Just assign a trigger like "#TRIGGER {Zugg tells you} {#CAPTURE ZuggTells;#GAG}" and everything that comes back to you would be shown in that window. This provides a good reason for child windows to have their own command lines now, which didn't really do anything useful in zMUD.

OK, so I got a good start on this today. Not all of it is implemented yet, but I've got all of the major architecture changes completed and turned everything around so that packages create windows. The package editor has a new Window type with it's own icon and it's own property panel. The #WINDOW command works. I still need to add the hostname/port and prefix/suffix properties, but that will be easy.

Along the way, I ran into some other wierd problems that I had to fix. For some reason, I couldn't use Ctrl-V to paste into one of the text boxes in the package editor. The text was getting pasted into the command line instead. First I thought this was related to the keyboard capturing that was causing the Tab problem a few days ago. But it turned out to be a bug in the zApp Memo component that I was using. So I got that fixed.

Then, I spent another hour dealing with another background thread problem. Yeah, I know, I was hoping to be done with those. This one was really nasty...if you had the Package Editor open, the program could lock up completely! Turns out that the final bit of cleanup done in the thread to synchonize all of the "attached" datasets that point to the master physical data doesn't seem to be completely threadsafe. I put a Synchronize call around it, so that it gets executed back in the main thread, and that fixed the lockups. Took a while to find it, but it ended up being an easy fix.

I should mention something about having the package/settings editor open...it's a bad idea. As you might recall, the new package editor is completely database driven. It's very modular and doesn't know anything about the old zMUD data structures. It's basically just an editor for the in-memory database.

Now, the problem with database applications is that they like to keep in synch with the data. It's a big no-no for a database application to display data that isn't current. It's sort of like having multiple users accessing the same data: you have the settings editor open where you are editing something, but in the background, you might have a trigger firing that also changes the value of a variable. That's two threads attempting to access the same data.

This works fine. The Database components I'm using are designed for exactly this situation. They handle fancy record locking to allow you to edit one record while other records are changing in the background. The problem is that everytime there is a change to the database, all of the database controls think they need to refresh themselves. This is especially a problem with the TreeView that displays the settings hierarchy.

It all actually works fine. The problem is speed. Having the settings editor open will *significantly* slow down your scripts. I mean a *lot*. The "quick settings" panel (the flyout panel in CMUD) is a bit better because it has fewer controls to update, but the main settings editor is pretty bad.

I am not going to do anything about this for the first beta. Hopefully someday I will find a way to cache these updates somehow so that it's not always updating the controls everytime something changes. This will be hard because it's all in the guts of how Delphi handles database controls. Getting rid of the database controls defeats the purpose of making this a clean database editing program and runs the risk of more settings corruption. So I have to be careful with this.

What I've done in the meantime is made it so that the editor is "disconnected" from the database when it's minimized. And the "quick settings" flyout window is disconnected when it is autohidden. So just minimize the settings editor and then everything runs at full speed again.

One thing I'm considering is having a mode where you can manually disconnect the settings editor from the database, and just have it connect itself as needed when you make a change in the editor. It's something I need to look into. Right now it's actually kind of cool to see it work. When you add a new variable from the command line while the settings editor is visible, you immediately see the new variable appear in the tree view. This is something that zMUD could never do. The zMUD editor was always disconnected, and you had to click that yellow Refresh button to force an update.

So, it was a pretty productive day for a change. I tackled some hard stuff and I'm brain-dead again, but I'm glad I got it mostly done. Just thinking about how child windows needed to work with packages was giving me a headache, and now I can move along and we can tweak how it works during the beta period.

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

<hijack>

It does sound like you had a very successful day, now I just need to have one of those myself. Deadline looming, customer antsy for an update, should make for a stressful week. Oh well, that book is due to arrive toward the end of next week, and the powerball jackpot is steadily building, so there is a glimmer light at the end of the tunnel.

Hope you had yourself a good weekend. I certainly did, had a bonus exp weekend on EQ2 so I collectively gained 19 levels between my four toons... bonus xp 4 teh win!!! The new expansion - Kingdom of Sky - is pretty cool so far, but I'm most looking forward to the one they will be releasing around Christmas - Echo's of Faedwyr - since I've always been big on elves and the expansion is basically the elf/fae homeland, it should be good.

E3 coverage wasn't as exiciting as I was hoping for, I got a little buzz out of Vanguard, the PS3 and Final Fantasy 13, but I have to agree with the other guy sayin' that CMUD is definately keeping me a lot more interested than those other things.

And while I'm off topic, did you see the Final Fantasy - Advent Children move? It just got released over here in Aussie and wow! The graphics were insane. My friends complained about the plot being thin, but I really liked it hehe :)

</hijack>

On the topic of database control refreshing - yeah, I don't think there's much you can do about that, except maybe do the update in a worker thread, or when the application returns to idle. I don't think the panel really needs to be instantly updated when a setting changes, especially if it's volatile like a variable. In fact, it might be enough to wait for the window.gotfocus event and reconnect the data adapter then - and disconnect when the window loses focus? Just a thought :)

Edit: Possibly have a toggle-button on the tool bar of the window which says 'refresh in real time' for when you are debugging scripts, but it is off by default using the gotfocus/lostfocus method of refresh?

Magician Joined: 29 Nov 2005 Posts: 482

personally i never had a problem with the refresh button. Worked fine for me, However in cmud this could pose a completely different issue. How about offering an option to connect to the database "OFFLINE" or watch it in realtime. It would actually be a great debugging tool that way.

And i agree it would seem that the best way is to send a data adapter "save" and close() if the window loses focus or if the settings window becomes minimized and then reconnect when its the active window again.

Posted: Mon May 22, 2006 5:55 pm

The only way to stop the refreshes and updates is to assign the Dataset property to nil...essentially disconnecting the dataset from the window. This stops updates, but then it also stops all of the data controls from working. For example, now you can't enter any data into any of the data fields. And all of the data fields show a blank value because they don't have any dataset to fetch the data from.

So there really isn't any way to set the Delphi controls to "offline". They still need to point to some sort of data source in order to display a value or edit a value. The way you normally implement an "online" vs "offline" mode in a Delphi database application is to create a local database for "offline" mode. So you point the dataset to the live data for Online and point it to a local database for offline. Then you try to resynch the local database changes with the master when you go "online" again.

But in CMUD everything is already local. We are already pointing to the local in-memory settings database. So, to implement a true "offline" mode, I'd have to create another in-memory database that was disconnected from the main database, then let you edit that data and then somehow merge your changes with the main database. That might be something for the future, but not for right now.

So, it only works to disconnect the dataset when you can't view the window (like when it's minimized). If I did this when the window lost focus, then all of the data fields would change to show a blank value, and that would be pretty useless.

This has always been a drawback of Delphi, but I'm not sure any other languages handle it any better. I always wished that the Delphi database controls could just act as normal edit controls when there was no dataset. I added some features like this to the zApp controls, so they work a bit better, but the settings editor uses some other Developer Express controls that don't handle a nil dataset very well.

-----
I didn't get much done on Saturday. Some friends from out-of-town came to visit unexpectedly and we did dinner and a movie (XMen2 in prep for X3 at the end of the month).

But I think it was useful to get a longer break this weekend. I finally figured out the design issue that was giving me a headache regarding the Preference settings and all of this new Window stuff.

Remember when I mentioned that the PrefData structure in zMUD contains your aliases, triggers, etc? Well, it *also* contains your character-specific settings, such as window colors, special characters, etc. So, in zMUD, each Window would load it's own *.MUD file into it's own PrefData structure, and this would contain *both* settings and preferences. Various "global" preferences, like whether to display the main toolbar, were loaded from the ZMUD.INI file.

For example, let's look at the window colors. These are the mappings between the 16 ANSI color codes and the actual color to be displayed in a Window. Obviously we want these color preferences to be associated with a *window*. Each window needs to be able to have it's own colors.

In zMUD, this was easy...each Window had it's own PrefData structure, which contained the color prefrences.

But we have changed that now in CMUD. In CMUD, you load a *package* which then creates a "Window" settings record. You can have multiple "Window" objects within a single package. So, storing the Preferences, like the ANSI color mapping, in the Package no longer works correctly. In fact, since the Preferences were still technically just stored in the PrefData structure, we have an even bigger problem since now in CMUD there is only a *single* PrefData structure for the entire program. The global PrefData structure corresponds to the in-memory database where all of the packages and all of the preferences were stored. We used to have a separate database for each Window, but now we have a single database and multiple window objects within it. So this means we only have *one* global color preferences.

Obviously this isn't what we want. And I was thinking about this over the weekend (yeah, I can't even stop thinking about CMUD on my days off now...its getting annoying). What I need to do is *split* the settings (aliases, triggers, etc) from the Preferences (colors, special characters, etc). The Preferences should not be stored as part of a package...they should be stored as part of the Window object.

Actually, it's even more complicated then that. Some preferences are global, some belong to the package, and some belong to the window. This seems a bit obvious when you look at it fresh or in hind-sight. But I was so caught up in how zMUD was doing it that I couldn't see the obvious solution.

Now, this obvious solution is a pain to code when calling existing zMUD code. As I've said, I'm using some of the zMUD modules, like the mapper and database, until I have a chance to write new ones specifically for CMUD. When these modules need a preference value, such as a color, they call something like MUDWindow.PrefData.Preference. When they want something like an Alias, they call MUDWindow.PrefData.Aliases.whatever. So *everything* is going through the old PrefData object.

But in CMUD, we have something more like PrefData.Package.Window.ColorPreference or PrefData.Package.Alias.whatever. The easy change I made last week to support the new window structure was to just point the MUDWindow.PrefData pointer over to the global PrefData record. So when the old code accessed MUDWindow.PrefData.Preference, they were really getting the global PrefData.Preference value.

To change this all so that the preferences are stored within the MUDWindow instead of PrefData means making a *lot* of code changes. I can't just fool the old code via some refactoring tricks. So this will take some careful editing.

What I think I need to invent is some global "Preference" object to replace PrefData which can encapsulate all of the preferences and then decide internally whether a preference is global or specific to a window. So when the mapper needs a preference value, it just calls the global Preference object (just like it currently calls PrefData) and doesn't need to worry about where the preference is coming from.

Anyway, this is what I'm thinking about today and what I hope to get solved. This should take care of both the Preference storing issue that I was having last week, along with finishing the new child-window structure so that they work properly. I'll let you know how this goes later tonight.

Wizard Joined: 14 Aug 2004 Posts: 1269

I was sort of wondering what was going to happen to colour preferences, and so on, in child windows... Maybe I should mentionned something!

Where are preferences stored (permanently) in CMUD? In the settings database? In binary files? In XML files?

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

For me, I only ever use the one colour scheme for all of my windows on a specific MUD, so storing the colours on each individual window is overkill. Will there be a way to 'use global colour scheme for this window' tickbox type thing?

Having an inline (not sure what they call that in Delphi) 'Get/Set' function for preferences is probably a good way to fly in general, so if you ever need to tweak the way prefs work internally then the rest of the code won't care... makes for much easier maintenance :)

Posted: Tue May 23, 2006 4:11 am

Seb, Preferences are stored as part of the Package. A Package is a *.pkg SQLITE database file. But what I'm dealing with right now is that there are multiple sets of preferences stored in this database. Each window within the package can have it's own preferences.

Rainchild, it doesn't work that way. Each Package database has a table called "Preferences". This is a simple SQL table structure of

ID INTEGER PRIMARY KEY
Value VARCHAR

So, only preferences that are different from the default values are stored. The DEFAULT.PKG package stores a value for each Preference ID. So, if all of your windows use the same colors, then nothing is stored for those windows or packages...they use the default values.

Not sure what you mean by the "inline Get/Set". Delphi has property Set/Get methods that can be used to isolate the rest of the world from internal implementation details. But that wasn't good enough.

The problem is that the ported zMUD code uses the same PrefData class to access *both* settings (Aliases, triggers, etc) *AND* Preferences. So as I mentioned, zMUD code has a lot of references such as PrefData.WindowColor (access the Window Color Preferences) as well as PrefData.Aliases (accesses the Aliases linked-list of Alias records).

Now, splitting the PrefData into two separate classes: PrefData for Preferences, and PkgData for Settings (package data) means that every reference to PrefData.Alias needs to be changed to PkgData.Alias. There is no OOC trick I can use for this.

With the new window system in CMUD, doing something like PkgData.WindowColor doesn't make any sense. This would access a property of the PACKAGE. But the WindowColor is a Preference of the WINDOW. You can have several Windows within a single package. The package record contains the default color for windows, but doesn't contain the specific color of any particular window.

-----
OK, I've spent all day on this. TEN HOURS of changing code and trying to get this all to recompile again. This has been a royal pain. And I'm still having trouble with some of it. CMUD doesn't even run now.

When new packages are read into memory, there is no problem. Preferences for the package get properly set in the package records, and preferences for the windows get properly set in the window records. The problem is reading old *.MUD files.

*.MUD files contain both settings and preferences, but at the time they are read in, the MUD window object doesn't yet exist. So there is no place to put the window preferences. So I guess I need to change the code so that when reading a *.MUD file, the default main window is created first.

As far as how preferences are stored, this was an interesting challenge. Here are the requirements:

1) You want a data structure that is very quick to access by Index (like an array). Each Preference has an ID value from the database. When CMUD needs a particular preference, such as the window color, or the Command Seperator character, it needs to look up the value based upon the numeric ID of the preference.

2) You need a hierarchy of preferences. There needs to be a way to determine if the current window or package overrides the default preference. Each preference can be seperately overridden. For example, a package might override the Variable Character (@) but not the Separator character (;)

3) You need to be able to store the following types: Integer, Boolean, Byte, String in their native formats without having to convert them. Again, access needs to be fast, so no conversion should be performed when reading a preference value from the memory structure.

4) Each preference value should only be stored once in memory. For example, if the default Look Command is "Look", and a window doesn't override this, then it should just point to the existing default and not make a copy of the default value.

What I ended up with is an array of VarRec pointers. TVarRec is a Delphi type used for COM. It's a packed "variant record" which means that it can store different data types on top of each other. A VType property specifies the data type, and a Data field hold a 4-byte value. If the value is a string or something else that is larger than 4 bytes, then a pointer to the data is stored.

So, each window and package has an array of these VarRec pointers, one for each preference ID. If the inherited value should be used, a pointer to the inherited VarRec is stored. The base array (the defaults) has a value for each of the VarRec records.

To look up a preference, we traverse the VarRec pointers back until we get a value. If nothing has overridden the preference, then we get the base default value. We can tell which preferences a particular window or package has overridden by just checking to see if the VarRec is a pointer type rather than a stored datatype (integer, string, boolean, etc).

This seems to work pretty well. It's fast and it's memory efficient. It involves a lot of pointer code, but hey, that's what you have to use for speed and memory efficiency.

-----
One big complication that I ran into is parts of the code that require both sets of data. For example, to compile a script, you need Preferences (such as the current Command Separator character), and you need Settings (such as looking up a token to see if it's an existing alias or variable). So the parser and script execution need pointers to both sets of data. Back when both were stored in the single PrefData record, it was easy. Now it's harder. The Settings (PkgData) are easy to get since there is only a single in-memory database now. But the Preferences are harder to get. You need to know which *WINDOW* is executing a script. Execution of a script requires a Window object because that is where stuff like the Hostname and Port are stored. And it's needed for certain commands like #ECHO or #SHOW.

But look at an individual setting, like an Alias. Settings have a method called "Execute". For example, you can call Alias.Execute to execute the compiled code for the alias. But now we can't just do that. We can't call Execute without a Window object. But what if this Alias is in a package somewhere. This alias might be called from multiple different windows. So getting the execution "context" for a setting like this is hard.

I ended up trying to keep track of the execution context and passing it to the Execute method as an argument. For example, when you Execute the command line, you pass the current window that has the command line to the Execute method. If it needs to execute any aliases, it passes along the same execution context. When a trigger executes, it passes the window that received the MUD input as the execution context.

What a mess. I feel like CMUD is all over the floor in pieces again. Not a good state of things when I was hoping this would be the last week. In hindsight, I probably shouldn't have messed with child windows at this point. I probably should have handled them during the beta. But it's too late now. I don't want to throw away all of the work I did today...I just need to keep at it until it works again.

Posted: Tue May 23, 2006 6:07 am

Cant the code just add another column to the table to add in the preferances for each window in the package?
The default value and variable type of each row should already be defined by the initial column.

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

In theory you could check out a second CVS version of it pre-mess and go to release with that, then when you do a CVS update of the 'mess' version it should grab all the changes you made and merge them fairly well. I think it's called "branching". I haven't used it myself, just read about how they were using it with redhat or mysql.. one of those big ones anyway.

At least you could do as you said and worry about it later without throwing away the work. I imagine there'd be some conflicts you'd have to resolve, but I don't think you would lose the whole day's work.

In any case - if you continue with it now, or at a later date - and bearing in mind I have no idea how you've written the execution code - but could you wrap the Execute function into a TZScriptEngine which has an 'Owner' window?

Please forgive the C++:

Posted: Tue May 23, 2006 6:44 am

shalimar, no. Adding columns to databases at run time isn't easy. It typically causes the entire database to be rebuilt. Not a very efficient operation. Some databases, like SQLITE don't even let you modify the structure at runtime. You have to destroy the database and recreate it.

Also, requirement (1) rules out using a database internally for accessing preferences. It's fine to store stuff on disk in a database, but the access in memory of preferences needs to be fast. Things like the Special Characters are accessed a lot by the parser and can slow it down a lot if it's not efficient.

Rainchild: no, that's bad too. The script engine is expensive to create. It has a huge parse table. So there is only one parser object in memory. It just needs a pointer to the current preferences data so it can retrieve stuff like the Special Characters.

The parser is generated by Yacc. Yacc is not object oriented. When it's converted to Delphi you end up with a TParser object and within that object is the main parse table as a fixed array. So the creation time of the TParser object is fairly significant and not something you want to do a lot.

So, rather than doing OOC, the code is more like ScriptEngine.Execute( MUDWindow, zScript) and the MUDWindow is kept in the runtime stack of the executor so that the executor always knows what it's window context is. This works fine.

I should really go to bed now and tackle this more tomorrow. I found and fixed a few problems tonight. A major problem turned out to be using Strings and the TVarRec array. Yet another case where memory managment gets in the way. Delphi handles the memory management of strings itself. Normally this is good. But they don't expose the low level routines to increment and decrement the reference counts manually. So, when I was adding a string to my array as a pointer, it's reference count wasn't being incremented and as soon as the routine exited, Delphi was garbage collecting the string, leaving a garbage pointer. So anytime CMUD tried to access a String Preference, it was getting garbage and crashing everywhere.

I finally ended up having to use an "array of Variant", and let Delphi handle the memory management of the Variant type (which is handles internally like with strings). I haven't gotten to the point where I can run my test script yet to see what the speed is, but I'm a bit worried because Variant types have a bit of overhead. But at least with this method my string values are not getting lost.

I finally got CMUD to run again for the first time. But it still isn't getting the preferences correctly because it doesn't know that # is the command character anymore, so I can't run any scripts yet. Hopefully this will be easy to fix tomorrow.

While I *could* use the version control system to go back a few days, I'd be giving up everything that I did today and everything I did last Friday and Saturday to implement the child window system. At this point I'm better off just getting this fixed and working rather than going back. I really don't want to release CMUD without child windows working and they weren't working at all before last Friday.

I was just venting because, once again, I didn't expect this to be nearly as complicated as it is. My problem is that I'm always concerned about efficiency. I don't like to implement stuff that I know will slow things down a lot or use up lots more memory than it should. That's what I get for being "oldschool".

Posted: Tue May 23, 2006 7:37 am

Yeah, I'm still awake. I fixed some more problems and got to the point where it runs my test script again. The speed seems to be ok. Still a few problems but I'll deal with them tomorrow. At least the basic architecture seems to be ok at this point. Man, what a long day. Now I need to figure out how in the world I'm going to get any sleep. When I have such a long day I can't get work stuff out of my head. I wish there was a switch on the side of my brain that I could just turn off at night.

Magician Joined: 09 Oct 2003 Posts: 459 Location: USA

Wizard Joined: 14 Aug 2004 Posts: 1269

shalimar, that's what I was thinking (although reading more of Zugg's post - it doesn't look like the database was the problem. Anyway, I imagine the Package database Preferences table would have an SQL table structure of

Posted: Tue May 23, 2006 5:51 pm

Adding them at runtime shouldnt really be an issue should it?
New windows would get added when the user desides they need another one, so its after runtime.
And as for reloading, its just that, its already in the saved version.

The only time I see them needing to be made at runtime is the initial conversion from zMUD to CMUD, but I could be wrong.

Posted: Tue May 23, 2006 6:54 pm

Well, it was a rough night. I actually take a hot shower before bed every night. I also read for at least 30 minutes. I was able to get to sleep, but then woke up after only a couple of hours and had programming stuff going through my head. I don't know if you've ever had one of these kind of "dreams" but it's like you are coding in circles and stuck in an infinite loop. I finally got back to sleep, then woke up late so my schedule is all out of whack today.

Creating an "array of variant" for each package and window to store settings is essentially the same thing as adding a column to a database if you think about that. The index of the array is the ID value of the preference, so that's the row index. Each class of preferences also has a pointer to it's parent. If the Value field is Null, then it fetches the value from it's parent. The top level (first column) has all of the defaults. So it's really the same thing, but without all of the database overhead.

You need to actually code stuff like this I think to really understand. It's easy to just throw stuff into a database. And for complex stuff like aliases, triggers, etc it makes sense. But databases always have an overhead. Even using the low-level direct-access tricks that I use for the main settings, fetching a value from a database is always slower that fetching a value from a simple array. Updating a value is even slower.

Since most windows do not override the default settings, you are often fetching several values for each preference that you retrieve. For example, retrieving the Command Character (#)...first you check the current window. It's null so you check the parent package preferences. It's null so now you check the parent Default settings. Then you finally get the # value. That's 3 retrievals. Special Characters are checked a lot in the parser. In the lexical analyzer, it's comparing almost every character in your script against various special characters (like the Quote character). Slowdown in the special character retrieval can have significant impact on the time it takes to parse a command.

In fact, special character retrieval is *so* crucial, that I added some additional code to cache the values at each level. So now, just retrieving the special characters from the current window will immediately return a value without even doing any array lookups.

For other preferences it doesn't matter as much. But it's always important to look at the details. Even though new preferences are only created when you open a new window or create a new package, they are retrieved all the time.

Wizard Joined: 14 Aug 2004 Posts: 1269

It sounds like you're much too far down this road to change approach now, but wouldn't another possibility have been to clone all the inherited settings to the child window when it is created? Then the child has all the settings and no nulls, and so no lookups would need be done against parents at other times. Obviously this might make child window creation slower. And it looks like you're going to cache the critical stuff, so it's probably not going to be of much performance benefit. Oh, just thought of the downside of this approach - it means that you can't have inherited preferences in the form we are used to - it's a one time inheritance only.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

I think a part of the issue your having is memory utilitization versus speed. I am still going to say that memory is cheap these days. Even cheaper when you start factoring in that the operating systems your targetting, Windows 2K and better, place no memory limits. They just slow down when memory usage exceeds reasonable limits, but keep plugging. Hardware is growing very rapidly, and we are no longer stuck with having to find ways to play in 64K of memory with 24K of it reserved as processor/chip IO locations, screen bit map, screen character map, and kernel. Yeah thats right I did assembly on a C64, and know memory limitations. Now memory might as well be unlimited. Don't be shy about using it when you can gain speed.

A child window is created build a table with the correct values for all its preferences, doing all the look up then. Save only those that overide parent defaults. Update the parent's table of children.

A new package is created again add more tables, just copy the parent's current table, look up once.

A package is assigned to a new parent. Compare tables: any that do not match are flagged as nondefault and saved. Display the change in parent status before saving.

A user makes a change to any of those preferences, update it in the table of the appropiate package, and inform all children that they are no longer default. Save all of, them when the user indicates, but first get rid of the window and preform repaints. The user is busy looking at recovered screen space and never notices that the save is in progress.

As long as user commands are responded to immediately and the screen is updated to look like what they wanted is done, then it seems fast to the user. This can often be done within the message loop of a program, something on the order of check for 1 user input that requires response, when no user items do 1 pending program item. In most cases, with current computers, the user is the slow point.

The only place you run into a problem with this scheme is when the user makes a mistake, and rapidly corrects it. To overcome that you use yet more memory in another table that indicates what the user did and what actions have been taken so far. If 2 actions are pending that cover the same save only do it once. Most likely a little more then you want to get into before tha beta, but it is a rough example of how to squeeze out some speed at only a small memory price.

Posted: Wed May 24, 2006 12:57 am

I think we are beating a dead horse here. I've already got the arrays...they are the same as the "tables" you are talking about. No need for a fancy database here. A traditional link list of arrays works just fine. That hasn't been the issue at all. The issue was all of the code changes needed to support splitting the Preferences into two pieces. I literally spent five hours just editing and letting the compiler find all of the variable references that needed to be changed. That's *five* hours of edit, compile, edit, compile, edit, compile. And Delphi is really good about flagging multiple errors in one pass. I hit the "too many errors to continue" limit in Delphi all the time.

That's how much code had to change to handle splitting the preferences. It was a big deal. And these were not trivial edits. In some cases it was a matter of replacing PrefData with PkgData. But in other cases I had to determine the proper place to get those new preferences from. Right now, the remaining problems involve this last matter of where you get some preferences. For example, think about an alarm. When an alarm goes off, what context should it be executed in? In zMUD this wasn't a problem...settings were assigned to a specific window, so an alarm always had a "parent" window to send it's results to. But with packages and multiple windows, what window do you send the results of an alarm to? The current window? Makes the most sense. What about when the package containing the alarm also has a window defined? Does the alarm belong to that window? No, alarms belong to packages, not windows.

In the past, everything belonged to a window. Now everything is reversed and a window belongs to a package. This makes the execution context much more complicated in some cases.

Finally, the other complexity with the "tables" that people have been talking about, and with the specific SQL structure that Seb mentioned is that the "Value" of a Preference is *not* neccesarily a string (varchar) value. In fact, most preferences are Integer and Boolean. Database columns don't like fields without a known type. That why I'm using an Array of Variants instead of a database table. A Variant can store any value and doesn't have to perform conversions. If you uses a VarChar column for the value, then you'd always be converting from strings to integers/booleans and that would slow everything down again.

I care less about memory efficiency and more about speed. I know that memory is cheap and that people have a lot more memory these days. That's why I'm using things like in-memory databases and stuff like that. Even a *.pkg package is much larger than the old *.mud settings file because it's a database and not just a streamed binary file. CMUD currently runs at about 30MB of memory during execution, which is more than zMUD. This doesn't bother me at all. What I care about is speed. I'll always sacrifice memory for more speed. So don't worry, I'm not being shy about this.

Finally (finally), remember that it's not just about user input. It's about script execution speed. In zMUD I never worried about parsing speed because my thought was always "well, the user is typing it on the command line, so speed doesn't matter". But that was wrong. The same parser is used for scripts, and scripts need to be fast these days. In the old original days of zMUD we were all using dialup, so network speed and the slow scrolling speed of Windows were the limiting factors. These days the network is fast and the video scrolling is faster and the bottleneck is the parser and script execution. That's why I've focused on stuff like the compiler so much.

Wizard Joined: 14 Aug 2004 Posts: 1269

Actually I was just adding a column to the SQL table definition you described a few posts higher up (where you defined the Value column as a varchar). But at the time we were both talking about permanent storage of settings - not the in memory storage. It certainly would not be particularly efficient to put all sorts of data types into a string database column - (but since you only have to read from it when a window is created, it's not a big deal).

I really need to upgrade my RAM - 256 MB is not good on XP Pro when you like to keep a few windows open!

Scrolling / parsing speed has been an issue for years (for some people) - even before home broadband really existed, a lot of people MUD at universities, and they have (and had) fast internet links (but often have slow computers)...

Posted: Wed May 24, 2006 3:50 am

Seb, true enough. Certainly the preferences disk database file has everything stored as strings. It just wasn't a good implementation for the in-memory storage where settings are being retreived and stored more often and needs to be quick.

256MB? Wow, I didn't even know Windows XP was useable with that. Everything here has at least 512, and my development and game systems have 1GB. With the price of memory so low these days, it's the cheapest and most effective upgrade you can buy. Seriously...it's like going from a floppy drive to a hard drive. Going from 256MB to 512MB will make such a *huge* difference in Windows and all application performance that you'll scratch your head and wonder how you ever worked without it.

-----
I think I've got everything put back together and working again. My test script runs fine now and the speed is good. As usual, the compiler just couldn't catch every possible thing that needed to be changed. For example, there were a couple of routines that took a PrefData object as an argument, but had a generic type of TObject listed in the declaration. This is sometimes needed in Delphi to avoid unit loops: you can't have two units that both "use" each other in their Interface sections. Makes some things a pain, but it means that Delphi can be a single-pass compiler, which keeps it lightning fast (my 1.2 million lines of code compile in about 10-15 seconds...pretty nice).

Anyway, so when I was passing PrefDat into a generic TObject routine, the compiler thinks that it's fine. But in some cases, instead of the PrefDat object, I needed to pass the new PkgData object instead. The compiler can't catch this...only testing can catch it. Fortunately, things crash pretty quickly when this happens, so it was easy to find.

I also found an interesting bug. When I was trapping some calls in the debugger to figure out why a certain preference wasn't being set correctly, I watched a button being created. Turns out that TWO button objects were being created on top of each other on the screen. It was hard to tell until you moved the button and saw the other copy stay behind. Wasn't a problem in zMUD...it was a bug added with the new database code. Anyway, I fixed that and things got a bit faster since it was updating half the buttons now.

So, the next step is to actually store these preferences into the package database. Now that the in-memory structure are all set up and working, this will be easy. And now I have a better understanding of how preference inheritence works and where the preferences will actually be stored. Creating the new Window setting object actually made this design easier and more straighforward, even though it was a pain to get to this point.

At least I'm making forward progress. I figure that if I keep getting something done each day, that eventually this thing is bound to be ready!

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

My lappy and 'surf the net' machine have 1 gig, and my games machine has 2 gigs... I don't think I could even cope with 512 anymore, heh. The step from 1 gig to 2 gigs made a huge difference to games, especially when zoning on MMO's or loading new maps/levels in single player games. My games rig is getting a bit old now - one of the first socket 775 P4's - so I'm saving for an AMD based SLI system now hehe.

---

Interesting what you said about the single-pass compiling/etc - I'm so used to things relying on one another it seems strange to have to cast it from a TObject. I guess it's all relative to what you're used to.

The revenge of the phantom buttons - hehe, they must have decided because you fixed the tool bar buttons, they wanted to appear somewhere else :)

Glad to see it's come back together - and making more sense - so hopefully that means a good night's sleep tonight? :)

Posted: Wed May 24, 2006 6:40 am

I know it's over several years... and lots of new code but

Wizard Joined: 14 Aug 2004 Posts: 1269

If one has a WinXP computer over a year old in Europe, there is a fair chance it has 256MB RAM only. The two standard amounts were 256 and 512 MB when I bought my laptop (the average on desktops had probably reached 512 MB by spring last year actually).

I bought a 1 GB DIMM early last autumn (fall), and I sure noticed the speed difference - but the DIMM was faulty and caused intermittent blue screens when I used a certain part of it so I sent it back for a refund. Then stock ran out and when it was in stock again, prices were up 40% on the 1 GB DIMM so I waited before buying again. Prices are almost back down again - I'll probably put in an order today. I'm looking forward to my hard disk stopping thrashing due to reading virtual memory.

Posted: Wed May 24, 2006 6:09 pm

Tech: Yeah, everytime I do a full build and see that number I'm still amazed myself. Yes, it does include some of the third party components, such as Developer Express controls, but I'd say that about half of 1.2 million is my code.

Seb: Getting quality memory is important. I've been burned several times by cheap and faulty memory. Nothing worse that having a flaky computer because of memory issues.

Rainchild: I think I got my first 1GB to play EQ2 (which I know you play). That was the first game that really required it. But it also ended up being a nice improvement to WoW. Not that I play any of these much anymore. For my development system, I got 1GB when I replaced my dead computer last year and it's been a big help. Delphi can really eat up memory when I'm doing lots of debugging and crashing.

For some reason I had another bad night last night. Didn't get much sleep again. And I had another really annoying programming dream. I dreamt that I was trying to figure out this horrible bug and was seeing my code in my sleep. Then in my dream I figured out this great solution to the bug, only to wake up and realize that both the bug and the solution were totally bogus. It made for a stressful beginning of the day. I'm obviously starting to lose it.

Anyway, back to the programming...I'll report back tonight.