I decided it was time to create the master thread for talking about the progress and status of the big 2.0 version of CMUD. I've been making posts in the main forum about some of the stuff that I'm doing, but the main forum isn't really the best place for regular status reports.
The bad news is that Chiara and I caught a cold and have been sick all weekend. Yep...it doesn't happen nearly as often on our low-carb lifestyle, but it apparently still happens. I was in bed most of the weekend, but today I felt good enough to do some work.
So today I continued with the Lua implementation. Last Friday I had added the "zs.var[varname]" syntax, both for getting and setting a CMUD variable from within Lua.
Today I added the "zs.cmd.comname(args)", "zs.func.funcname(args)", "zs.CommandOrFunctionName(args)", and "zs.getalias('aliasname')" functions. In the case of the zs.CommandOrFunctionName, I changed my design and decided to search for commands first, and then only look for the function if the command wasn't found. In the documentation I had it reversed, but I found that there were way too many commands and functions with the same name, such as #ALIAS/%alias, #EXEC/%exec where I really wanted the command to be executed and not the function.
In other words, I really wanted "zs.alias('name','value')" to call the #ALIAS command, and not the %alias function.
This is all working well. The only commands/functions that it can't execute yet are the ones that are buried in the zScript byte-code language execution module, which are commands like "#IF", "#WHILE", loops, etc. There is no reason to execute those from Lua since Lua has it's own commands.
But I'll need to add some exceptions for this. For example, I want Lua to be able to use the CMUD #SWITCH command. Also, the new thread commands (like #WAITFOR) are in the byte-code module (they need some low level access to the stack). And I'd like to execute those from Lua too. So that's still on my to-do list.
I wrote the basic zs.getalias("name") function and it returns an alias "object" (userdata in Lua). For example, if you do this:
Code:
zs.alias("test","#show hello world")
a = zs.getalias("test")
print(a)
it displays "Object(alias:test)". But then you can do stuff like:
Code:
print(a.name)
a.value = "#show this is a new script"
zs.execalias("test")
So far I've only implemented the "name" and "value" properties of objects like this.
The main feature I wanted to implement today was the double-pointer system for keeping track of which objects are still valid. In the above example, imagine what might happen if the "test" alias gets deleted. Then what does "a" point to? What if you do "print(a.name)" in Lua when the alias has been deleted in zScript (like in a background trigger or something)?
The way I've implemented Userdata in Lua is that Lua points to an intermediate record, which then points to the actual CMUD data record. CMUD keeps a list of these records, and when you delete a CMUD object (like an alias), it goes through the reference list and clears the object. So when Lua tries to access the Userdata, it sees that the object has been cleared, so it can just return nil.
In the case of zsMUDWindow objects, when the underlying CMUD window object is nil, it returns the current window with keyboard focus. So, if you do something like this:
Code:
win = windows["tells"]
(now close the Tells window)
win.echo("hello world")
then it will actually echo "hello world" to the main window, since the Tells window has been closed. I'm still debating whether this should be the correct default, or whether it should just not do anything at all.
Anyway, it's all still working well. My brain is tired, so I'm stopping for today. Still lots more work to be done.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Wed Jul 18, 2007 2:50 am
I was up all last night coughing my brains out. Didn't get much sleep at all (neither did Chiara ;). But, I still got a full day of programming done today. Today I implemented the full list of properties and methods for all of the various UserData types: alias, var, path, function, trigger, event, menu, statusbar, button, module, class. Lua is set up nicely to allow me to directly interface with Delphi's own GetProperty and SetProperty routines. So it was pretty easy to implement. I'll eventually document all of the properties of these objects.
I also added all of the "get" routines: zs.getalias, zs.getvar, zs.getpath, ...etc. These all take the name (or ID in the case of triggers and buttons) and return the object. Instead of the name, you can also specify the "index" value. The "index" is the global counter used to store all of the objects in the internal cache. Various routines such as zs.numalias, zs.numvar, zs.numtrigger, ...etc return the number of objects in the global cache. So, you can iterate through all aliases on the system using code like this:
Code:
for i=1,zs.numalias do
print(zs.getalias(i).name)
end
There is a global list for each object type (alias, trigger, var, etc). *All* objects are stored in this list, regardless of package or module. One of the properties of all objects is "module", which returns the module UserData record for that object. I have implemented the "eq" operator for these UserData types, so you can do something like this:
Code:
if (zs.getalias(i).module == zs.curmodule) then...
In the above example, it is comparing the module of the ith alias with the current module of the current window. There is also a "class" property, and a zs.curclass function that returns the current class of the current window.
Remember that windows are just a type of module. So you can actually do something like this too:
Code:
if (zs.getalias(i).module == windows["Tells"]) then...
to see if a particular alias is stored within the specified window. Lots and lots of power with this. In a way it's similar to the COM API for CMUD, but this new API actually exposes a lot more properties of a lot more object types.
I also got the zs.param(i), zs.params(i), and zs.numparam functions working. These were a bit tricky because the parameters are stored witin the zScript runtime stack, and retrieving this stack data from within a Lua script isn't easy. But I've got it working now. I think I have even figured out how to do named arguments. The idea will be that instead of $argname, you will use the syntax _argname within Lua to represent a local named argument.
I was also trying to allow _1 to represent zs.param(1), and _2 to represent zs.param(2), etc. Unfortunately, I ran into my first big problem with Lua: there is not way to tell Lua that _1 is the same as the _1() function call. Function calls in Lua *must* have the parenthesis. I really don't like this, and I haven't found any way to trick it yet. I can put it within a table and use a metatable to handle the "index" call, but then you end up with syntax like "_p.1" where "_p" is the global table name. I'm using the "_" character in front because I don't want it to conflict with any other local variables that you might be using in your script.
I was really hoping I could get _1 to work. I'll have to think about it overnight. I might have a kludge to make it work...it involved pre-pending a string with the script text "local _1 = value;local _2 = value..." to the beginning of your script text (like the script of your alias). This would trick Lua into defining the _1.._99 variables as local variables within your script block so that you can reference them. But this just adds overhead that I'd like to avoid, since Lua would need to compile this text as part of your script, and I'd have to deal with nested quote values and stuff like that. I'd like to do it via the API, but I haven't yet figured out how to create local (named) variables via the C API.
Anyway, that's what I worked on today. Got a lot done, and it's getting really easy to write scripts in Lua now. Still lots more to do. Tomorrow I will be creating the makealias, makevar... functions, and I will start working on converting to/from tables and zScript stringlist/database variables...and that's going to be really cool.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Wed Jul 18, 2007 3:19 pm
Hmm, another annoyance...you can't even do: _p.1 as a shortcut to _p[1]. Some of these syntax restrictions are getting rather annoying.
So, we've got the basic zs.param(n) and zs.params(n) functions. I've also been able to add the global _p table, where you can index by number, so _p[1] is the same as %1 in CMUD. Or, you can access it via local argument name, such as _p["argname"] or simply _p.argname. I guess that's as simple as I can make it and still stick to the Lua syntax rules.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Thu Jul 19, 2007 3:43 am
I finished all of the "make" functions. They are really nice. For example, you can do stuff like this:
Code:
t = zs.maketrigger{"pattern", "script", id="id", priority=10}
If you include "unnamed" tables items, the first item is the name/pattern/caption, the second item is the script, and the 3rd item (not used in the above example) is the class name. You can include any other named arguments that you wish. In this case, we are setting the ID and the Priority of the created trigger. A really nice and useful syntax.
I also implemented the conversion between Lua tables and CMUD stringlist/database variables. This is now handled transparently. A string list is created from a normal table with purely numeric keys. If you use non-numeric keys, a database variable is created. Nested tables is supported using the item|(sublist)|item syntax in CMUD.
Here is an example...in Lua you can do this:
Code:
zs.var.mylist = {"item1", "item2", "item3"}
and this will actually store the string list "item1|item2|item3" into the CMUD variable @mylist. But when you print this to the screen, it converts back to a table and displays the original Lua list. To get a database variable, do something like this:
This will store a database variable in the CMUD @mydb variable. You can also do stuff like: print(zs.var.mydb["class"]). However, I haven't figured out how to *set* a value yet. If you try to do something like zs.var.mydb["priority"] = 20 then it doesn't work. Even when I set up a metatable on-the-fly, it seems that even when you just set a single field, the __newindex metamethod is called for each of the fields and has the old value instead of the new value. Not sure what's going wrong here, but I'm still looking into it.
In any case, this transparent conversion of tables between Lua and CMUD is very powerful and should make it really fun to manipulate your string lists as if they were Lua tables. In the future I might even use Lua internally to store string lists and database variables within CMUD, so this is a step in that direction.
Finally, I also implemented the "pre-defined system variables". These are things like %secs in CMUD. You can use zs.secs from Lua. If the name of the variable conflicts with a command or function name, then you can force it using the syntax zs.sys.secs. So this is similar to how the "zs.cmd" and "zs.func" objects work.
So, the Lua implementation is almost done. I can't believe it's come together so quickly and easily. Tomorrow I'll clean up some loose ends, and do some more testing.
Friday is busy with some household stuff, so on Monday I'll be starting to implement the new human-readable XML format for import/export. I'll also be trying to fix the copy/paste bugs and the package library import bugs that are related to the XML issues. That will probably take a couple of days, then it's on to the trigger tester screen and the ATCP/Telnet option trigger types.
The first week in August I'll be attending the Sony Fan Faire (I'll talk more about this in another blog post tonight). I'm not sure if I'll release the first CMUD 2.0 BETA before or after this trip. It will depend upon how rushed I feel. I don't want to release anything that is a big mess and then go away for a couple of days. But if it looks like everything is coming together smoothly, then I might release it so people have time to play with it while I'm gone. So, it's either going to be around July 30th, or the following week after I get back. Give how ambitious this 2.0 version has been, I'm really happy with how well it's all going.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Thu Jul 19, 2007 6:15 pm
Why, Lua, Why??? Why don't you call some sort of metamethod when a value in a table is changed?? Sigh.
This is the first real problem I have encountered with Lua, and I cannot find any workaround. When you assign a metatable to a table, you can specify the __index and __newindex metamethods. The __index is called whenever you access a particular key in a table. For example, if you call print(t[x]) then Lua will call the __index metamethod, passing "x" as the desired key. This is a very nice feature.
But the __newindex metamethod is only called for a table where the key doesn't yet exist. So, when you first call t[x]=value, then Lua will call __newindex with "x" as the key, and "value" as the value. But once this is done, if you later change the value of the key, like with t[x]=newvalue, then Lua DOES NOT CALL __newindex!! Why? Why? Why? This is stupid. This prevents us from using __newindex to notify us that a value has changed (like so CMUD can update it's background database!).
This single omission makes it impossible for me to use Lua internally to manipulate string list and database variables. Because of this, I don't have any way to notify CMUD when a table key value has been changed.
The only way around this is to use a UserData type instead of a table, but that defeats the entire purpose of using the Lua table internally. If I had my own UserData type that implemented table hashing, then I'd just already be using it in CMUD in the first place. For UserData, Lua always calls "__newindex", since it has no way to tell if the key already exists in the object.
I'd consider patching the Lua source code myself to fix this. But I don't want CMUD to use a special version of Lua. I want the standard Lua DLL to work with CMUD, so that you can update the DLL later without needing to do anything special.
I guess the honeymoon for Lua is over. It was exciting to see all of the wonderful things that it could do, but like anything else, once you get deep into it you start to see it's flaws.
Anyway, what this means for CMUD is that you cannot use the following syntax:
In other words, even though CMUD string lists and database records are converted into Lua tables, any changes that you try to make directly to those tables will not work. The only way to do this in CMUD would be to copy them into real Lua tables first, and then copy the result back to the CMUD variable, like this:
Yep, it's a pain all right. I'll continue to think about this, but I've already spend waaay too many hours trying to make this work and it's time to move on to something more important now. Otherwise I could spend lots of days trying to tweak all of this.
Joined: 26 Jan 2004 Posts: 5155 Location: United Kingdom
Posted: Thu Jul 19, 2007 7:36 pm
...but that sounds like exactly what __index is for. __index is for non-new but changed keys. It's called in exactly the situation you described above.
EDIT: Almost. I remembered where I'd heard about using __index for this and it wasn't, it was using __newindex, basically by creating a proxy table. The real value of zs.var.mystringlist is just a blank table, so any attempt to set a key will call __newindex. __newindex can then do whatever you like as well as holding the values in a proxy table. __index will obviously retrieve values from the proxy table. But I'm pretty sure doing it that way will bugger up iteration. This is definitely mentioned in PiL though.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Thu Jul 19, 2007 8:36 pm
Fang: That is correct. But if zs.var.mystringlist is a blank table and I'm using __index and __newindex to handle get and set, then I would need to use my own hash table algorithm to get and retrieve values. So like I said, using this method essentially overrides the entire purpose of using Lua tables in the first place. The purpose of converting stringlist and database records into tables was so you could use the fast and efficient Lua table hashing (which CMUD doesn't have normally).
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Tue Jul 24, 2007 11:36 pm
I made more good progress today. I spent most of the day with optimizing the threads and getting Lua working within threads. Previously, whenever CMUD needed to execute a script (or a trigger, etc) it would create a new "code" thread, and then execute the thread. For normal scripts that didn't do any fancy wait commands, the thread would terminate and be deleted. Then the next command would create another thread.
All of this thread creation/destruction was unnecessary and causing CMUD to be slower than it should. So I wrote some code to reuse any thread that has terminated normally. Now the speed is back to what it was before I added threads, since most of the time it's just reusing the same thread over and over again, instead of creating new ones.
Once I fixed a silly stack bug in Lua, I was able to get it to work within my threads also. The next tricky step was adding support in Lua for the new wait* commands. Some commands in CMUD (like wait, waitfor, etc) are executed directly within the byte-code execution module. I had already used a trick to access this module from Lua to support the param() and params() functions. To execute CMUD commands like "#WAIT", I made it so that Lua actually creates a short CMUD byte-stream in memory, and then passes it to the normal CMUD execution module. This allows me to execute anything within the byte-code executor that I want from Lua.
Once I added access to these commands, I tested the zs.wait and zs.waitfor commands. They worked great the first time! They work just the same as in CMUD. The Lua thread gets suspended until the wait operation is complete, and then resumes itself. Using the zs.waitfor("pattern") command, I tested executing multiple Lua threads at the same time.
The standard thread warnings apply: be careful accessing global resources from multiple threads. I've tested the basic cases where you want to send multiple commands to the MUD with a zs.wait between them, and I've tested the case where you want to wait for text from the MUD and then send a result (using zs.waitfor) and these simple cases work.
I had an hour or so of programming time left, so I also went ahead and implemented the embedded Regular Expression syntax that was suggested in the forums. I am using the syntax %/regex/% for example:
Basically, anything between the %/ and /% is passed directly to the PCRE engine when the trigger pattern is converted to a regular expression internally. This seems to be working great. The only minor issue is the crude syntax highlighting in the Pattern field. In the above example, the "%n" at the beginning of "%normal" is flagged as a "%n" CMUD trigger wildcard. The /% is highlighted in silver (as a delimiter), but the "n" after the "%" gets highlighted as a wildcard character. Nothing I can do about this without writing a smarter syntax highlighter for my memo control, and I'm not going to do that any time soon.
Tomorrow I will start on the new human-readable XML format. I expect that will take a couple of days.
At this rate, it doesn't look like I will have 2.0 released before I go to the Sony Fan Faire next week. I still want to add the trigger tester module, and then I need about a week of solid bug testing and fixing. So it's looking more like the 2nd week in August at this point. I know everyone is anxious to start playing with this new version, but even though it's a BETA I still want it to be useful when I release it.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Thu Jul 26, 2007 11:26 pm
The new XML format is coming along, although it's taking longer than I wished. As usual, the Package Editor is causing me problems. I added a new tab to each setting called "XML" (it's down where the Compiled Code tab is). When you switch to the XML tab, the new readable XML for the setting is shown. You can edit this XML text and click Save (or switch tabs) and it will update the changes.
Turned out that updating any part of the setting from the XML tab (including the name), could cause problems with the TreeView in the settings editor. That TreeView continues to be a big headache. The routines for handing the XML data do stuff like this:
Name := GetXMLAttribute( 'name');
where "Name" is the property of one of the settings objects (such as an alias). The "setter" for the Name property looks something like this:
Each property has similar code (although the real code is more efficient than this and uses some other helper routines to hide the database and cache details). But the main point is that when updating a *PROPERTY* of a setting record, it does an "Edit" and "Post" for the database.
Well, this causes a problem when clicking the Save Changes button in the settings editor. The settings editor is already a database-aware editor. When you change the value of a field, that puts the DataSet into Edit mode. The Save Changes button does a DataSet.Post to save the changes. Each type of setting has it's own editor page, and it's own SavePage routine that is called by the DataSet.BeforePost event. This allows each editor page to assign the various edit boxes to the proper database fields. For example, in the Alias editor, the SavePage routine looks something like this:
Again, the real code doesn't really look like this, but it gives you the basic idea. The values from the text boxes (checkboxes, etc) are assigned to the appropriate database fields.
So, clicking the Save Changes button calls SavePage to set the database field values, and then calls Post. What about the Properties of the internal Alias Object? There is an event that is called (AfterPost) that updates the internal cache for the objects. For example:
fName := DataSet.Fields[NameField].Value
updates the internal cache for the Name property.
OK, so here is the problem: When we call SavePage while in XML edit mode, the XML routines change the value of the property (like the Name property). As mentioned above, changing this object property sets the internal cache, then calls DataSet.Edit and DataSet.Post. But SavePage is already being called from within the "BeforePost" event for the dataset. And while you are in the BeforePost event, you cannot call DataSet.Post again...this results in an infinite loop.
So, the SavePage routine cannot change the values of the object properties. It can only modify the database fields themselves. Originally, the settings editor didn't even know about the internal objects...it was purely a database editor.
There are two solutions: 1) Rewrite the XML routines so that they set the database variables instead of the object properties, or 2) Perform some sort of trick so that you don't set the object properties within the BeforePost routine.
Solution (1) isn't very nice because I'm using the object-oriented nature of the settings to perform the XML conversion. For example, all objects inherit the common properties, such as "Enabled", or "Notes", or "Name". But an Alias adds the property "AutoAppend", for example. Using my object-oriented settings objects makes it much easier to share code for importing and exporting XML in human-readable format. That's the whole purpose of the new XML format. The old XML format used the database itself to generate the XML, and the database had no knowledge of the underlying objects. So that's why the old XML wasn't very readable...it was just a dump of database record fields. So if I'm going to make the XML human readable, I *must* generate the XML from within the objects themselves. And that means using the object properties and not the database fields.
I decided to go with solution (2). It's a bit of a kludge, but it's a pretty common occurence in Windows. The solution was to use a Windows Message. In the SavePage routine, if I detect that we are in the XML page, I post a new message to the message queue for the current window. This message tells the settings editor to call a variation of the SavePage routine. This routine is called PostSavePage. Because this routine is called from the message queue, we are no longer in the middle of a BeforePost event. At the point the PostSavePage is called, the Save Changes button has already posted the normal changes to the database, and the database is no longer in Edit mode. So, in PostSavePage, it's perfectly ok to set the object properties, which themselves call the Edit/Post routines.
Once I figured this out, then it started working pretty well. I've got the new XML working for the Alias records. I can view the XML, edit it, and save the changes. Tomorrow I will use my Alias test case to add this feature to all of the other settings types. Then I will clean up the Copy/Paste and Import/Export functions to use these new XML routines. I'm hoping I can get that all done tomorrow.
Joined: 26 Jan 2004 Posts: 5155 Location: United Kingdom
Posted: Fri Jul 27, 2007 1:28 am
And to those thinking "zomg Zugg takes such a long time to update ", I present the list of features in the new version and the date they were published. I hope you're sleeping, Zugg.
In theory, you could use the verbose database XML format and run it through an XSLT to produce a cleaner, reduced XML format. I used to have a lot of fun with XSLT when I did a lot of work with XML. The XML Cooktop editor (Windows freeware) made my life so much easier, too.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Sat Jul 28, 2007 5:17 pm
Larkin: Yeah, that's essentially what I am doing internally. But it still needs some knowledge of the different setting types. For example, in the database there is a generic "Opt" field that is an integer field of bits. Each bit is an option. Different settings assign different features to these bits. For example, an alias uses bit 1 as the AutoAppend bit, whereas a variable uses bit 1 as the "UseDefault" option. Bit 2 is for a different options, etc. For a human readable XML, instead of just having "Options=1", I want to have AutoAppend="true" or UseDef="true". Each option bit gets its own XML attribute.
Anyway, it's going better now. I still ran into another tough problem yesterday with my solution (2). When you edit a setting (putting the database into Edit mode), and then click on a different setting in the TreeView, the TreeView control tells the database to save the previous record (calling Post), and then moves to the selected record in the database. With method (2) of putting the PostSavePage into the Windows message queue, the PostSavePage was getting executed *after* the TreeView had moved to the next record. This causes all sorts of trouble.
So now I'm not using the Windows message anymore. I've gone back to the actual database events. The original SavePage was being called by the "BeforePost" event for the database, and now I'm calling "PostSavePage" in the "AfterPost" database event. This works better and solves the original problem, plus it calls PostSavePage before the TreeView moves to the next record.
I swear, I really hate working on the settings editor. One of these days I'll just take a month to rewrite it (again). The old zMUD settings editor was a big mess. The CMUD settings editor started out nice and clean as a pure database editor, but now it's a mess again.
Oh, and Fang: you said "I present the list of features in the new version...". Was there supposed to be a list in your post? Or did you end up going to sleep too ;)
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Sat Jul 28, 2007 10:29 pm
Wow, this is a lot of work. It's a lot more code than I realized to add all of this XML import/export. Each setting type (alias, trigger, macro, etc) needs routines for creating the XML and then importing it again. And even with my helper routines, it's just a lot of typing. If I had named some of my properties better in the first place, then this would be easier. But alas, many of my internal property names are just as bad as the current obscure XML export.
On top of that, each setting editor page has to be modified like I did for Aliases to have an XML tab. And some settings, like Classes, Modules, Windows, Paths, etc, didn't even have any tabbed pages before, so that's a whole set of additional work.
After about 5 hours of typing and coding, I've finally got it compiled again. Looks like there are plenty of bugs and problems, but I'm going to have to wait until Monday to fix it all up. Then I still need to figure out how to properly handle sub-objects, such as trigger states, button states, sub menus, etc. So it looks like it's going to take me several more days to get this XML stuff working. Hopefully I'll get it all cleaned up before I go to Vegas next week.
I normally don't work on Saturdays, but since I'm losing so many programming days next week because of the Sony Faire, I'm really feeling like I'm behind and have too much work to do again. Hopefully 2.0 will be worth all of this work. I just hope I can get it cleaned up so that it's not a disaster.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Mon Jul 30, 2007 11:05 pm
Man, working on the settings editor is *such* a pain.
First, I wasted several hours on what I thought was a syntax highlighter problem. The XML had a <script> tag for the scripts of each setting. For some reason, the text of the script was getting a sqiggle line under it, sort of like what you get from the spellchecking when something is misspelled. I was pulling my hair out trying to figure out what was causing this. I turned off spellchecking and even ended up commenting out my code that handles spellchecking, and the sqiggles were still there.
After several hours I decided to do some Google searching on the Scintilla component. Well, I found out that they are being lazy and are using the HTML lexer for the XML syntax highlighting. And in the HTML lexer, it does some tricks to allow syntax highlighting of JavaScript within <script> tags. This is really annoying because XML should not assign any special significance to the tag values. It should not treat <script> as any special case. And the people who implemented the HTML lexer didn't even think to add a property to turn this off and on. So as far as I can tell, there is no way to turn off the <script> syntax checking for Javascript. I don't know what the squiggle lines are supposed to represent, but it's very annoying.
Since I can't figure out how to turn this off, I have changed my <script> tags into <value> tags. Not quite as meaningful, but now the syntax highlighting of the XML is working properly.
Yes, I know I could probably recompile the Scintilla stuff with a fixed XML lexer. But like with the Lua.dll, I'd like CMUD to work with the standard scilexer.dll file from the Scintilla project. If I get some time, I'll post something to the Scintilla forums to ask an option to be added for <script> handling in a future version.
After fixing that, I had to tackle another nasty bug. The bug where the name of settings gets changed as you quickly click on different setting values is back. Looks like this is caused whenever changing the current database record causes anything to be written to the database when the TreeView doesn't think it should be. In other words, if the DataSet is not in Edit mode, but something else still causes the database to be changed, then the TreeView gets confused and screws up the Name of the item in the tree.
What was happening is that sometimes the syntax highlighter would try to determine if a variable was valid (to determine the color of the variable name in the script). In order to search for a variable, it was looking for *enabled* variables. Checking the Enabled flag for a variable that wasn't yet loaded into the cache was triggering a database read operation to read the enabled flag. The Cache routine had a line like this:
Enabled := GetDBField( RecNum, EnabledField)
to read the Enabled value into the cache. Well, "Enabled" is the actual property name. So this line actually triggers the "setter" method for the Enabled property. And the "setter" method looks something like this:
Code:
procedure Set_Enabled( Value: Boolean);
begin
if Value <> fEnabled then begin
fEnabled := Value;
SetDBField( RecNum, EnableField, Value);
end;
end;
In other words, it updates the "fEnabled" cached value, and then writes the new value to the database. Do you see the problem? The act of fetching the Enabled value from the database was also writing this same value back to the database needlessly. Normally, the proper way to fix this would be to change the original line to:
fEnabled := GetDBField( RecNum, EnabledField)
But this isn't ideal because some properties *need* to call the "setter" method of the property to update some other internal values. The proper way to change this was to actually modify the SetDBField method so that it first calls GetDBField and then only actually writes to the database if the new value is different than the currently stored value. Since GetDBField is fast, this is a good solution and actually has a nice side effect of speeding CMUD up a bit more since it no longer needlessly writes values to the database (which is a relatively slow operation).
This fixed most of the problems in the setting editor with the name being overwritten. But it still happens in some odd cases, and I'm still trying to track that down.
But I've gotten most of the XML problems fixed today. I've also got it creating the XML for nested settings (like classes, multistate triggers and buttons, etc). But when you change the XML, it only updates the XML for the primary setting. I still need to work on this some more so that you can edit one of the child settings and have it get saved properly. Then I still need to fully test the Copy/Paste and Import/Export routines to make sure they are also working. Since they all call the XML import/export, they *should* still work, but I'm not holding my breath on that.
So, it definitely looks like another full day of working on XML stuff. At least I'm making forward progress, even if it's slow.
That whole mess with the <script> highlighting in the Scintilla XML lexer still really bugs me though. I hate wasting my time on stuff like that.
Joined: 26 Jan 2004 Posts: 5155 Location: United Kingdom
Posted: Mon Jul 30, 2007 11:32 pm
The whole treeview/drag-and-drop synergy really seems to be causing problems. The sheer number of posts about it is testament to that, and I'm sure there've been plenty of times when it's caused a problem but you haven't posted. I can't help but wonder why there isn't a component to do this much more painlessly (I'm sure I've seen programs that do this, but windows explorer is the only one I can think of, and that only lets you drag and drop folders), or if there's a better way to handle it without wasting loads of screen space. And I suppose you're limited in what you can do with a vast change (or dare I say it, rewrite) like that because you need to keep compatability. It's a conundrum, that's for sure.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Wed Aug 01, 2007 7:22 pm
Quote:
It's a conundrum, that's for sure.
Yes it is. One of these I'm going to need to take an entire month or so and redo it again. But my worry is that even after spending that kind of time and effort, it might not be any better. The problem is that I need a component that most people don't need: a "partially connected" database control.
First of all, good TreeView components themselves are few. The Windows TreeView is horrible. The best 3rd party tree component is something called VirtualTree, which I've used in some other programs. When I wrote CMUD, my idea was to have everything stored internally within a database (DataSet). Then the Settings Editor could just be a pure database editor. In zMUD, the settings editor edited the object-structure directly. It was object oriented, and each page would edit a different type of setting object (AliasObject, TriggerObject, etc). In CMUD, the object structure is still present for zMUD compatibility, but most of CMUD uses the internal database cache. And it's the internal database cache that is dumped to the external PKG files.
The advantage of just editing the internal database is that A) it's less likely to cause data corruption, because the database is designed for multiple access. So having the settings editor access the database at the same time the background MUD connection is accessing it shouldn't cause a problem, and B) the internal database is the *true* master source of the data, since it is what gets written to disk.
The data-aware components in Delphi (and Developer Express) are quite good. So the original idea in CMUD was to make the Settings Editor completely data-driven. Each control would be a data-aware control and would directly edit the database. This was a nice clean design, and was relatively easy to implement.
The problem with the data-aware Settings Editor is that all data components are designed to always update to reflect the current data in the database. If you change a field in the database, the data-aware control is notified so that it can immediately update itself. So, this led to the infamous speed problem with the new Settings Editor. If the Settings Editor was open, then CMUD slowed down tremendously, because each time you changed the database, the Settings Editor data controls needed to be notified.
So that's when I changed the settings editor to remove all of the data-aware components. This happened back in the Winter sometime. All of the controls in the settings editor were changed to normal edit controls. When you click on a new setting in the tree, the controls in the editor are populated. When you change the value of a control, the Save Changes button is enabled. When you click the Save Changes, the new values are saved to the database.
However, the TreeView was left as a data-aware control. This is because the DevExpress TreeView actually brings a lot of value to the table. It does a lot of stuff that the free VirtualTree component doesn't do. It knows how to load itself from the database and show the correct structure of the settings. It can be easily filtered (to show just aliases, or to just show settings within a specific class, etc). If I switched to a non-data-aware component, like the VirtualTree component, then I'd need to write my own routines to load and update the tree. I'd have to write all of my own routines for sorting, filtering, drag-drop, incremental searching, etc. It's a lot of work.
In the end I think I would end up with the same problem that we have now: what to do when something in the background changes a setting or adds a new setting. Somehow you need to reflect that in the tree view. In zMUD, it wouldn't update the tree with changes...instead, the Refresh button would light up and you'd have to click it to update the changes. There were many problems with that method which could lead to some of the famous settings corruption problems in zMUD that I'm trying to get aware from in CMUD.
My current method is to make the DevExpress TreeView be "disconnected" and ignore updates from the database. A background timer keeps track of whether the tree should be updated and will update the tree as needed. The problems come in when you try to do something in between the updates. At those times, the TreeView doesn't reflect the actual database data, so if you try to save a change or add a record, you can get database problems (again, corrupted data, although most of the time it will just cause an access violation rather than corrupt any data).
So, even though the current system still has problems and is a royal pain to work on, that's been true of every settings editor. It is *not* a trivial challenge. There is a lot more to the user interface of the settings editor than meets the eye, and a lot of work done to improve speed performance. Load custom edit forms for each setting type, handling saving changes if you just click another item in the tree, handling drag/drop including the package tabs, handling the different sorting and filtering. There are a lot of details in the settings editor that some people use, and some people don't. It's quite a complex editor program.
The Tree representation is the best I have come up with. Especially in CMUD where you have multiple trigger states, button states, sub menus, etc that are all represented in the tree. The only alternative is to just have simple folders and settings within folders. But that takes up more space. It's like adding an editor to the Windows file explorer. Imagine that when you select a file in the Windows explorer that you wanted an editor panel to automatically display the contents of the file and allow edits. And when you switch to another file, any changes made to the previous file is saved. Microsoft has been trying to do stuff like adding "Previews", but these are not editable views, and are often "thumbnails" of the file. You end up needing a 3-pane interface (folder, contents of folder, contents of file/message), which makes it more like an email program. But even in my email program I don't like the fact that I can only see a few messages within a folder without scrolling.
With email it's not so bad because usually you are only interested in the last few messages. But with settings, you want to see *all* of the settings within a folder (or as many as possible). I've considered making a 3-column view, like the latest Outlook, for people with widescreen monitors (like me). But you really end up needing as much space for the editor as possible, because it's not pretty when all of your scripts end up line-wrapping. So the editor panel really needs to be as large as possible for people with complex scripts. Most people who ask for a 3-pane view don't have complicated scripts. But anyone who writes large scripts will understand the need for the editor screen space (which is why most options in CMUD can be hidden in the Advanced panel at the bottom so that the editor window can be as large as possible).
So, I've never really found anything better than the consolidated tree view. Makes it easy to move stuff around to different folders, makes it easy to sort and filter, and makes it easy to show the complex relationships between settings and substates.
Anyway, I've put a lot of time into the current settings editor, so I'm not going to trash it without some good reasons. I could easily spend a *lot* of time and effort and just have lots of new problems to deal with. The Settings Editor really is the heart of CMUD. If you look at other MUD clients, they have pretty basic ways to edit settings interactively...usually just a different dialog for each setting type. There is a lot of power in the CMUD editor that just works and you don't have to think about it. All you need to do is spend some time with another MUD client to appreciate all of what the settings editor is doing for you.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Tue Aug 07, 2007 11:00 pm
OK, now, where was I? Oh right...XML. Sigh...
The XML export was working fine. The XML Import gets tricky. When allowing the user to edit the XML (or allowing XML import), you need to be able to decide whether or not you are modifying an existing record, or adding a new record. But I want the XML to be nice and clean and human readable. Normally you would just use the record ID value to identify existing records. This is fine if CMUD creates the export from within the settings editor (because it already knows the record ID of the setting being edited). But it would be nice to support XML Import where the record ID field is not specified.
Then CMUD should be smart enough to look in the current class where the Import is happening and look for an existing alias called "Test". If "Test" is found, then it's script value will be overwritten with "#SHOW Hello World". If "Test" is not found, then a new alias is created in the class.
To *force* CMUD to create a new setting, regardless of whether an existing setting with the same name is found, I've set it up so that you can specify ID="-1" in the XML attributes. For example, normally when you export the above Alias (or click the XML tab in the editor), you will see something like this:
Notice that the XML Export and XML Tab places the existing record ID into the attributes. If you change it from ID="1" to ID="-1" then when you click Save (or Import it), a new alias will be created instead of overwritten the existing record.
So, that handles editing existing records and adding new records. It was a bit trickier than that because some settings use the "Name" attribute, while other settings (Triggers, Buttons, Status bars, etc) use the "IDName" attribute. Triggers are a bit trickier because when there isn't any short IDName set for the trigger, it uses the Pattern tag to locate the existing trigger. But anyway, that part of it seems to work fine now.
The next complexity was to handle importing child states. As I've mentioned in the past, child states are a bit of a mess because of zMUD compatibility. Trigger states, Button states, and sub-menus are not handled the same way within the object cache. And we are using the object cache for the XML import/export now. It was easier when we just dumped the internal database, because child states are just handled with a ParentID field, and the Priority field. When a package is loaded, the records with the same ParentID are added to the Parent record in the cache in the order of their Priority field.
For triggers, the Parent trigger has an internal linked list that points to it's child trigger states. Each child trigger is a separate record in the database, and there is a separate internal Object for each state. But for buttons, there is an internal array of captions, scripts, and colors for each button state. Each button state has it's own record in the database, but there is only a single Object (each child button record in the database points to the same internal parent button Object).
So, it's always a pain to handle button and trigger states. If you change the priority order of a trigger state, the linked list of the master trigger needs to be reordered. If you change the priority order of a button state, the array of captions, values, colors for the master button record needs to be updated.
What I'm trying to achieve is to hide all of this mess from the user. When exporting a multistate trigger to XML, I just want each trigger state to have a STATE number, and allow you to edit that number to reorder the triggers. For example, here is the current XML output for a multistate trigger:
Notice that the XML just shows the trigger states as additional triggers within the master trigger. You can edit the STATE="1" value to change the order of the states. For example, if you change the state of the first trigger to "2" and the second trigger to "1", then click Save Changes, you get this:
Notice that the two trigger states were swapped, just like expected. If you add another trigger state in the XML code (just don't specify the STATE attribute, or give it STATE="3") then it will add a new trigger state.
So, this is finally working as expected and should allow easy editing of the XML code. As I said, the ID attributes are optional. If present, then they help ensure that the correct setting is changed. If missing, then it uses the Name field, or, in the case of trigger states, it uses the STATE attribute to determine what state is being edited.
Notice that only the master trigger has a Priority field. The Priority fields of the sub states don't matter...the Priority field is just used to order the trigger states. But it's easier for people to edit if the trigger states start with STATE="1" rather than showing the raw priority value. You can still edit the raw priority value in the settings editor if you want, but it's not needed in the XML. The Priority just determines the order of the sub triggers.
I've also got this working with Menus and Sub menus. In the case of sub menus, it shows the Priority of each sub menu item, and you can just edit the priority values as needed to rearrange the menu order. If you leave out the priority, then it just creates the sub menus in the order shown in the XML code.
Buttons are a whole different mess, and I don't have it working yet. Since each button state points to the same internal Button Object record, each button state currently exports the same XML as the master button. I need to fix this so that the master button property outputs the XML for all button states properly. That's a job for tomorrow.
Sorry this is taking sooo long. I'd really like to be done with the XML soon...it's no fun. But I've got a lot more testing to do. I haven't even started testing the Copy/Paste or Package Library stuff to see how it handles the new XML format. This stuff isn't any fun to work on, and it's hard and tedious work. If I didn't need to have zMUD compatibility in CMUD, then I would have done things much differently. And someday when I have time maybe I'll clean this stuff up. But I don't know when I'll have that kind of time. Anyway, hopefully it will just be another couple of days for this XML stuff. Then I still want to spend a week cleaning up the high priority bugs on my bug list.
So, as I said in the main CMUD forum...Another week? Hopefully next week.
Joined: 18 Oct 2000 Posts: 2733 Location: Atlanta, USA
Posted: Tue Aug 07, 2007 11:56 pm
I hate to throw monkey wrenches but are you handling if someone has multiple sub-triggers with no states or with gaps between the states or would spit it out.
And what if they edit the XML so several states have the same state number?
Edit: Wouldn't it be simpler to not have the state number in the XML? It isn't needed, as it can be calculated based on the order of the states, and it avoids these issues.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Wed Aug 08, 2007 4:45 pm
Yeah, Seb, I think you are right. I think just removing the state number and letting the order of the XML determine the states would be better. I had originally thought that it would be cool to just edit the state numbers and that actually rearranging the XML would be more complicated. But I think the state numbers make it more complicated than I had thought. That would also make it more like the button states and submenus which also don't have state numbers.
Joined: 25 Sep 2000 Posts: 23379 Location: Colorado, USA
Posted: Fri Aug 10, 2007 1:48 am
OK, I got a lot done today, although most of it was bug fixing instead of XML.
I got the XML working for multistate buttons. I also fixed the multistate triggers to remove the STATE tag as suggested by Seb. That seems to work better now...just rearrange the <TRIGGER> tags in the order you want your states. You can also remove states and add states by just editing the XML text.
With child triggers, they really are full-blown triggers, which is why the child states still use the <TRIGGER> tag as shown in the above example. With multistate buttons, the child states are not full buttons. Child states of buttons only have the caption, value (script) and color properties. So a multistate button looks like this in XML:
Like with trigger states, you can rearrange the text, add a new state, or delete states by just editing the XML text.
While I was working on this, I came across all of the various bugs in multistate buttons that people have reported. So I decided to work on those today too. The bugs that I fix include:
Color of child button state not working if parent button was transparent
Multistate button captions had a lot of extra blank space to the left of the caption, and didn't draw the drop-down arrow in the right location
When assigning a variable to a multistate button, changing the variable value often didn't update the button caption, and never executed the button's command
Same problem as above for Toggle buttons
Deleting the variable assigned to a button (via #UNVAR) would mess up the button, even when the variable was recreated. CMUD now actually recreates the variable itself when needed even if you try to delete the variable.
When loading multistate buttons or multistate triggers, sometimes the order of the child states wasn't correct. It was setting the order of child states based upon the position of the record in the database, rather than using the priority values
Sometimes a multistate trigger would stop showing the current state (the blue arrow) in the settings editor. Actually, this bug was causing the settings editor to stop the background updates when there was a glitch in the drag/drop code (the settings editor would think that you were still in the middle of dragging stuff)
The test button in the button editor wasn't clearing it's menu when switching between a multistate button and a normal button
Probably some others that I have forgotten.
I also ran into a really nasty wierd bug. Now and then, when testing the multistate buttons, the main MUD output window would stop scrolling. Turns out the scrolling text control was getting out of synch, and there was an obscure situation where once it was out of synch, it would stay out of synch.
The cause of it getting out of synch in the first place was caused by the fact that the buttons were being updated when it's variable changed value in a background thread. It's very important to always update user interface controls from the main Windows thread. CMUD handles this most of the time. However, there was one case where setting a variable in a thread was calling the button Invalidate routine directly, instead of telling the main thread to update the button. When the button was invalidated, it would execute it's button command in the background thread, and if this command caused text to appear on the screen, the screen was getting out of synch.
Took several hours to track down that problem. It's possible that there are some other cases where this happens. The way to tell if this happens is that you won't see the output of the command on the screen until you click on the screen or until it scrolls. I have fixed the scrolling text component so that it can detect when it is out of synch and fix itself. So hopefully that will also help with these thread-related problems.
I also wasted an hour with a really stupid and frustrating Delphi issue. I was debugging Buttons and I was watching the "Compiled" property for the button in the Delphi IDE debugger. When the Variable assigned to the button was changed, the code was supposed to set "Compiled := false" to force the button's Toggle expression to re-evaluate itself. I was going crazy because even after single-stepping over this line, the IDE debugger was still showing "Compiled=true"!! How could the value still be true if I just set it to false?? After a lot of cursing and swearing, I discovered that one of the other values I was watching in the debugger was the "Caption" property of the button. Well, the "Caption" property actually calls a "Setter" routine that compiles the button and returns the current caption value. So, since the debugger was evaluating "Caption" first, this was resetting the Compiled property back to True every time. Sometimes these debuggers can get in the way when they evaluate expressions like this that have side effects.
Anyway, a long and tiring day, but I made some good progress. Tomorrow I'll be testing the Copy/Paste, Drag/Drop and Package Library stuff that is related to the XML changes.
You cannot post new entries in this Blog You cannot reply to entries in this Blog You cannot edit your posts in this Blog You cannot delete your posts in this Blog
", I present the list of features in the new version and the date they were published. I hope you're sleeping, Zugg.
I should have thought to look there. I just jump right to the last post in the thread so I hadn't seen your updates at the beginning. Nice job!
