 |
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
 Posted: Tue Oct 10, 2006 5:21 am
CMUD v1.10 Development Blog
|
Time for yet another blog entry.
This week my focus is on writing CMUD help files. Today I added the entries for the new commands and functions and started working on the "Changes from zMUD" section.
Of course, what takes extra time with documentation is when I find that some example doesn't work. Today I discovered several bugs in CMUD, including a bug where the named arguments didn't work. The example:
| Code: |
| #ALIAS make($spell) {#ALIAS $spell($target) {cast $spell $target}} |
wasn't working in v1.09. I've got this fixed now in 1.10.
I also fixed a few issues with the help system. The help system in 1.09 doesn't display some topics in the correct order. The Sort field in the database got declared as CHAR instead of INT so it's not doing a correct integer sort order.
Then I got sidetracked on some performance issues. I was writing the topic that talks about compiled scripts and their advantages. I was trying to put together some examples to see the speed difference and ran into the situation that some people have posted in the forums about CMUD being so slow.
The simple example I was timing was:
CMUD was taking nearly 3 times longer than zMUD to execute this loop. The CMUD compiler was working fine, but the problem was that variable assignment in CMUD was much slower than in zMUD because CMUD needs to update the internal database and indexes. This is overhead that zMUD didn't have.
I worked a bit on this and am happy to say that I'm made some more good speed improvements. The above statement now executes the same speed in CMUD. However, when you start making the loop command more complicated, CMUD starts to beat zMUD easily. For example, I tried this:
| Code: |
| #loop 1000 {a=%i;b=@a+%i;c=@a+@b} |
and CMUD is twice as fast as zMUD for this example in v1.10.
and it you use local variables instead of regular variables, like this:
| Code: |
| $local=0;#loop 1000 {$local=%i} |
then it's impossible to measure the speed difference because CMUD is instantaneous (it takes less than 1 ms). As a comparison, the CMUD and zMUD loop using the @a variable took about 40ms.
So local variables are going to be a *huge* improvement in CMUD. Using local variables whenever possible will greatly speed up scripts since these values don't need to be persisted in the database.
This is nice, but got me distracted from the documentation a bit. So tomorrow I'm back to the documentation. |
|
|
|
|
Seb
Wizard
Joined: 14 Aug 2004
Posts: 1269
|
Posted: Tue Oct 10, 2006 9:48 am Re: CMUD v1.10 Development Blog
|
| Zugg wrote: |
The simple example I was timing was:
CMUD was taking nearly 3 times longer than zMUD to execute this loop.
...
I worked a bit on this and am happy to say that I'm made some more good speed improvements. The above statement now executes the same speed in CMUD. |
... as in zMUD? or as before in CMUD (1.09)? |
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Tue Oct 10, 2006 4:54 pm
|
CMUD 1.10 and zMUD 7.21 execute it at the same speed now. So I sped up CMUD by a factor of 3 for this example over v1.09.
|
|
|
|
|
Seb
Wizard
Joined: 14 Aug 2004
Posts: 1269
|
Posted: Tue Oct 10, 2006 5:43 pm
|
Really nice, Zugg! Impressive, when one considers that "CMUD needs to update the internal database and indexes". How did you acheive this magic? By finding optimizations elsewhere?
|
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Tue Oct 10, 2006 7:22 pm
|
No, it was the database code. I think I've mentioned before that one of the "features" of the in-memory kbmMemTable database component is that it allows multiple datasets to point to the same physical data, allowing multiple cursors, indexes, and filters to be running on the same data. So, for example, the fly-out settings can be filtering to just look at triggers, while the main settings editor is filtered to just look at aliases, while in the background the unfiltered database is being used to process MUD data.
However, when a change is made to a database record, it loops through all of the "attached" tables to update their various buffers. And this is an expensive operation, since it means refetching the current record from the physical list into the individual attached tables active buffers. Since it moves the entire record and there are several "blob" text fields, this involves a lot of memory allocation and deallocation.
I was able to determine whether or not an active buffer really needed to be refreshed from the physical data. This stops about half of the attached datasets from needing to refresh their buffers when a variable changes value. I did a few other improvements in the compiler, but this was the major optimization. And there is actually still a bit more that I can do with this that I'll probably experiment with later tonight.
So far I haven't seen any adverse effects from this optimization, but as we know, messing around with the database module has often caused other trouble in the past. But I've got my fingers crossed. |
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Tue Oct 10, 2006 9:16 pm
|
Ha!! I did it!
When I was investigating the issue with updating these attached tables and indexes, I realized that I had some tables that weren't always needed (for example, the table used for the fly-out settings only needs to be active when the fly-out settings window is open, etc) and I had some indexes left over from some past debugging. I removed all of this extra stuff and optimized when various tables are needed.
To get better timing, I increased the loop count, so now I'm doing this:
last=%secs;#loop 10000 {a=%i};#show (%secs-@last)
and the results on my computer are:
zMUD: 580 ms
CMUD 1.10: 380 ms !
So now CMUD is faster even for this simple parsing case. The time it takes zMUD to look up the @a variable reference each time through the loop is now longer than it takes CMUD to store the value in the database and update all of the indexes. So finally the compiled scripts are showing their speed.
Man, this is fun. Do I *have* to go back to writing documentation?  |
|
|
|
|
slicertool
Magician
Joined: 09 Oct 2003
Posts: 459
Location: USA
|
Posted: Tue Oct 10, 2006 9:28 pm
|
| Zugg wrote: |
Man, this is fun. Do I *have* to go back to writing documentation? |
Yes, because I can't remember all of the threads where changes were discussed to figure out what I need to do  |
|
|
|
|
Arminas
Wizard
Joined: 11 Jul 2002
Posts: 1265
Location: USA
|
Posted: Tue Oct 10, 2006 9:31 pm
|
| Quote: |
Man, this is fun. Do I *have* to go back to writing documentation?
|
Well, you COULD give us 1.10 to play with, but then we would probably just respond with more work for you to do.  |
|
_________________
Arminas, The Invisible horseman
Windows 7 Pro 32 bit
AMD 64 X2 2.51 Dual Core, 2 GB of Ram |
 |
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Tue Oct 10, 2006 11:33 pm
|
Bleh...I knew you guys would say that 
Well, I went back to the documentation and ran into yet-more-bugs (can I trademark that?). Seems that none of the COM stuff works. And actually, it's worse than that. It seems that when a variable contents a variant type (COM reference or array) that it can get overwritten sometimes. In tracking this down it appears that the variant record for a variable is stored on the runtime stack, instead of a copy, and this can cause the original value to get overwritten.
I fixed this problem, but now there is still another problem where the reference count of these variant objects is getting messed up, causing the object to be destroyed even when there is still a reference to it somewhere. I hate debugging this kind of problem. Doing your own reference counting is a huge headache and it's really hard to track down the source of the problem.
On top of that, there is another problem causing a syntax error from the parser when trying to use various COM variable references that I'll need to track down once the reference count problem is finished.
I suppose I could just add these to the bug list and worry about them later. But I've got to fix these issues eventually and these are still rather large issues. So I might as well fix them while I'm writing the documentation so that my examples have actually been tested.
And yeah, you'll still get 1.10 at the end of the week as always. I'm trying to stick with a weekly release schedule so that the beta testing is mostly working with up-to-date versions and not just dealing with bugs that I've already fixed. |
|
|
|
|
Seb
Wizard
Joined: 14 Aug 2004
Posts: 1269
|
Posted: Tue Oct 10, 2006 11:36 pm
|
You really are a wizard, Zugg! Now cast your magic over the help files! [Edit: But, by all means, fix the bugs you find!]
|
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Wed Oct 11, 2006 1:27 am
|
OK, the COM problems and related parser problems are fixed now. Turned out to be just an issue with expanding default COM properties that was causing the reference count problem, so that's why it didn't show up in my test scripts.
However, I *did* find and fix some interesting problems regarding the #IF statement and how it was testing strings for non-nulls. The statement:
a=test
#IF (@a) {true} {false}
was returning "false" because @a wasn't a number. I've fixed this so that it checks strings for being null, and also expanded this to work with COM types. So instead of using
#IF (!%null(@comvar)) {...}
you can now just do this:
#IF (@comvar) {...}
and it @comvar will be true if there is a com object assigned to @comvar.
So, I only got a couple of help topics done today. In case you want to follow along, I'm currently still working in the "Changes for zMUD users" section. Tomorrow I'm going to start the articles in the main section that describe Local Variables, and then Packages and Modules. |
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Wed Oct 11, 2006 4:25 am
|
While I was messing in the parser, I fixed several of the issues with auto-typed variables and with floating point values (the exponential syntax). I continue to be amazed at the number of problems I'm still finding with the parser. Yeah, sure, it's very complicated code. But come on...I've had this parser for a year now. Sigh.
Oh well, I think it's getting better. I just hope I didn't add more new bugs than I fixed. It's been another one of those frustrating days. |
|
|
|
|
Seb
Wizard
Joined: 14 Aug 2004
Posts: 1269
|
Posted: Wed Oct 11, 2006 8:30 am
|
| Quote: |
| It's been another one of those frustrating days. |
Oh, come on! Have you forgotten the success of your performance improvements already? Let me remind you:
 |
|
|
|
|
Rainchild
Wizard
Joined: 10 Oct 2000
Posts: 1551
Location: Australia
|
Posted: Wed Oct 11, 2006 11:35 am
|
Helpfiles schmelpfiles... being a programmer I also feel the pain when it comes to writing documentation, functional specifications, or any non-coding type of tasks... much prefer playing with things like
| Quote: |
and it you use local variables instead of regular variables, like this:
$local=0;#loop 1000 {$local=%i}
then it's impossible to measure the speed difference because CMUD is instantaneous (it takes less than 1 ms). As a comparison, the CMUD and zMUD loop using the @a variable took about 40ms. |
... with C++ I use a high resolution timer which uses large (64 bit) integers and can measure things down to the ... well I think it's hardware dependent, but on my lappy it measures things down to the 0.000279 th of a millisecond, but it's certainly enough to measure stuff like your local variable loop :)
See the QueryPerformanceTimer and QueryPerformanceFrequency functions in the kernel32.dll ... not sure how you import that into delphi, maybe it already has the functions defined in windows.h (well winbase.h really, but you #include "windows.h")...
Just to test, I did a for( int i = 0; i < 100000, i++ ) in 0.236343 milliseconds :) I know that compiled CMUD code isn't going to be quite that quick, but it should be impressive none the less :) |
|
|
|
|
Rorso
Wizard
Joined: 14 Oct 2000
Posts: 1368
|
Posted: Wed Oct 11, 2006 4:16 pm
|
| Rainchild wrote: |
Just to test, I did a for( int i = 0; i < 100000, i++ ) in 0.236343 milliseconds :) I know that compiled CMUD code isn't going to be quite that quick, but it should be impressive none the less :) |
You have to be careful when doing such tests. For example if you look at the code generated by Visual C++ with optimizations turned on it might even remove the loop:
E.g "for (int i = 0; i < 100000; i++) {}" is just removed so code for it isn't even generated. When you look at CMUD's code generator on the other hand code is generated even when the block is empty. |
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Wed Oct 11, 2006 5:10 pm
|
Yeah, you certainly can't compare the CMUD compilers with stuff like C++. For one thing, they are *highly* optimized compilers that generate direct machine code. CMUD is still generating an intermediate byte-code and isn't going to be nearly as fast.
I know about the PerformanceTimer API, but I've just been using the %secs variable in CMUD to do basic testing. Maybe someday I'll add some functions for higher-precision timing, but that's something for another day. |
|
|
|
|
Rainchild
Wizard
Joined: 10 Oct 2000
Posts: 1551
Location: Australia
|
Posted: Thu Oct 12, 2006 2:53 am
|
I was running it in debug mode, so none of the optimizations were on. Regardless, it was just an example of how accurate the PerformanceTimer API is more than a comparison of C++ vs CMUD byte code -- I fully understand and expect CMUD to be slower than C++ but I'm still curious :)
Anyway yeah you got more important thing to do than implement a higher resolution timer - 0.1 ms vs 1ms might be a factor of 10, but from a human perspective they're both instantaneous. |
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Thu Oct 12, 2006 4:00 am
|
Today I did a bunch of organizational work on the help files. I've also got Chiara working on going through the "Introduction to..." topics and editing them for CMUD. I transfered those Intro pages from zMUD so that I could fix some issues with the help system regarding images. In v1.10 I've moved the help files into a "cmud/help" subfolder and have set it up to cache the downloaded images there. This allows CMUD to cache the images. I then modified it so that when a new help topic is downloaded, any cached images references in that topic are removed from disk so that new ones can be downloaded. This will allow me to update the images over time. (in v1.09 the images just get downloaded to your Windows Temp directory and then deleted).
Now that I have a bunch of placeholder topics created, I'll start filling in details tomorrow. Then in Friday I want to fix the issue with how aliases that don't compile are handled and fix some crash bugs in 1.10 so that I can do a release Friday night. I'm gone Sat and Sun so the release will need to be on Friday this week. |
|
|
|
|
Zugg
MASTER
Joined: 25 Sep 2000
Posts: 23379
Location: Colorado, USA
|
Posted: Sat Oct 14, 2006 12:13 am
|
Well, I got v1.10 released today. I didn't do much with the help files today...I was busy just trying to fix some more bugs. Still more bugs to fix, but I think it's getting pretty good.
The main bugs that I didn't get a chance to fix yet are dealing with multistate triggers. I'll try to test multistate triggers and make sure they are working in the next version. But many of the other reported issues are working now. Also, the help system has been fixed and should work better now, including the image caching.
The actual help topics still need a LOT of work and many are still just straight ports from zMUD. I haven't even updated any of the screen shots yet. I hope to spend most all of next week on help files and screen shots while people find more bugs.
I think I'm still on target to announce CMUD to all zMUD users at the end of October. |
|
|
|
|
|
|
|
