Register to post in forums, or Log in to your existing account
 

Post new topic  Reply to topic     Home » Forums » Zugg's Blog Goto page Previous  1, 2, 3, 4  Next
Zugg Posted: Mon Jul 16, 2007 10:49 pm
CMUD v2.0 Status Blog
Zugg
MASTER


Joined: 25 Sep 2000
Posts: 23377
Location: Colorado, USA

PostPosted: Tue Aug 14, 2007 1:56 am   
 
Over the weekend I realized that I didn't finish implementing several planned features. So today, instead of doing the Package Library bug fixes, I went back and finished some other features that were half-done. Today I finished the new Trigger Testing tab. Rather than creating a full Trigger "debugger" at this time, I mainly ported the zMUD Test page and then added some new features to it.

In the new "Test Pattern" tab of the Trigger editor, the "Sample text to test the trigger" is now a drop-down combobox field. When you click the drop-down arrow, the auto-generated sample pattern is shown, along with the last 20 lines from the MUD. When you select a line from the MUD to test, or when you enter your own test pattern, the test pattern will also be saved with that trigger. The next time you look at the Test tab for the trigger, you'll see the previous sample text that you tested.

The Test screen also handles the named trigger argument patterns. So if you have ($local:subpattern) in the pattern, then the matching argument box will show "$local : subpattern" instead of "%1 : subpattern". The syntax highlighter for the trigger pattern also handles the named subpatterns now.

Took a while to get this all working properly. While getting this to work, I found an obscure bug that was preventing some optional arguments from saving. These were options that used the XML "options" field in the database (which is where the test pattern is stored for each trigger). Not sure if this bug was causing any other problems or not.

I also found a problem with the Save flag and the UseDef flag for variables that was getting messed up in some cases. This took a while to fix because of how convoluted the code was. I finally changed the variables so that the UseDef property actually uses the Save flag directly, instead of trying to maintain two options that were used for the same thing. Then I fixed it so that when loading an existing package with UseDef, it will properly convert it to the new Save flag option. The difficulty was that the Save Flag is available for all settings via the database "flag" field, whereas the UseDef was using the "opt" field which is different for each setting type.

I also wasted several hours earlier today messing around with some attempted hacks on our web site. Seems that there is a script that attempts to hack any stores running xcart. They modify their session cookie values to attempt to inject bad SQL into the store, and it looks like they are hoping that I'm dumb enough to be running PHP with "register_globals" enabled. Fortunately it doesn't look like they did any damage, but they did cause a lot of SQL errors to get into the log file. So I had to clean out the logs and then modify the PHP scripts to detect this kind of hack and abort before generating the log file error. I also added a bunch of IP addresses to the global site ban list. I really *hate* wasting my time because of hackers. Not what I wanted to work on today.

The other new feature that isn't completed that I need to work on tomorrow is the Telnet Option triggers (ATCP support). I went through and cleaned up my to-do list and consolidated it with some lists that I had written on paper, so now I have a better and more complete list of what stuff still needs to be done before the release.

But I still feel like I need a full week of pure bug fixing, so now it's looking more like next week again for the release. Yeah, I know, it's always one more week. I still have *way* too many items in the bug list to fix for 2.0, so I'm just going to have to do the best I can to identify the highest priority issues to fix.
Reply with quote
Tech
GURU


Joined: 18 Oct 2000
Posts: 2733
Location: Atlanta, USA

PostPosted: Tue Aug 14, 2007 2:19 am   
 
Noooooooo.... I was looking forward to taking a crack at 2.0 this weekend. While it's disappointing, I'll continue to be patient. I know you're a perfectionist and want to make sure we get a quality product.

I'm looking forward to when it's finally released.
_________________
Asati di tempari!
Reply with quote
Zugg
MASTER


Joined: 25 Sep 2000
Posts: 23377
Location: Colorado, USA

PostPosted: Tue Aug 14, 2007 2:26 am   
 
It's very stressful, because "Yes", I *am* a perfectionist, but it will *still* probably be full of bugs. But I just can't release it with some obviously half-finished features, and I really want to fix at least *some* of the more major problems in 1.34. I really do understand that a lot of people are waiting for 2.0, and sales are certainly reflecting that as well. But I still really need to emphasize that 2.0 will be a BETA version and will have plenty of problems. Too much low-level stuff has changed and it's going to cause trouble no matter how many bugs I try to fix.
Reply with quote
slicertool
Magician


Joined: 09 Oct 2003
Posts: 459
Location: USA

PostPosted: Tue Aug 14, 2007 5:02 am   
 
Sadly, I've been promoted to coder status on my mud and can't really do any real playing until I finish my first project and get promoted so I get a second char.

Besides that, SCA tournament season is gearing up. Twisted Evil
_________________
Ichthus on SWmud: http://www.swmud.org/
Reply with quote
Zugg
MASTER


Joined: 25 Sep 2000
Posts: 23377
Location: Colorado, USA

PostPosted: Wed Aug 15, 2007 12:09 am   
 
Today I got the basic Telnet Trigger type added. This will be the start of ATCP implementation. But in v2.0, I'm just going to have telnet triggers and not any actual ATCP code. In other words, you'll be able to use telnet triggers to implement ATCP yourself. In a future version I'll add more actual ATCP implementation and recognize the various suboption text that is sent.

The way telnet triggers work in v2.0 is that you just create a trigger and in the Type box there is a new option called "Telnet". When this is selected, the parameter value field is displayed to enter the telnet option number that you want to intercept.

When CMUD receives a telnet option request from the server, it will check to see if you have a trigger assigned to this option number. If your trigger is enabled, then CMUD will send the IAC WILL command back to the server (saying that the client supports that option). Any telnet trigger assigned to that option number with a blank pattern will also be fired.

If CMUD receives telnet option sub text via IAC SB Option Text IAC SE from the MUD, then CMUD will fire any trigger assigned to the option number that matches the specific "Text" sent from the MUD in the suboption command.

A new command called #SENDSB has been added to allow you to send suboption text back to the MUD. The format of #SENDSB is:
Code:
#SENDSB OptionNumber {Text}

This will send IAC SB OptionNumber Text IAC SE back to the MUD. If you use #SENDSB in your trigger, any normal response from CMUD to the server will be supressed. For example, you can intercept the option 24 (TERMTYPE) telnet option and use #SENDSB to send your own terminal type, suppressing the normal terminal type that is sent by CMUD. The TERMTYPE option is currently the only telnet option that sends suboption text like this.

When I was done with this, I started looking into some bugs. I was working on the bug report about COM and ADO in the forums. I discovered that I had forgotten to initialize OLE/COM in the background threads, so COM stuff was completely broken. After fixing that, I then uncovered the bugs that were messing up COM in v1.34. There was a case where the execution stack was getting overwritten when a COM variable was expanded, causing weird problems.

Also, the syntax @rs("Field") was not working properly when @rs was a COM variable. It was calling @rs as a user-defined function instead of as a COM reference. So I also fixed that.

Finally, COM variables were not getting expanded properly for some function calls (like %concat). So doing

#SAY %concat("Value: ",@rs("name"))

wasn't working because %concat wasn't expanding the @rs reference. @rs("name") normally returns the full field record, which has a default property of "Value". So it just wasn't converting a COM record on the stack to it's default property when a string was needed.

And then I found *another* bug in a low level routine that was preventing non-integer and non-string values from being properly converted to strings. This routine was old and wasn't used much. I had a newer routine that handled the auto-conversion of any variable to a string value. But this old routine was doing it the old way instead of calling the new routine. So, not only were COM values not getting converted to strings, but even the new string list and record variables were not converted to strings.

Anyway, I got all of that fixed, so it was a pretty good day for bug fixing. I think I've got most of the new features finished now (finally). So now I just need a few days to go through the high-priority bug list and try to fix as much other stuff as I can. Hopefully during this more extensive testing I'll uncover any other problems with the new features (like the threads).

Quote:
Besides that, SCA tournament season is gearing up.

Yeah, I actually haven't played any SCA for several years now. The politics got ugly around here and I was too heavily involved in officer stuff (I was the "Reeve" or accountant for our local group and saw too much annoying stuff). It was taking too much time away from work stuff and the business was suffering because of it.

Anyway, by the time you are done with your first project, CMUD 2.x will probably be nice and stable :)
Reply with quote
Seb
Wizard


Joined: 14 Aug 2004
Posts: 1241

PostPosted: Wed Aug 15, 2007 10:04 pm   
 
Zugg wrote:
In the new "Test Pattern" tab of the Trigger editor, the "Sample text to test the trigger" is now a drop-down combobox field. When you click the drop-down arrow, the auto-generated sample pattern is shown, along with the last 20 lines from the MUD. When you select a line from the MUD to test, or when you enter your own test pattern, the test pattern will also be saved with that trigger. The next time you look at the Test tab for the trigger, you'll see the previous sample text that you tested.

Having a saved pattern will be nice. Also, it would be better if the last 20 _unique_ lines from the MUD were available, as many may be the same, in case of prompts or score lines. Definately looking forward to getting the Trigger Tester back again. Smile
Reply with quote
bortaS
Magician


Joined: 10 Oct 2000
Posts: 320
Location: Springville, UT

PostPosted: Wed Aug 15, 2007 10:16 pm   
 
Seb wrote:
Definately looking forward to getting the Trigger Tester back again. Smile

Me three! Believe it or not, this is the reason why I haven't been using CMUD. I design and program at work all day, and being an old guy now, don't enjoy doing that when I'm trying to relax by playing MUDs. Confused I REALLY despise writing triggers by hand. Razz
_________________
bortaS
~~ Crusty Klingon Programmer ~~
Reply with quote
Seb
Wizard


Joined: 14 Aug 2004
Posts: 1241

PostPosted: Wed Aug 15, 2007 10:17 pm   
 
Is there a way to log all the raw telnet stuff? Also, I doubt I'd use it, but is there a way to send IAC WON'T in a telnet trigger? It looks like you coded it so that it sends IAC WILL automatically in a telnet trigger. I don't know much about telnet options, so I don't know if it would be useful to be able to refuse a telnet option, but I imagine it would be, at least for testing MUD server code.
Reply with quote
Seb
Wizard


Joined: 14 Aug 2004
Posts: 1241

PostPosted: Wed Aug 15, 2007 10:24 pm   
 
bortaS wrote:
Seb wrote:
Definately looking forward to getting the Trigger Tester back again. Smile

Me three! Believe it or not, this is the reason why I haven't been using CMUD. I design and program at work all day, and being an old guy now, don't enjoy doing that when I'm trying to relax by playing MUDs. Confused I REALLY despise writing triggers by hand. Razz

It is one of the reasons I have not been using CMUD much too - I got bored of having to keep zMUD and CMUD open at the same time so I could use the Trigger Tester in zMUD. Then I thought, well if I still need to run zMUD, then why run CMUD... It doesn't sound like it took very long to port over to CMUD, so I think perhaps it should have been done earlier, even if that was only 1.35 to enable non-beta testers to have the feature earlier.
Reply with quote
Zugg
MASTER


Joined: 25 Sep 2000
Posts: 23377
Location: Colorado, USA

PostPosted: Thu Aug 16, 2007 12:00 am   
 
Yes, it's the last 20 unique lines. Actually, whenever anything is added to the dropdown list, it only adds it if it doesn't already exist. So, if a MUD line also matches one of your 5 saved test patterns, then it won't be added again either.

If your trigger assigned to the telnet option is disabled, then CMUD will send an IAC WONT instead of the IAC WILL. So the Enabled property of your telnet trigger controls whether the option is sent to the server, which is how you'd mostly expect it to work.

If you use the #DEBUG command:

#DEBUG test.txt test.raw

then the "test.raw" file will contain a low-level log of all network activity to and from the MUD. You can use that to debug low-level telnet stuff (that's what I use myself).

Quote:
It doesn't sound like it took very long to port over to CMUD, so I think perhaps it should have been done earlier

Maybe, but not really. The code depended upon some other changes that I've only made recently. Basically, all of the work I did to add the XML tab to each setting made it a lot easier to add another tab for the trigger tester.

So, today I started working on bug fixes. I've posted some of them to the forums already. Part of the work was also to fix how MXP triggers handle XHTML tags, and to generally improve how MXP triggers worked (and fix some of the bugs with them). I also worked a lot on the ANSI color problems. Specifically, the problem using ANSI color numbers instead of names, and also some bugs concerning the Style system.

One of the toughest bugs was dealing with using the Default Style, but then sending an ANSI code to just change the background (keeping the foreground color the same as the style). This was hard to handle because the ANSI code all assumes that everything could be fit in the normal word-sized attribute (16 foreground colors, 8 background colors, and various flags). The way Styles are implemented, the upper bit of the attribute is set, and the lower bits point to an indexed style. But when you have a style and then just want to change the background color (or just the foreground color), it didn't work.

What I ended up having to do is create a new local line style (the local line styles are used for MXP and HTML stuff, for example), and set the colors of this local style to the same as the global style being used, and then change the foreground or background color within the local style to match the ANSI color being used.

The only downside is that if you go into your Preferences and change one of the ANSI colors, the text on the screen that has already been displayed on top of the default style won't change because the color was stored in the local line style as if an HTML color command was used. But this is pretty minor. Obviously any new text that is received will display the updated ANSI color.

Also, if you always send *both* the foreground *and* the background ANSI colors, then it knows not to modify the style but to just use the straight ANSI attribute code. And in that case, changing the ANSI colors in Preferences *will* update existing text on the screen.

Just more complexity trying to force a system that was only designed to display a small number of colors into something that can handle any colors. I don't like creating local style records for *every* color change unless I need to...this helps decrease memory usage and increase performance. So when the raw ANSI attribute code can be used, it's better.

Anyway, that seems to be working now. Still a lot of bugs to fix, so I'll report again tomorrow. My guess is that I'll work through the weekend for a release early next week.
Reply with quote
Tech
GURU


Joined: 18 Oct 2000
Posts: 2733
Location: Atlanta, USA

PostPosted: Thu Aug 16, 2007 3:17 am   
 
Keep up the great work Zugg!!! For all the fixes already posted CMUD will be awesome.

I can't believe I'm going to say this, but while we all want CMUD don't push yourself too hard. We need your fresh and alert for the "forum explosion."

Seriously though, go at a healthy steady pace, otherwise Fang or myself will be forced to drop Chiara a line. Twisted Evil
_________________
Asati di tempari!
Reply with quote
Fang Xianfu
GURU


Joined: 26 Jan 2004
Posts: 5155
Location: United Kingdom

PostPosted: Thu Aug 16, 2007 10:07 am   
 
From what she's posted before, I don't think she needs any help, Tech :P
_________________
Rorso's syntax colouriser.

- Happy bunny is happy! (1/25)
Reply with quote
Daagar
Magician


Joined: 25 Oct 2000
Posts: 461
Location: USA

PostPosted: Thu Aug 16, 2007 12:21 pm   
 
All sounds very encouraging!

A question on the ATCP stuff... if we can just create the telnet triggers ourselves, couldn't someone just release a Package with whatever mud's options already triggered that folks could just customize on their own? I guess I'm just unclear on why what you have already in 2.0 isn't "enough".
Reply with quote
Fang Xianfu
GURU


Joined: 26 Jan 2004
Posts: 5155
Location: United Kingdom

PostPosted: Thu Aug 16, 2007 12:26 pm   
 
I'm not sure how Zugg is intending to implement ATCP. He could have an MXP-like system that creates variables automatically based on ATCP values, but as you say, someone could just write a package to do that.

I think it also has something to do with ATCP requiring authentication (see this post).
_________________
Rorso's syntax colouriser.

- Happy bunny is happy! (1/25)
Reply with quote
Fang Xianfu
GURU


Joined: 26 Jan 2004
Posts: 5155
Location: United Kingdom

PostPosted: Thu Aug 16, 2007 6:16 pm   
 
Supplementary question about the Lua implementation in CMUD 2:

I know you've changed print to be something like #say or #echo, but have you changed io.read as well? The way it normally works is that it tries to get input from stdin, the prompt that you enter scripts at in interactive mode. Will this work on the command line in CMUD 2, waiting for the next text to be typed on the command line? If not, is there a way to make it so without using some complex oninput construction? Would oninput triggers even work in Lua mode? I doubt it since nothing is being sent to the MUD.

Additionally, how would CMUD's Lua implementation handle something like print("hello" on the command line? Lua in interactive mode will wait for the bracket to be closed before it runs the code. Will CMUD do that or will it run the code immediately?
_________________
Rorso's syntax colouriser.

- Happy bunny is happy! (1/25)
Reply with quote
Zugg
MASTER


Joined: 25 Sep 2000
Posts: 23377
Location: Colorado, USA

PostPosted: Thu Aug 16, 2007 7:04 pm   
 
Nope, I haven't done anything with io.read. You'll need to use something like zs.prompt to call the #PROMPT command. Accessing the command line directly, or trying to wait until the next command line is typed would be too complicated. OnInput triggers do not work in Lua mode, since the line is being sent to the Lua interpreter and not to the MUD. The Lua command line mode is more for debugging Lua and not for MUD playing.

Right now, if you enter print("hello on the Lua command line, you'll get a syntax error from Lua. CMUD doesn't handle the interactive mode right now.
Reply with quote
Zugg
MASTER


Joined: 25 Sep 2000
Posts: 23377
Location: Colorado, USA

PostPosted: Thu Aug 16, 2007 10:34 pm   
 
OK, I'm about to throw damn Windows through my window. I've wasted my whole day on stupid Windows crap.

Remember when I found the problem of updating stuff like Buttons from the background thread? You can't update the UI in a background thread, so it adds a message to the queue of the main window telling the main window to refresh a setting when it can (like a button).

So what happens if the button gets deleted in the background thread before the update message is processed? The button gets deleted in the background thread, but the main UI still has a message in it's queue to refresh the button. When this message runs, it now references a memory location that is no longer value (the button object has been freed).

I thought about using the PeekMessage API routine to remove the update message from the queue, but STUPID WINDOWS only lets you remove messages in the queue of your own thread! So even though a thread can *Post* a message to another thread (the main UI thread), it cannot *remove* a message from another thread. This is stupid! How am I supposed to remove the update message from the main queue so that it doesn't reference the object that has been removed? I can't "synchronize" the background thread with the main thread every time I want to remove a settings object...that would make things horribly inefficient.

I have no idea why Microsoft has this stupid restriction on PeekMessage. If you can *add* a message to another queue, then there should be a way to *remove* a message from another queue.

This doesn't just apply to buttons, but applies to all settings. And the main problem is with temporary triggers that are created in CMUD to do stuff like the #IF (a =~ b) pattern testing that get deleted after they are needed.

I can't believe I'm wasting my entire day on this crap. But this crap is causing CMUD to crash all over the place at weird times, so I have to figure out how to fix it. But right now, I'm completely clueless. Grrrrrrr.
Reply with quote
Heretic
Newbie


Joined: 27 Oct 2005
Posts: 7

PostPosted: Thu Aug 16, 2007 10:56 pm   
 
I don't especially care for you, but a friend of mine who's more generous pointed out your problem and I can sympathize in a way, so I'll offer my advice.

If you're looking for a solution that's not likely to require major rewrites, you're probably trying to solve the wrong problem. Why worry about removing the update from the queue, when you can worry about deleting the object cleanly instead?

Post a delete message. I believe you can specify custom messages with something like WM_USER+some_offset. Then the situation goes something like this:

1) Post update message
2) Post delete message
3) Queue gets processed, update runs. You may also be able to set some sort of a flag on the object to prevent this update from actually doing anything at the same time as you post the delete message, if that's what you want.
4) Queue gets processed, delete runs, and object is destroyed.

Problem solved.

Don't they tell you in any CS class that you can only delete from the front of a queue? ;-)
Reply with quote
Zugg
MASTER


Joined: 25 Sep 2000
Posts: 23377
Location: Colorado, USA

PostPosted: Thu Aug 16, 2007 10:57 pm   
 
Talk about a kludge. Listen to what I came up with to solve this problem:

OK, so I can *add* messages to the main queue, but not delete them, right? And the problem is that there is a message in the queue that references an object that I'm about to destroy. I can't remove this message. So instead, I add *another* message to the queue. This is a "ObjectDeleted" message. The message itself doesn't do anything at all when it is executed.

Now, in the message handler for the Invalidate message, I look at the message queue for any ObjectDeleted message. I pull out all of these messages and look to see if any match the object that I'm about to Invalidate. If I find a match, then I don't call the Invalidate method of the object (because it's been deleted). Then I put all of the ObjectDeleted messages back into the queue in case there are any other Invalidate messages that still reference it.

The reason for the silly business of pulling all of the messages out of the queue and putting them back in is that the Windows PeekMessage routine for looking for a message in the queue can only look for a message with a specific "message id" value (the ID of "ObjectDeleted" in this case). PeekMessage cannot search based upon the wParam and lParam arguments to the message. So I can't look for a specific ObjectDeleted message where lParam is the object pointer. All I can do is extract *all* of the ObjectDeleted messages, then look through them for one where the Msg.lParam matches the lParam of the Invalidate message that I'm processing. Then I put them all back into the queue.

What a mess. I can't believe I'm being forced to write code like this. All because of the stupid thread restriction of PeekMessage. Geez. At least it works now...after wasting 5 hours on this.
Reply with quote
Zugg
MASTER


Joined: 25 Sep 2000
Posts: 23377
Location: Colorado, USA

PostPosted: Thu Aug 16, 2007 11:02 pm   
 
And Heretic, don't get me started. I've been programming for 30 years. You can't just post a "delete" message when you are removing the object from a background thread. Have you ever even done any thread programming? The main UI thread only runs when the background thread goes to sleep. The background thread needs to remove the object immediately from the internal database...not when the main UI thread gets around to processing it's messages.

And you can delete from any position in a queue that you want. The PeekMessage API does exactly that...it removes a message anywhere in the queue. So don't give me that kind of answer and assume that I'm an idiot.
Reply with quote
Tech
GURU


Joined: 18 Oct 2000
Posts: 2733
Location: Atlanta, USA

PostPosted: Thu Aug 16, 2007 11:14 pm   
 
[Edit] I was late to the party, so you can ignore my comments.

Does the UI thread know if the message was from another thread? Maybe you can just have it check the ones that were added externally to it ( and have possibly been removed. ) If the UI thread can look ahead in it's queue, then when you remove an object you can post a message to the background UI queue. Then it can lookahead in it's queue to see if theres are remove message and handle accordingly. If the API allows you to see the contents of the UI queue then your can make it a little bit more efficient by first checking to see if there's an update on the UI queue before posting the remove message.

Not sure if any of that is feasible since I don't know the API, but it's an idea.
_________________
Asati di tempari!

Last edited by Tech on Thu Aug 16, 2007 11:21 pm; edited 1 time in total
Reply with quote
Heretic
Newbie


Joined: 27 Oct 2005
Posts: 7

PostPosted: Thu Aug 16, 2007 11:16 pm   
 
Yes, I have experience with thread programming, using my own GTK MUD client.

Without knowing your code base I'll admit I could be making some invalid assumptions, but in the general case I can't see any reason why any object needs to be deleted immediately. The worst I can imagine is that some trigger or some such may be able to run in the time between the posting and processing of the message. Even if this is the case, then my third point from before resolves it. Post a delete message and set a flag on the object that marks it as disabled, so it doesn't get used any further. The object will still be deleted -very- soon, and especially as it pertains to the user interface no one would ever know the difference anyway.

My comment about the queue is entirely true. Quoting Wikipedia:

"A queue (pronounced /kuː/) is a particular kind of collection in which the entities in the collection are kept in order and the principal (or only) operations on the collection are the addition of entities to the rear terminal position and removal of entities from the front terminal position."

Try deleting from the middle of an STL queue object in C++. The windows message queue is somewhat more generous than most queues, but as soon as you hear the term "queue" you shouldn't -expect- to have that ability. I'm not assuming you're an idiot, my gripes with you are in another form entirely, which it would do neither of us any good to get into here.

In any case, I concede that without being intimately familiar with your code I can't know all possible cases here. Glad you have a solution, even if you're unhappy with it.
Reply with quote
Zugg
MASTER


Joined: 25 Sep 2000
Posts: 23377
Location: Colorado, USA

PostPosted: Thu Aug 16, 2007 11:47 pm   
 
I only called it a "queue" because that's what Microsoft calls it: the "message queue". I'm fully aware of the definition of queue...you don't need to be smart of quote the Wiki. My point was that Microsoft already provides the PeekMessage API to remove a message anywhere in the queue, and they should have a version for removing a message from a queue that doesn't belong to the current thread. They already provide PostMessageThread to put a message into a different thread, so they should have had an equivalent PeekMessageThread routine. We shouldn't have to play these kind of "message games".

There are several cases where you want an object deleted immediately. You could have the following code:
Code:
#VAR a 123
#UNVAR a
#SHOW @a

That code executes in a thread and if the #UNVAR didn't remove the variable from the internal database immediately, then it would still show a value with the #SHOW command, which isn't correct. Trying to "mark it as disabled" in the database would be worse that my kludge above. A record is either deleted or not, and maintaining consistency between the internal database and the internal record cache is not something you want to start playing games with.

Tech: Nope, the UI has no idea who put the message into the Windows message queue. But yeah, your idea is basically what I ended up with. That's the reason I post stuff like this in the first place. Usually the act of ranting about it gets it clear in my head so I can come up with a solution.
Reply with quote
Zugg
MASTER


Joined: 25 Sep 2000
Posts: 23377
Location: Colorado, USA

PostPosted: Fri Aug 17, 2007 12:53 am   
 
Moving on to other more interesting stuff today:

I finally finished the implementation of the #WAITFOR command. As you might recall from various forum posts, the tricky part of #WAITFOR was figuring out how the script should access any subpatterns in the pattern that it was waiting for.

Normally, %1..%99 and "named" subpatterns are implemented as local stack variables. These are handled at compile-time. But in the case of #WAITFOR, the pattern is not computed at compile time. #WAITFOR takes an arbitrary string argument for the pattern, so it needs to be tested at run-time. Since the named subpatterns are not defined at compile-time, any reference like $subpattern will fail with "unknown local variable".

The solution was pretty easy once I also worked on the #IF (a =~ pattern) expression. It has the same problem with run-time vs compile-time (again, the pattern can be a run-time string). The solution that I used with #IF was to create a function called %pat that returns the nth subpattern. For example:
Code:
a = "say hello"
#IF (@a =~ "say (*)") {#SHOW "Fired:" %pat(1)}

would display "Fired: hello".

I decided to use this same syntax with the #WAITFOR command. In your script after the #WAITFOR you can access any subpatterns using the %pat function. I also improved the %pat function so that it will accept a name for a "named subpattern". So, now you can also do this:
Code:
#WAITFOR "say ($message:*)"
#SHOW "Fired:" %pat(message)

For the %pat function, the $ is optional. If you want to include the $, then you need to be sure to enclose the name in quotes, like %pat("$message"), otherwise the compiler will treat it as a local variable reference.

I also ran into another pretty obscure threading problem with COM stuff. When using COM within a background thread, you must call the Windows CoInitialize at the start of the thread, and CoUnInitialize at the end of the thread. This sets up the COM object marshaling within the thread. CMUD actually uses CoInitializeEx to set the COINIT_MULTITHREADED marshaling method, which allows free access to objects across multiple threads.

The tricky part about this is that when you call CoUnInitialize at the end of the thread, it will hang if there are any outstanding COM references that were created within that particular thread. It's another annoyance of mine that CoUninitialize should *ever* hang. There should be a way to tell the COM system to free all COM references within the thread, or it should at least produce a system exception of some kind. It shouldn't just hang.

Well, given that CoUninitialize *does* hang, there were a couple of solutions. One solution is to loop through the entire internal database to look for any remaining COM references. But there doesn't seem to be a way to determine what thread a particular COM object was created within. I'm sure it must be stored somewhere in the low-level details of the COM object, but I didn't see any obvious way to determine that. And even if there was a way to determine which COM objects belong to the thread that is terminating, it's a big job to loop through the entire database looking for stuff like this. That would make the thread termination routine take a long time, and that's always a really bad idea.

I can't expect CMUD users to always remember to clean up their COM objects either. Having CMUD hang just because you forgot to clear a COM variable would be a pretty bad behavior.

The other solution (and the one that I went with) was to try and ensure that COM objects are always *created* by the main thread. I marked the %comcreate function as non-threadsafe, which tells CMUD to always execute it within the main thread via synchronization. I also had to remember to handle the internal COM objects, such as %cmud, %session, %map, and %curroom. Once I made this change, then CMUD stopped hanging. The threads can still access the properties and methods of the COM objects, and since none of the objects are created by the thread, CoUninitialize does not hang.

Sometimes I regret adding threading to CMUD in the first place, but hopefully it will be worth it once it's fully debugged. Anyway, that's all I got done today. Not nearly as productive as I had hoped.
Reply with quote
Zugg
MASTER


Joined: 25 Sep 2000
Posts: 23377
Location: Colorado, USA

PostPosted: Sat Aug 18, 2007 4:17 am   
 
More bug fixes today, and more thread bugs fixed. Several of the bugs were related to some of the new features. For example, I uncovered some problems with the optimized string list and database record variable hash tables. It wasn't updating the string-value cache in some obscure cases, and it wasn't converting old database variables (like those stored in the session.db file) into the new format.

I also fixed the problem with Prompt Triggers not firing on IAC EOR or IAC GA. So that should be helpful to a lot of people. I also fixed the #MENU command to work with the mapper, and uncovered a problem where closing the map window and then reopening it could prevent any of the mapper functions from working. This might explain those cases where the mapper stops following people or stops working in other ways. Hopefully this will fix it and make the mapper more reliable.

The thread bugs were trickier to track down and fix. When CMUD 2.0 executes a command, it creates a new thread (or uses an existing thread from it's internal pool), and then tells the thread to execute the command. It then waits until the thread finishes, or suspends itself.

This is just what you want normally. For example, if you have a script like this:
Code:
#TRIGGER {tells you (*)} {#VAR message %1}
#SHOW "Zugg tells you Hi"
#SHOW @message

You want the trigger to fire on "Zugg tells you Hi" so that the following line displays "Hi" as the @message value. So, when CMUD spawns a new thread to execute the trigger, it's important to wait for that thread to finish before continuing with the rest of the script.

Well, there were some cases where the WaitForThread routine was returning too early. This routine is very tricky. You never want to deadlock the system, and so you can't just wait for a thread to finish with a simple WaitFor command. Threads typically need to do stuff that updates the UI, and this is done in Delphi via "thread synchronization". Within the thread, you can call "Synchronize(procedure)" and Delphi will cause the named procedure to be executed in the context of the main thread.

The way Delphi implements Synchronization is via Windows Messages, and also via Windows Events. The Events are the better way to handle them. You can use the Windows API MsgWaitForMultipleObjectsEx to wait till an event is triggered, or until the thread terminates (which also signals a Windows event). This routine also allows you to process windows messages, so the user interface doesn't appear to be deadlocked. It can still respond to keyboard and mouse events, for example.

Using the MsgWaitForMultipleObjectsEx API properly can be tricky and is the subject of many articles on the Internet. So it took a while to get it to process the correct messages, but not process the network messages. It's important not to process network messages in this Wait loop, because you don't want a trigger from MUD text to fire while it's in the middle of executing another thread.

Looks like it's working pretty well. I was able to succesfully run a background computation loop, and then at the same time execute a complex script from the command line, and both threads executed in parallel without any problem.

Threading issues are still my biggest worry with v2.0. I still need some more testing time to make sure it's stable. I still have a couple dozen bugs on my bug list that I want to try and fix before the release. But I'm still hoping for something in the middle of next week. It's definitely getting closer!
Reply with quote
Display posts from previous:   
Post new topic   Reply to topic     Home » Forums » Zugg's Blog All times are GMT
Goto page Previous  1, 2, 3, 4  Next
Page 2 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
© 2009 Zugg Software. Hosted on Wolfpaw.net