Zugg Software :: View topic

Posted: Wed Jul 16, 2008 12:29 am

This was brought about because of this quote:

Wanderer Joined: 19 Nov 2005 Posts: 53

You're lucky. I don't even get to click on "OK". It just exits for me.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

Grr! You should know better then to let a bug like that go! I know what the bug is since I that is the preferred PCRE.dll address space, and checking that specific address is the same code point where there was a crash before. It is a stack overflow problem.

In order to fix that crash I adjusted the dll optimizations to get the lowest memory usage (the right optimisaztion setting cut the memory usage by 50%), and I also worked out a more efficient regex for one of the compatibility reports tests. If you can narrow down the specific script that is causing the crash it would allow Zugg to test which of the compatibility reports regexes is causing the crash. If the offending regex can be improved that would be my aim, if it can't then either Zugg will have to adjust the stack options he is building with (I think currently 4MB main program, but only 1MB for each thread), or I shall have to switch the dll to use a slower stable mode. There might be a 4th option that can be thought of, but until we know more I can't guess at it.

Posted: Wed Jul 16, 2008 2:16 am

Hah... much easier than I expected. Here is the offending xml:

Posted: Wed Jul 16, 2008 2:19 am

Hmmm... maybe I'll just... see if I can truncate it quite a bit... Give me a few. :P

Charneus

Wanderer Joined: 19 Nov 2005 Posts: 53

I don't have that script. I sent my mud file to zugg anyway like he asked.

Posted: Wed Jul 16, 2008 2:31 am

Mine wasn't a straight conversion - it was all manual. Therefore, I didn't have .mud file to send. I use compatibility report to sort of throw things my way that I may have overlooked. Like the #PRI commands I missed earlier.

Charneus

Wizard Joined: 14 Oct 2000 Posts: 1368

When testing this I found that the problem is related to many statements separated by the semi-colon.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

Thank you for the procedure Rorso. I am seeing the crash with my test app and the pattern I wrote for doing %nn stuff.
I am pretty much out of time this morning so I will work on improving the regex as soon as possible.
I can probably adjust some of the quantifiers to make it greedier and more able to lock things in properly.
As it stands now it appears that any really long line can cause the crash. For those that care this is the regex

Posted: Wed Jul 16, 2008 12:19 pm

Hah! I can't believe that was the reason, but sure enough, it was... I thought I had gone through and reformatted all of the aliases/triggers and whatnots. Guess I still missed some. :P Thanks, Rorso! And thanks, Vijilante, for looking into this. :P

Charneus

Posted: Wed Jul 16, 2008 5:52 pm

I found the problem. It works fine in CMUDPro. But not in CMUD. I'm glad Vijilante posted about the PCRE stuff because I would have never found this otherwise.

The problem was with the stack size build option for CMUD. In Delphi 2007, there are different "build configurations". For example, "Release", "Debug", etc. When I'm testing here, I build stuff with the "Debug" configuration to trap memory leaks, etc. The versions built for everyone else is the "Release" version.

Well, it turns out that Delphi 2007 has different stack size options for each different build configuration. In CMUDPro, *both* Release and Debug had a stack size of $00800000. But in regular CMUD, the stack size for Debug was $00800000 but the stack size for Release was just $00100000.

So, I have fixed this for v2.32 so that CMUD will have the proper stack size. When tested on this stack size, I don't get any crashes with the current regular expressions.

Adept Joined: 04 Feb 2005 Posts: 246

Unfortunately, I'm having the same problem in 2.32. I had done a clean uninstall, and install of 2.30 yesterday or so, and just installed 2.32 on top of this.

I specified a .mud file for the package name. It converted it, and I got the access violation when trying to look at the compatibility report.

Opening the session online, and then trying to open the PE on the converted .pkg file hangs CMUD. What was displayed before the hang showed a couple classes repeated l5-10 times, and at the root level (i.e., not children of the .mud package).

If you want the .mud file, I can send it.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

Zhiroc, your problem sounds like a conversion error. I would suggest sending a copy to Zugg.

It is good to hear the stack size is bumped up, that should take care of immediate problems with the compatibility report.
I will probably still run some further tests on the regex to see if I can improve it. I didn't quite like the speed numbers I was
seeing while replicating the crash procedure.

Also it is important to note that last time I checked threads use a 1MB stack size in CMud. Since most user scripts are run in a
thread other then the main thread it is very possible for a user to cause such a crash. I am going to have to spend some more
time seeing if I can plug it once and for all. Hopefully I can find some C standard functions that get me details on the stack size,
and then I can put in some code to return an out of memory that I could reasonably submit to the author of the PCRE.

Posted: Thu Jul 17, 2008 2:14 am

Zhiroc: Definitely send me the *.MUD file that causes the hang to sales@zuggsoft.com.

Vijilante: Remember that CMUD doesn't actually use threads unless it is necessary. The command line uses a thread (so that ESC can be used to abort it), but other scripts, like triggers, only use a thread if the script uses the #WAITxxx or other thread commands.

So for trigger matching, threads are not as common as you might think.

I tried to do some testing of methods to catch the stack overflow that was happening. Unfortunately, once the stack has overflowed, you're pretty much screwed. Can't call anything or do anything. The only way to really fix it would be to create a recursion "counter" in the PCRE source and allow a limit to be specified to control the recursion. Since Delphi actually stores the stack on the heap, I'm not sure sure the traditional C stack size routines are going to return the correct info or not. Maybe since PCRE is written in C it will be ok.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

I think I finished tweaking around on the regex. I probably went a bit over with the possesiveness, but it seems to match all the lines it should and fail all the lines it should. It also is quite happy with a %repeat("a",20000) test in a 1Mb stack Very Happy

.

Posted: Thu Jul 17, 2008 9:36 pm

Thanks Vijilante! I'll get that new regex into the next version and will consider some of the other suggestions.
I think I also need to improve the regular expression that looks for the old [] immediate expansion syntax and handle ~ characters before the brackets.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

Hrm, the [] evaluation might be easier then the <> expansion. I think for the [] all that is really needed is to check for a
tilde before the openining (?<!\176) and then make sure the close is on the same line. That makes the regex this "(?<!\176)[[^\n\r];]*]".
If anyone thinks it needs more detail speak up.

The immediate expansion is much harder because it needs to know a lot more about what is around it. Consider the text
"#IF ((a<b) && (b>c))" we don't want that to match the <> pattern. We also don't want anything in an #MXP command to match, but #SAY, #SHOW, #ECHO, and thier
prompt versions should match when the is no preceding tilde. I am not sure it can be done with a straight regex, but I will think about it.

Posted: Fri Jul 18, 2008 3:07 am

OK, thanks for the regex. I'm actually not as worried about the < > syntax because it wasn't used nearly as much as the [ ] syntax.