Zugg Software :: View topic

Wizard Joined: 17 Jun 2006 Posts: 1201

I guess we still need to use the other dll file until 2.24? CMUD version 2.23 still terminates when running the report.

SubAdmin Joined: 18 Nov 2001 Posts: 5187

I will aim at trying to isolate this. Hopefully it is the regex Zugg posted earlier, in which case, I can work at refactoring it until it works.

SubAdmin Joined: 18 Nov 2001 Posts: 5187

After a few hundred crashes I came up with this smaller regex that seems to match everytime it should and doesn't match when it shouldn't. The key thing is it doesn't cause a crash.

SubAdmin Joined: 18 Nov 2001 Posts: 5187

This is just plain frustrating. I compared both sources and came up with a 16 byte difference in the stack allocation for the function that is causing the crash. Comparing the assembly I have $2BC for the 7.0 and $3DC for my build of the 7.6, that is 272 bytes too many.

I also skimmed through about half the assembly for the problem function and found it wasn't using many of the allocated addresses. Evidently the compiler optimized those out but increased the allocation amount.

I decided I really should make a base line and turned off all the compiler optimizations. That build increases the file size by 20K over the fully optimized build, and has the stack allocation in the problem spot down to $280. I could barely even recognize the assembly that was produced with all the extra crap.

Hopefully I should get a happy medium as I tweak through the optimization settings. If I can't find one I am liable to go hunting for a new C compiler.

SubAdmin Joined: 18 Nov 2001 Posts: 5187

Adjusting the compiler optimizations did the trick. I actually got some better numbers then the version oldguy2 tested. The data he provided to replicate this problem now successuflly passes the compatibilty report. However, I can still create a crash with some regex patterns when used within scripts.

I have emailed Zugg the new dll and some files replicate the further crash.

Posted: Mon May 05, 2008 5:23 pm

I've got the new PCRE.DLL file from Vijilante, but I'm still confused and concerned about this. As far as I can tell, v2.23 increased the stack size in CMUD by a factor of 8 over v2.22. So I don't understand why OldGuy's crash still happened if it's related to stack size issues.

But I'll put the updated PCRE.DLL into v2.24 and also put in the improved regular expression for testing %1..%99 and %i,%j,etc within quotes. Thanks for helping with this Vijilante. It's one of those great examples of how frustrating writing Windows software really is these days with stuff like this that works fine for most people, but not on some computers.

Posted: Mon May 05, 2008 9:16 pm

Btw, the new regular expression still matches the following case:

SubAdmin Joined: 18 Nov 2001 Posts: 5187

There isn't really a good way, because the " character is both the opening and the close, and then there is the use of a tilde before it to make things even more confusing. In the old pattern I tried to lock in the start of the string and have the pattern work its way through the text. This is why it tended to recurse so deeply. The new pattern I aimed more to let it fail from time to time allowing the recursion to break and the top matching loop to move along, neither solution is great.

If you look at the regex I use for parsing all the syntaxes for the subregex you will see the way to handle this is through a pattern defined recursion. That is what I did to cover all the nesting of conditionals, but in order to really make it work I had to use callout items to do the variable width look behinds that were needed. Some problems are complex enough that regexes struggle to handle them without supporting code.

Adjusting the new pattern so it was slightly anchored seems to have helped.

Posted: Mon May 05, 2008 10:21 pm

I've got to give you credit Vij...you sure know regular expressions a *lot* better than I do. I'm taking your new pattern on faith :)

I did a bit of google searching on matching quotes with regular expressions, and it does seem to be a classically hard problem. I think this is good enough for what the compatibility report needs. Thanks!

SubAdmin Joined: 18 Nov 2001 Posts: 5187

I think I got the best possible regex for this. It drops out like the one above, and also uses a recursion pattern to match quote pairs. I am using a sizable amount of data for tesing on this to make sure it doesn't cause a crash. New patterns in the test alias

SubAdmin Joined: 18 Nov 2001 Posts: 5187

Bah, I found another way to make the above regexes fail when they should match. I will include a new regex in an email with final updates for %subregex.

Wizard Joined: 14 Aug 2004 Posts: 1269

Time has gone by since:

Posted: Thu Jul 24, 2008 4:54 pm

Detecting a stack overflow and preventing it is very difficult. In this case, CMUD can't do anything because the problem code is in the DLL. Vijilante has talked about looking into the PCRE code itself to improve this. All I can think is that RegexBuddy is not using the PCRE.DLL file like we are. But once CMUD has called the DLL function, there isn't anything it can do to trap a stack overflow. There is no general purpose way to trap a stack overflow like other exceptions. By the time you detect it, it is too late and often impossible to recover from. I've done some research into this for Delphi apps and haven't found anything helpful.

SubAdmin Joined: 18 Nov 2001 Posts: 5187

I am going to ramble a lot in this reply. Many of the details are far too complex to get into, but I would hope the tip of the iceberg is enough to understand why CMud has sunk in the past on this issue.

RegexBuddy does not use PCRE. If you look at thier documentation they wrote thier own engine with the express goal of displaying each step in the standard regex alogarithm. Last time I checked they also didn't support anything that was added to the Perl regex syntax since 5.00.

My current thoughts on eliminating the stack overflow are to:
1. Determine actual stack usage, as I showed above this is dependent on how the dll is compiled, so I have to come up with some assembly to do it. Any code in C would be fair game for the compiler to muck in, and also likely would not give me consistent results. I tend to think that to be the easy part.
2. Determine how much stack is available. This is the hard part. Stacks can be dynamically allocated and I have to probe the applications allocations to determine the one that is applicable. Each thread actually has its own stack area so I have to make sure I am matching to the space I am running in at the time. I would like to make it as a non-OS dependent, ANSI C function. That way it can be included into the PCRE source by the author, and Zugg wouldn't have to be concerned about having this problem when he upgrades to a new version. If I can't come up with a general way to do this then it becomes another of the items in the custom dll, that requires maintainance in order to update the PCRE.
3. Calculate how far the regex can go. Again this is an easy part. Simple math handles it. The PCRE already provides a mechanism to stop the pattern from creating a stack overflow, but the problem is in setting a cuttoff and letting the user know they hit the cutoff instead of just having a no match. It is up to Zugg to decide how he might inform the user of having hit the cutoff. It is up to me to make the cutoff dynamic so that an arbritray limit is not required.

We could currently set the default limit in the inerface low enough to ensure that no overflow can occur, but the actual limit really needs to be dynamic to accomodate different things. Having a trigger pattern fail because of some limit is a problem, having CMud crash is worse. Most trigger patterns can't cause a crash because the data being tested is a single line. The most recent crash found in the Compatibility Report required a much larger dataset, that didn't contain linebreaks. I set up the regex to release on a line break; however, a small changes later produced a regex that didn't crash even under more extreme circumstances. The latest version I have supplied Zugg maintains speed while eliminating the crash, but recognizing the validity of that change comes from study of the PCRE code.

I fully expect users of CMud to write bad regexes. I have made hundreds of them, and caused both CMud and zMud to crash, burn, and die a thousand deaths while testing regexes. That is something I am aiming at preventing, if I can.

Wizard Joined: 14 Aug 2004 Posts: 1269

Well, you're both clearly aware of all the issues, and it does sound like crashes are very unlikely at this point in time, but obviously crashes with no crash dump can be difficult to track down, especially if it was MUD output that caused the crash instead of the Compatibility Report, so any improvement on this would be great. But I'll leave you to decide what (if anything) to do about it.