Zugg Software :: View topic - [2.25] Cmud lockup caused by trigger using list including a null item.

Posted: Wed Jun 04, 2008 2:38 am

As I mentioned in a previous post about lockups, I had found that having a trigger using a list with a null item can cause a hard lockup on Cmud. I'm still having intermittent lockup issues not related to this, but I thought I'd post about this in case this is fixable in the code.

Creating a list with a null member may seem a bit contrived but this happened to me in normal usage because something with the #additem/#delitem can at times produce a list with a null member. Perhaps this is a separate bug, but I wanted to make clear that lists with a null member was something that Cmud code might do.

Procedure:

Posted: Wed Jun 04, 2008 3:11 am

I'm revising my opinion that the %null somehow resulted from #additem/#delitem. I came up with a small stress test to add in 10,000 values to a list then to randomly delete the values out of it. I also add one null item in myself to make sure the detection works properly. It passed with flying colors so I can only suspect that I am somehow adding a null in somehow myself.

The code below is interesting in that it shows how bogged down Cmud gets when dealing with lists of great size.

Posted: Wed Jun 04, 2008 5:37 pm

I'll test the lockup later today and see if it's a PCRE problem or not.

But regarding the "shows how bogged down CMUD gets" issue, let me explain some internal details.

1) The #ADDITEM command doesn't add an item to the list if the item is already in the list (doesn't allow duplicates). So, each time you use #ADDITEM, CMUD needs to search the existing list to see if there is already a member. In an unsorted list, this search will take longer and longer. In a sorted list, this search is very quick. So after your '#additem list ""' command, you can add a '#sort list' command and then you will see that it is *much* faster.

2) To avoid this issue entirely, you can use the %additem function instead of the #additem command. The %additem function allows duplicates in the list, so it doesn't need to search for the existing item before adding the new one.

P.S. What's *really* fun is to replace the local variables with regular variables and then try this script in zMUD to see just how much slower zMUD really is.

Posted: Thu Jun 05, 2008 5:44 am

Does Cmud automatically keep the string list sorted if I click the sorted button on the variable? I've never understood well what that box was supposed to do.

Posted: Thu Jun 05, 2008 5:33 pm

Yes, clicking that box does the same thing as using the #SORT command.

Posted: Fri Jun 06, 2008 12:53 am

Wow, I can't believe how much effect clicking on the 'sorted' box has on the speed. The adding portion finishes nearly instantly. The deleting still takes a bit of time, but the effect on the adding was very profound.

I'm curious now why #delitem wasn't speed up just as much having the list sorted.

Posted: Fri Jun 06, 2008 1:21 am

When you have the tree sorted, Dephi is creating a binary search tree. When you delete an item, I think it needs to rebuild the tree. I'll have to look at their code for deleting an item in a string list to be sure. But I'll check it out to make sure there isn't a bug in that.

Posted: Wed Jun 11, 2008 11:30 pm

Posted: Thu Jun 12, 2008 12:09 am

The good news is that the delitem speed is fixed in v2.27. The bad news is that I don't know why ;)

When I tested your script today, both the Add and Delete phases were really quick (with the #SORT command added). The Delete was definitely slower a few days ago, so it must be related to some other change that I made.

But it's definitely not rebuilding the tree when you delete an item. So deleting items is nice and fast now.

As far as the lockup goes, it seems that PCRE doesn't like null values in a list like that...at least not at the end of the list, which is where CMUD was putting it when it sorted your list. In 2.27 I am putting the null value at the start of the list, which prevents the lockup.

Normal CMUD patterns don't have this problem because when the pattern is converted to a regex, null values inside {a|b|c} lists are removed. There is a comment in my zMUD code that says "Null values not allowed in standalone lists", and this issue has been around for a long time. A PCRE expert would need to tell us more about it.

So in 2.27, it probably won't lock up, but your regex might not work as expected either.

Posted: Thu Jun 12, 2008 12:10 am

Btw, v2.26 also had a new bug that was preventing @list stringlist and database variables from expanding properly within a RegEx trigger. This is also fixed in 2.27.

Posted: Thu Jun 12, 2008 12:20 am

Wow, some good fixes going on, thanks for all your work.

Posted: Thu Jun 12, 2008 1:39 am

On thinking about this a little bit more I should probably just use Cmud's search instead of Perl. In fact I just might do that, I guess I'm just more comfortable with Perl since I've been using it for years and am quite good with the regular expressions.

I was wondering, is there any reason why null values are allowed in string lists? Given your response above it sounds like it might be a better idea to just disallow them altogether and avoid the whole issue. For me they are just a nuisance. They sneak in there from time to time onto my lists making life a pain for me. I have code that just goes through my lists and deletes the null values that are encountered, but it seems like it would be better if they just couldn't exist in the first place.

Posted: Thu Jun 12, 2008 5:29 am

If you use CMUD trigger syntax, then null values will be eliminated automatically. I might do that with normal regular expressions too, but I'm waiting more for a regex expert to explain how to properly use a null value in a list. When the null value is at the start of a list, they kind-of work, which is why I left them in for now. I didn't want to prevent someone who was a regex expert from doing something that they understood and wanted.

Posted: Thu Jun 12, 2008 8:00 pm

My lockeup error here:

http://forums.zuggsoft.com/forums/viewtopic.php?t=30924

contained the null value at the start of the list.

Posted: Thu Jun 12, 2008 9:47 pm

It should work in v2.27. Give it a try and let me know. Otherwise it's possible that having the null value at the start of the list still causes problems in some cases and I'll just need to completely strip nulls. Still waiting advice from a regex expert on nulls in lists.

SubAdmin Joined: 18 Nov 2001 Posts: 5187

I don't know of any reason that a null item in an alternation list would cause any problems for the PCRE. I actually use them quite frequently as a way to make a section optional during the match. This is because zScript pattern syntax doesn't have an optional notation. I have done it that way for years.

A proper conversion for the pattern "{a|b||c}" would be "(?:a|b|c)?" and the pattern "({a|c|b|})" would be "(a|c|b)?". The instances where order is important are quite few, generally can be written in another way, and always require more complex regex syntax like a conditional item.

I believe it would be fine to strip null values from a variable in both the zScript conversion and regex conversion. If a user is writing with a regex and intended to have the section optional, then they would have put the appropiate notation.

Posted: Fri Jun 13, 2008 2:58 am

Hmm, I didn't think about just adding the ? to the end of the paren list! That's brilliant. CMUD is currently converting "{a|b||c}" to "(?:|c|b|a)" and this causes weird problems.

But you are correct. Since you can easily use a regex of (@List)? to handle a list with a null value, I think I should probably just strip the null value from the string list or database variable. When converting zScript to regex, I'll consider adding the ? to the end of a null value is detected.

Can't believe I didn't think of that solution. Brilliant as always Vijilante!