Zugg Software :: View topic

Wanderer Joined: 31 Jan 2006 Posts: 55

I have a 6 line trigger script to put items in a database. I used it many times, and it seemed to be working good...

But now, it is not. It is adding many duplicates of database items for no reason. Most entries that are added to database scroll very fast on screen, with little delay. I think this is the problem. I put #echos in the script and it appears that either the %query to check for duplicates or maybe the #NEW to add new records is not processing before the next trigger happens. The result, is %query is returning no match for duplicates, even though there are duplicates in the database..

I also ran test to trigger the script using #echos.. if I spaced them #wait 300 apart they were added perfect with no duplicates. But when it is scrolling on screen there is no way to slow it down so this is not practical..

I don't understand this. The database only has about 300 entries. (well it numbers to 1100 but most of those were deleted...). My computer is very fast, and database file only about 25Kilobyte. Why is this a problem? Is there any way to fix it?

Adept Joined: 21 Sep 2005 Posts: 250 Location: Austin, TX

Well...it'll be hard to help you fix the problem without you posting (a) your script that is failing, and (b) sample MUD output that it should be operating on. Be sure to include prompts before and after, plus the command(s) that you are typing.

Wanderer Joined: 31 Jan 2006 Posts: 55

sorry, didnt think it could be my script, since it worked when the db was small, here it is.

MUD output like:

SubAdmin Joined: 18 Nov 2001 Posts: 5182

It does seem like speed is the issue. Try using #PRIORITY like this.

Wanderer Joined: 31 Jan 2006 Posts: 55

Hi, it worked! well, so far at least. I will just have to see how it does when the database gets bigger, and maybe go from there.

Thank you

ps: is there a way to put the db list in order? my highest db number now is about 600 and anytime I add new items to database they are labeled 1100+ because of the ones I deleted. I don't know if this slows it down more or not....

Adept Joined: 21 Sep 2005 Posts: 250 Location: Austin, TX

The reason that the #PRI works (I think) is that some ZMud can continue to process MUD output while it is executing code. Thus, it's possible for a trigger to fire on a later line while a trigger is still executing on an earlier line. In practice, only a subset of commands actually exhibit this behavior - mostly the looping commands like #WHILE, #FORALL, etc. However, the query commands may also be in this set, because there is a (potentially long) wait while the DB code responds.

Putting #PRI around a block forces ZMud to finish executing the block before it examines any more MUD output. As your DB gets bigger, your queries may take longer to execute, which could cause a noticeable pause while the query executes. I have no wisdom to offer on how big the DB needs to be for that to occur. (anyone?)

As for the numbering - don't stress about it. The DB has an internal counter, and increments it for each NEW, so that the ID is always unique. It's not a reflection of the number of entries that the DB code has to examine. Also, there is no way that I know of to change the IDs of new entries.

Wanderer Joined: 31 Jan 2006 Posts: 55

Yes i'm afraid i've found that out. It did not take many records before it started to slow down much more. I now have about 1800 records in database. Now that i've added more records to database, new records added are starting to go very slowly. At the rate of about one per 5 seconds and rising. Sad

I am also starting to get lag with my separate script that queries the database too, but not so much. To query database for one item, it takes about 1 sec now.

I fear it will get very very slow soon. I estimate the database may get as large as 10,000 to 15,000 records. At this rate of lag, it will take 45secs to add one new record with above script, and as much as 10secs to get one record out of the database with %query, once the database is full....

Is there any solution to these problems? Vijilante mentions about capture data into a list variable, then process it when no more coming in. This may solve the problem of lag when adding new records, could you elaborate?

But Is there any way to speed up the single item queries too once database gets very big? Or is it just a fact of having such a big database, and nothing can be done.

GURU Joined: 26 Jul 2001 Posts: 1958 Location: Canada

Ever consider #WRITE 'ing it to a text file in a formated way then using %grep? I dont think you'll see much loss in speed with grep in a large text file (untested). But I've used it in the past with large success.

Might be faster.

You could have a child window with something like:

#ALIAS dowrite {#WRITE 1 {%-1}}
#ALARM *5:00 {#close 1;#file 1 mydatabase.txt}

I say a child window so you dont see the spam of the write/read text.

dowrite is only called if a %grep of the name/room is not found (ie no duplicates)

Adept Joined: 21 Sep 2005 Posts: 250 Location: Austin, TX

As I understand it, there's not much to be done at the moment. The DB module basically just manages a big flat file, so it doesn't scale all that well (as you're discovering). To get better performance, you need to get a real DBMS that can do indexing, caching, etc. involved (Access, SQL Server, etc.). In ZMud, the only way to do so is through moderately heavy COM programming (see ADO Programming in ZMud).

In the new CMUD, however, there apparently will be a brand-spanking new DB module, which will use ADO under the hood, so you won't have to jump through such hoops. See the "New Database Module" post in this thread.

Wanderer Joined: 31 Jan 2006 Posts: 55

Hm I don't know about the %grep, it wouldn't be hard to test I guess ? I could make a script to go through database records and write them to a file. I will try that. But wouldnt you think the zmud database would be quicker, since its designed for lots of records?

While I was looking through search I saw this post, I was going to ask about it:
http://forums.zuggsoft.com/phpbb/viewtopic.php?t=20089

I would need to download MySQL to do a database? I have excel on my computer, don't have much else for a database. And then use COM like in the post? Do you think it would be very hard to do? I don't really do much scripting for the databases, just the simple script above and another to query it.

Wanderer Joined: 31 Jan 2006 Posts: 55

This was a surprise. I changed my scripts over to use the file instead of database and it works almost 10x faster. Wow!

I copy pasted bunch of duplicate records into the file too, had almost 15000 total. With this many a single query to file took only 0.75secs, even less than the db query did with only 1800 records...

Still there is some lag but so much more better than the database now. It is less convenient not having GUI to use with it but for 10x the speed its a good trade. Thank you for the idea.