Zugg Software :: View topic

Posted: Tue Oct 24, 2006 1:28 am

I'm really getting tired of this. Yes, I'm dealing with 3rd party code problems AGAIN.

I started working on the trigger state problems today. The problem with manual triggers turned out to be relatively easy. The #SET command was getting confused with the #SETPROMPT command because of how the hash table was getting set up and the order in which the command database table was being read. Took just a couple of minutes to fix.

So, I figured, while I was working on trigger states, I might as well try to fix the user interface for trigger and button states.

It seemed like such a simple idea.

But no...the Developer Express DBTreeView control decided to waste my entire day. I'm getting SOOOOO SICK of this component that I'm about to throw it out. I just hate to do something that drastic this far into the beta testing.

The problem is with the updating of this tree. It seems to update itself when I DON'T want it to (which makes CMUD incredibly slow when the settings editor is open), but then it DOESN'T update when I DO want it to.

In this case, I was trying to draw an icon to show which trigger state was currently active. The option field in the database record has a "IsCurrent" bit that is set for the current state. So I thought I could just easily test this bit in the tree view to decide what icon to display.

But the stupid tree view does not update itself when the option field is changed for a record other than the current record. So, if you are looking at the main trigger record, the icon for the current state doesn't update.

There also seems to be a really wierd effect when I disable something. When I disable a trigger, all of the custom text shown for different trigger states (such as Wait NNN second, Manual:, Alarm:, etc) goes away. Again because the tree isn't updating properly. You have to click on the tree to force it to update itself again.

I'm *really* getting sick of this component. I know that a DBTreeView is a hard component to write, but this is getting ridiculous. Everything that I seem to try to improve the UI just doesn't want to work with this.

The whole idea of using database components in the settings editor was to ensure data integrity. By displaying the data as it exists in the internal database, I was hoping to ensure that you were always seeing the correct data.

If I use a non-DB tree view, then I'm going to have to build the tree using the internal linked-list cache. But if there is a bug with sychronization between the linked-list and the database, then you might be seeing the wrong data in the linked-list records. It's also difficult to tell the tree view that it needs refreshing when some field in the linked-list changes since these are in completely different modules. The database has events to fire when something changes, which the current DBTree hooks into. I'd have to set up a similar event structure for my linked-list, which is duplicating a lot of code that already exists for the database.

And the other whole idea in CMUD was to use this database and NOT use the linked list. The linked list is for zMUD compatibility.

No...the more I think about this, the more I don't want to get rid of the DBTree. It would also break all of the filtering and search code. I just need to find some way to tell the stupid tree to update a different record other than the current record and figure out how this whole updating stuff works and make it work properly.

But man, what a pain. This is NOT what I want to be working on. I'm only a week from the planned CMUD announcement and I should be cleaning up bugs and not messing with this kind of crap.

Posted: Tue Oct 24, 2006 3:10 am

Sorry to hear it's been such a fustrating day Zugg. Are there any other good DBTreeView componennts by other vendors? I know with things like this the devil's in the details so it's hard to test how good it really is unless you're deep into it.

Take a breath, do some *fun* coding tomorrow and go back to the DBTreeView later.

Posted: Tue Oct 24, 2006 3:24 am

Nope, unfortunately the DevExpress stuff is still the best.

I finally got it to update the trigger state. There was an option called "SmartRefresh" that was causing the problem.

I've run into this problem before. In one of my past blogs I mentioned having this problem when deleting records. The solution was to delete the record from the tree so that the tree would know how to update itself. Well, for changing the current trigger state I can't do that from within the tree. It can only be done in the database itself.

So, it looks like I need to turn SmartUpdate off again. Not sure why I wanted this turned on. My guess is that it was a speed issue. But since the settings editor slows everything down so much anyway, I'm not sure it matters at this point.

I found a property that will let me turn off updating the tree until it needs to be updated. Looks like the other big speed problem with the settings editor is all of the other "data aware" controls on the form. I'm going to try replacing the data-aware components with normal components and load their values myself (like I already do with the checkboxes) and see if that helps. I can modify just a single form (an easy one, like the Direction settings) and see if that works. If that helps, then I can go through and modify all of the other forms.

I think it was Rainchild who mentioned a while in the past that he'd never found a data-aware component that worked well, and he ended up loading it himself. I think I'm finding myself in the same situation. Data-aware controls are fine for quick and dirty application development, but are not designed for performance.

Wizard Joined: 11 Jul 2002 Posts: 1265 Location: USA

I agree, I have NEVER found an auto made data-aware form that I liked once I opened the hood and had a look around. That said, anything made by Microsoft incourages the use of data-aware forms... This in itself raises red flags!!
The selling point is supposed to be ease of build, SURE its easier, if what they mean is nothing ever works the way you as a builder thinks it should. Evil or Very Mad

So you end up either doing it their way and the application never works right, Or you end up working twice as hard going back and rewriting your interface.

Posted: Wed Oct 25, 2006 3:58 am

Well, this is slow and frustrating work.

I have replaced the data-aware components for the right-side panel of the settings editor. Now the only component that uses the data source is the TreeView. I am trying a trick/kludge where I keep the DBTreeView locked most of the time so that it doesn't receive updates. The only database update that it *must* receive is when a record is deleted from the database. If you don't update immediately after a delete operation, and then the user clicks on that record in the tree, then you get a crash. And there doesn't seem to be any way to trap this particular crash...it causes all sorts of data corruption.

So, the tree still has to be updated when a setting is deleted. Fortunately, things like #UNVAR should be much less common once people switch to using local variables for temporary storage.

However, the TreeView doesn't like to be kept locked all the time. I unlock it when you click on a setting (so that the editor screen can properly update), and I unlock it when you drag/drop stuff.

I've cut the slowdown from the settings editor being open by a factor of 5 or so (makes scripts about 6 times slower now instead of 30 times slower). So that's progress and there is more progress to be made. The downside is that I've also broken most of the settings editor. Almost every little function doesn't work properly now. So it's going to take a while to fix all of the problems. Even simple stuff like changing the record and then clicking on another setting in the tree without clicking Save Changes first doesn't work. Selecting a class from the pulldown class view at the top doesn't work. The right-click CopyTo and MoveTo don't work. Copy/Paste doesn't work. And for some reason, even though Ctrl-Clicking works for selecting multiple items, Shift-Click to select a range no longer works.

Looks like I've got several days of hard work ahead to fix this mess. I really debated whether or not I should fix this. But after thinking about it for a while, there really isn't *any* "good" time to do this. And it's something that I was going to need to fix sooner or later. A lot of people bring up the settings editor to make scripts, and having this slow down MUD playing by a factor of 30 just wasn't reasonable. And as we saw over in the beta forum, people don't think about the settings editor being open and just end up thinking that CMUD is much slower than zMUD because of this. So, it's probably better to fix it now when it impacts fewer people. But I'm probably going to have to do a release on Thursday this week just to give testers a chance to see what I've broken in time for me to fix it before the big announcement next week.

I'll probably even reset the trial counter again just to encourage more testing over this weekend (not a promise, but likely).

I considered throwing out the DBTreeView and doing it myself with a normal tree view. But I think the work involved making stuff like drag/drop work would still be large, and I'd still have to deal with when to update this tree view. Right now I'm planning to put a timer in the OnIdle event handler so that the tree view is updated periodically even when it's just sitting there. Updating it once every few seconds would still be a huge improvement over what it's doing now.

Anyway, stay tuned. Maybe I'll get lucky and get these problems all solved tomorrow. Cross your fingers.

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

Yeah I load my tree views on demand from the database (which also means that it only performs a 'select' query on subsets of the database as a user requires them. I have done very rudimentry drag and drop operations which ended up just a matter of casting the 'tag' as a pointer (this is C++ builder) and attaching a small data packet to every cached item in the tree... it took a little bit to get working properly but I had some more complexities in that some of the data in the tree wasn't compatible, eg you can't drop a user into a location, or a location into a part, but you could drop a part into a location, or a part onto a delivery... and at least if something ends up deleted in the database (in a multi-user system) and you do the drag/drop you just get a SQL violation rather than an application crash, although I'm not sure if SQLlite supports foreign key enforcements so you might get orphaned records instead?

Posted: Thu Oct 26, 2006 3:56 am

WooHoo!!!!!!! (Zugg is doing the HAPPY dance!)

I have *fixed* the speed problem with the settings editor. With a few minor edits to the DevExpress source code, I was able to get more control over the TreeView updates. There is now ZERO decrease in speed when the settings editor is open (well, effectively zero...certainly hardly noticeable).

I also fixed many of the bugs, including clicking another setting while in the middle of editing, selecting a class from the pulldown, CopyTo and MoveTo and drag/drop. I also fixed the Shift-Click for selecting a range of items. The Paste function still doesn't work right and I'll be looking into that tomorrow, along with a couple of other remaining bugs. Then I need to apply these same changes to the fly-out settings editor.

I'm really happy about this. I wasn't sure that this was possible. I was going to be happy with just the 4-5 times slower that I achieved yesterday. I never dreamed that I'd get it running just as fast with the settings editor open. It required a lot of tricks to turn off the database update mechanism for the tree and still have it remain functional for the user interface.

I'm sure there are more bugs, but I feel like I'm over the hump with this. I had been depressed that it could take over a week to get this working. Never imagines to do it in two days. I think I'll stop early now while I'm ahead :)

(Btw: Rainchild: the in-memory database is not SQLite. It's not even SQL. It's the kbmMemTable database that is optimized for memory operations and interfacing with Delphi data-aware controls. SQLite is only used for the on-disk PKG files).

Wizard Joined: 14 Aug 2004 Posts: 1269

Nice one, Zugg! This is going to be a huge performance boost! I often MUD with the settings editor open so that I can quickly make script changes, e.g. while testing if they work, or just to keep my place in it, or to avoid the initial opening time. *clap*

Posted: Thu Oct 26, 2006 11:19 pm

Wow, we had a big snow storm today. Fortunately our UPS systems kept the computers running during some of the flaky power issues, so it didn't really effect work very much.

I thought I might get the new settings editor stuff all working today, but I didn't. Several of the obscure bugs took a long time to figure out, and I've still got some issues to fix. I fixed the Copy/Paste stuff (although I still need to investigate Import/Export based upon a forum post). Most of the day was spent on the New command. It was a bunch of little stuff, but this little stuff was often a sign of bigger problems with this speed kludge.

For example, when you selected something like New Alias, the new blank alias wasn't highlighted in the tree view (not selected). While this seemed like just a visual issue, it turned out that not having the proper record selected in the tree was a bigger problem that needed fixing. Once I got that working, then there was another obscure problem. If you select New Alias but then don't enter anything, and then click on another setting, CMUD is smart enough to delete the blank setting that was just created. After doing this, the tree editor was going into "rename mode". In other words, it thought that I had just clicked on the same setting twice (which puts it into rename mode). This was another indication that something wasn't getting set properly in the tree and took a while to clean up.

Then I added the background update timer. It updates the tree with any database changes every second. However, typing in the settings editor will delay the timer, as will any other "forced updates" (like when a setting is deleted). The update process is fast enough that it's not noticeable (it was only slowing down CMUD because it was getting called for *every* database change, which was hundreds of unnecessary updates).

However, I ran into an issue. There are two types of updates that can be performed: a "quick" update (just updates the current record), and a full update (updates all records). I was just doing the quick update, so it wasn't picking up a lot of changes. When I changed it to use a full update, I ran into problems because the tree view FullRefresh routine actually destroys all of the tree nodes and recreates them.

This is actually what you want to do (since just changing a single database value, like the Parent field, can actually change the structure of the tree). However, the DevExpress people were doing it quick and sloppy and not maintaining a lot of state information in the tree. For example, it would forget which nodes were expanded, and which nodes were selected, and was changing the vertical scroll position of the tree to put the current focused record at the bottom. All of this is really annoying when it happens every time a setting is changed in a script running in the background!

I've got the routines working for saving/restoring which tree nodes are expanded. But I still need to save/restore the other state information. The idea is to not have any change to the user interface when other settings are changed. For example, if a setting becomes disabled (by the #T- command), then the name of the setting in the tree should just change without effecting anything to do with the current record being edited.

This gets rather complicated. As I said, I was hoping to get this all done today for an early release for people to beat up, but it looks like I've got at least another day's worth of work on this settings editor stuff.

I also haven't touched the fly-out settings yet. So there is still lots to do. And the rapidly approaching Halloween announcement date is stressing me out. I'm just going to have to keep reminding myself (and everyone else) that just because I'm announcing CMUD to zMUD users doesn't mean CMUD is out of beta testing. I can't get stressed out and treat this like a public release. I know that some zMUD users might be turned off by bugs and partial help files and stuff like that, but they'll just have to deal with it. I'm going to get complaints and whining no matter what I do, and it's no reason not to announce CMUD to the majority of zMUD users who will be very interested in the news.

Apprentice Joined: 11 Aug 2006 Posts: 100

I am impressed and over joyed at the effort you have put into that damn tree view.
Taking the time, and doing it right!

And, I pity those who cannot see how CMUD could eventually take mudding to another level!

Posted: Sat Oct 28, 2006 3:20 am

Still working.

Today was fairly productive. I think I've fixed many of the remaining bugs with the new settings editor. However, this settings editor does so much different stuff that it's actually hard to completely test it. I also got the fly-out settings working the same way.

In order to properly update the tree, it can slow things down slightly if you are deleting settings. Deleted settings must be updated immediately to avoid crashes in the tree. Adding/changing data triggers an update on the next "tick" which is currently every second. You can turn off this background updating tick in the settings editor View menu.

Realistically, I've seen about about a 20% speed decrease when the settings editor window is open. Increasing the update tick from one second to something higher would speed this up, but in practice, you really want the data to update pretty fast. You want to be able to see settings getting enabled/disabled and see trigger states update, and stuff like that.

Anyway, this 20% decrease is certainly better than the 4000% speed decrease that previous versions had :) Here are some concrete numbers:

Test script running without any settings editor: 0.8 seconds (it's a big script)
Test script running with settings editor open (v1.12 speed improvements): 1.0 seconds
Test script running in v1.11 with settings editor open: 15.1 seconds

So, this particular test script was only showing a 1800% speed decrease, which is a bit better than some other scripts. But you get the idea. v1.12 is pretty dang fast.

Compare this to zMUD:

Test script without settings editor (zMUD 7.21): 0.5 seconds
Test script with settings editor open (zMUD 7.21): 1.3 seconds

The zMUD test script is slightly shorter than the CMUD script since it doesn't test some of the newer CMUD scripting features. And, of course, in zMUD the settings editor isn't actually updated until you manually press the Refresh button. So zMUD is slower and isn't even keeping the editor tree updated.

So, what's left before I can do another release? Well, as part of the settings editor changes, I also changed the Name vs IDName fields. Most settings don't need a separate Name field. For example, with an Alias or Variable, the Name is always the same as the IDName. The IDName is a 64-character string that can be indexed in the database. The Name field is a "memo" or TEXT type that is unlimited in length. The main setting that needs the full Name field are Triggers since trigger patterns can be long and complex.

This means I am removing the IDName in the "Advanced" popup panel at the bottom of the settings editor. I'll add it specifically to those settings that need an ID Name, such as triggers, buttons, menus, and status bar items.

What I'm trying to do is simplify the screen. I'd like to get rid of the Advanced panel if I can. I can move the comment field somewhere else (just make is a popup prompt in the Edit/Edit Comment menu like zMUD had). The only thing left I'm not sure what to do with is the Priority number.

Also, I want to fix/improve the Button settings page for multistate buttons based upon some discussion threads in the beta forum. This should mostly complete the settings editor changes and allow me to take some screen shots and start working on the help files for it.

At this point, I'll plan a release for later tomorrow night. Doesn't give people as much time to test it before the announcement to zMUD users next week, but hopefully it will be enough.

GURU Joined: 23 Dec 2000 Posts: 4834 Location: USA

Posted: Sat Oct 28, 2006 4:56 pm

Matt, you can already do that. When you select the Sort by Priority option, then the tree is sorted by priority and the priority column appears in the tree. And the priority column is editeable as expected.

But I don't like showing the priority normally. It really clutters the screen and could confuse new users more than the hidden advanced panel.

Posted: Sun Oct 29, 2006 2:43 am

OK, now I'm *really* burned out.

I went ahead and released v1.12. I'm honestly not very happy with it. There was a lot more I wanted to fix, but I just had to get *something* out so people could play with the new settings editor and find bugs/crashes before the announcement to zMUD users next week.

I intended to get Events added, but didn't get to this. Instead, I got to spend 3 agonizing hours today tracking down a bug in how Windows/Delphi was repainting the background of a control. When I redesigned the settings editor, I moved all of the advanced options in the Advanced panel at the bottom of the window. This includes all trigger options. When doing this, I split the trigger options into 3 columns. Each column was within a Panel component. Each Panel component is transparent to allow the theme background to show through.

For some reason, the center panel wasn't updating it's background. When you closed and opened the Advanced panel, the previous window contents were being shown in the background of the middle panel.

This drove me insane. I wrote a test program and it worked fine there. Since these are nested panels within a DevExpress PageControl, which is within a dynamically loaded Frame, which is within another Panel and yet another PageControl, this gets rather complicated. I've run into bugs with both the PageControl and with Frames in the past, so I thought that was the cause here. But I couldn't get it to fail in a test program.

I ended up copying/pasting part of the settings editor design into a test program and finally got it to fail. But then suddenly it was working again with no apparent reason. I then made another copy of the settings editor and it started failing again. What the f is going on here!?

After some long cursing at the computer, I discovered that for some reason, the exact position of the PageControl within the form design (Delphi DFM file) caused the problem. If the PageControl was defined before the panels below it, then the bug happens. If I just move the PageControl to the bottom of the form design file, then it works fine. I've *never* run into a case where the order of the components in the form file mattered. I don't know if this is a bug in Delphi 7 or a bug in the PageControl component. But at least I finally found a way around the problem. Can't believe I wasted THREE HOURS on a stupid transparent painting issue.

So, I'm tired and burned out. I've thrown v1.12 out for people to download it, but I really haven't finished a lot of testing that I had planned. While I was working on the settings editor redesign, I also improved the editing of trigger states and button states. I hope you like the way the new button editor works. However, it still has problems dealing with multiple button states.

Actually, the design of CMUD allows some new features for buttons. Because each button state is stored in it's own button record in the database, each state can actually have more than just different colors and names. Each state can have it's own icon and it's own other properties. CMUD doesn't handle this properly yet. The settings editor lets you change the icon for each state, but only the first state icon will be shown in the main MUD window.

Also, using the pulldown menu to select whether a button is toggle/push/multistate/etc doesn't really work properly. You must currently create new button states using the New Button State button (also in the New menu). There are really several rough edges of how this all works that need to be addressed in a future version.

But right now I'm more interested in crashes in the settings editor. I'm a bit worried about the way in which I kludged the performance improvement. If the tree view gets out of synch with the database, it could cause all sorts of crashes. The tree view just isn't designed for this kind of "offline" method that I'm using. I've played with it a fair amount today, but the settings editor is so big that I'm sure I didn't test everything. I expect to see a bunch of new crash reports.

But for now, I'm going away from the computer for at least a day. I'm getting bad about working too much again and have had several very long days in a row. And I really don't like making such huge changes to the core architecture at this late stage in the beta. Makes me really nervous and stressed.

So, go play with CMUD, report any new bugs, but be kind. I hope this all turns out to be worth it.