Zugg Software :: View topic

Posted: Mon Mar 19, 2007 5:47 pm

Chiara is recovering well and doesn't need as much attention now (even though I'm still doing some of her chores for the next several weeks). The Delphi CodeRage conference is over. So, that means it's time to get working on the CMUD 1.27 release.

My main priorities in this next release are:

Fix crash bugs. There are several bugs causing crashes that were new in 1.25 and 1.26. I am also going through the crash dumps for 1.24 to try and remove more problems that cause actual crashes.
Support for alternate scripting languages. I want to restore the support for languages such as VBScript, JScript, PerlScript, etc. I will be using the scripting host that I wrote for zApp rather than the Microsoft Host that I used in zMUD. This will allow a lot more flexibility for scripting than zMUD had.
Trigger tester. This will be the first step towards a better trigger debugger. The initial version will be pretty simple, but should help with basic trigger testing.
Lay ground work for SSH. I'll be working on the network socket layer to prepare CMUD for the upcoming SSH support. But 1.27 won't have SSH yet.
New script editor. I'm evaluating a new editor component for the package editor. It is better designed for script editing and has better syntax highlighting. It also has line numbers and row numbers and hopefully won't have the problem jumping to a specific line/row that the current editor component has.

As usual, I'll also be fixing other bugs from my current bug list. Bug fixes are something that I do in every version. But unlike the previous two versions that were *only* bug fixes, I need to start moving forward on some of these other improvements.

So, that's the current plan for v1.27. As always, I don't announce any formal release dates, but my hope is to keep to the current monthly schedule and have the new version ready by the end of the month.

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

Out of curiosity, is this new version going to be done in Delphi 2007, or is that planned for a little further down the track (1.3x/1.4x)?

Posted: Tue Mar 20, 2007 5:03 am

No, this version is still in Delphi 7. The Delphi 2007 stuff will take a while, as I mentioned in another blog. It will probably take me a couple of weeks to get all of my 3rd party components working for it, and to get the other stuff in Vista that I need to actually do real development. I'll definitely let everyone know when I make the switch to Delphi 2007.

Posted: Tue Mar 20, 2007 6:30 am

Tonight I was able to get regular Delphi 7 working within Vista. This will be a good intermediate step and will allow me to determine why Vista causes some problems with CMUD. There are some COM/OLE issues with the new Vista UAC I think. My local Zuggsoft customer database application cannot access Microsoft Outlook via COM unless I turn on the WinXP compatibility for it. And CMUD sometimes gives an error accessing the OLE registry.

Now that I've got Delphi 7 working within Vista, I should be able to directly debug these kinds of issues. This means that even if I don't get Delphi 2007 working for a while, I'll still be able to improve the Vista support in CMUD.

The bad news is that I was not able to get Delphi 5 to run, so there is still no way for me to compile/test/debug zMUD on Vista. Oh well...I didn't expect that to work, but I thought it was worth a few minutes of work to try it.

Posted: Wed Mar 21, 2007 10:51 pm

I've mostly finished my evaluations of various code editor components. It's been a bit disappointing actually. Here were my requirements:

Implemented in Delphi, or a Delphi interface available
Syntax highlighting. Support for existing scripting languages (VBScript, JavaScript, PerlScript, RubyScript, XML, etc)
The ability to add new language support so I could add highlighting for zScript
Handle wrapped lines properly, with routine to jump to a specific line/column and to display the current line/column
Line number display
Customizable colors/styles.
Support for spellchecking (preferably using the Addict spellchecker already used in CMUD)
Support for tooltips (syntax hints)
Support for code completion (templates)
Optional: Export to HTML
Optional: Code folding. I'd like to have code folding the {} sections in zScript
Optional: Bracket matching. I'd like it to display the matching brace/bracket/parenthesis
Documentation
Under active development. Must already support Delphi 2006.
Licensed for use in commercial software.
Ability to handle large files
High performance

Here is what I reviewed and what I decided about each option:

Dream Editor: Didn't look at this very much because it hasn't been updated in years. Last version was for Delphi 7. Commercial.
PlusMemo: Meets most of the requirements, but the code folding is supported by a 3rd party and it doesn't work very well. But it does have nice Addict spellchecking built in (the only component to have this already). Commercial.
SynEdit: This doesn't have code folding built in. A 3rd party has an editor called Mystix that has added code folding to SynEdit. The problem is that while SynEdit uses the Mozilla license (which would allow commercial use), Mystix uses the normal GPL license on their enhancements. So I don't think I can use the code-folding version in CMUD. Also, no spellchecking and it wasn't obvious how to add it. Also, no documentation at all. Technically the component looked very good, and it's all pure Delphi code. The lack of documentation was the real killer here. Open source.
Scintilla for Delphi: Scintilla is a very popular open-source editor component. And while it's written in C, a nice DLL with a Delphi wrapper is available. Since I already need to distribute the SQLite.dll file with CMUD, I don't really mind adding another DLL. Scintilla meets most of the requirements and is quite a nice and robust editor component. I was able to add Addict as-you-type spellchecking in about an hour. The Code-Folding works very well. Documentation is a bit scattered, but since many people use it, Google usually produces some useful information. This looks like the best component, but, unfortunately, it has a difficult "lexer" system to define language syntax. This allows it to product some of the best syntax highlighting, but it also means it will be a complex and difficult project to create a Scintilla Lexer for zScript. Scintilla already includes lexers for many languages, and I might be able to adopt one and modify it. But that will involve writing C code and recompiling the Scintilla DLL for Windows with the new lexer included. The Delphi wrapper already works with both Delphi 7 and Delphi 2006. Open source.

I really like Scintilla, but I wish I could find a general purpose lexer for it. SynEdit is all Delphi and mostly works, but I'm really hooked on the idea of code-folding (especially for a future feature that will let you edit the XML code for your scripts directly). I'm really annoyed that Mystix choose to only use the GPL and not the original Mozilla license that SynEdit uses.

It's possible that one of the existing Scintilla lexers will work with zScript. But I haven't been able to find any documentation on the different lexers that are provided. I'm going to look through some of the lexer source code tonight and see how hard it is to figure out. If I can find a language already implemented that is generic enough, I might be able to use it. For example, their Lua lexer is pretty general purpose, with many user-definable keyword lists.

So, I think I'm going to put some more work into the Scintilla stuff tonight and then decide for sure which component to use. As usual, there is no "perfect" component. Although either Scintilla or SynEdit will be far superior to what CMUD currently uses for script editing. No matter which one I pick, the syntax editor is going to be *very* cool. I just need to make the best decision for future development.

Posted: Wed Mar 21, 2007 11:34 pm

People tend to see the GPL and go "Wow, we're making free software, we should make it FREE for EVERYONE! Lets use this!" without really considering the implications of it. I can imagine your frustration :(

I think it ultimately comes down the question of which is more work: Adding spellchecking and code folding (which sounds like an awesome feature by the way) to SynEdit or adding a lexer, which you can't really do without, to Scintilla. Only you can answer that, and I'm not sure whether I'd prefer code folding without syntax highlights or syntax highlights without code folding in the short term.

Posted: Wed Mar 21, 2007 11:46 pm

Ah ha! I found a feature in Scintilla that allows you to perform the lexer functions within your own source code. This is called "container lexer" using the SCLEX_CONTAINER lexer type. When this is set, the Scintilla component calls an event (in Delphi) whenever it needs to know what style to set the characters.

Thus, it should be possible for me to write my lexer in Delphi myself without messing with C and rebuilding the Scintilla DLL file. This is just what I was looking for.

This illustrates part of the difficulty...the documentation and really scattered around and it take a long time to figure out all of the features that are available. But once I find the examples, then it works well. I had this problem when adding the spellchecking. Initially, I couldn't figure out if Scintilla could even handle something like a "squiggle underline" that is used for as-you-type spellchecking. Support for this kind of style needs to be built into the component. For example, SynEdit doesn't seem to have any support for fancy styles like this. In Scintilla, I didn't see any support for this initially either. But then I learned about "indicators" which are separate from "styles". The "style" indicates the syntax type, and is provided by the lexer. But "indicators" are additional features that can be set in addition to a style. And Scintilla already had a "squiggle" indicator. This allows text to be set with an indicator without effecting their syntax highlighting. Typically this is used to flag syntax errors, and stuff like that.

Once I discovered "indicators", then it was very easy to add spellchecking to Scintilla. But the lack of good documentation made it take a while to figure this out.

Now that I've discovered the "container lexer", I think I'm all set. I also found some code examples on implementing tooltips for syntax "hints". I also found an example for autocompletion for potential "templates". So Scintilla seems to have all of the features that I'm looking for. I'm very happy about this, since Scintilla was definitely the most robust and stable component that I found.

So this is going to be pretty fun for the next couple of days. I just love implementing really cool stuff like this!

Posted: Thu Mar 22, 2007 1:08 am

Incidentally, will hovering over variable names showing a tooltip with the value be making a comeback? I personally found that really useful, and it'd be great to know which variable the alias is looking at and which value it's getting without having to testfire it.

EDIT: Oh, and while I'm thinking about it, do you have any concrete plans for that expanded forums/website/blog/discussion thing your were talking about a while ago? I seem to remember you saying something about wanting to use AJAX for it, and you said that Delphi for PHP would have many of the features you wanted, which implies you're going to wait for DfPHP before you move forward. Is that the plan? Some of the ideas you had were really interesting.

Posted: Thu Mar 22, 2007 1:57 am

Fang: Yes. I'm planning both the traditional "mouse-over" syntax hints, as well as syntax "completion" hints like other editors perform. The first version will probably just have some basic stuff since more detailed syntax information requires that I integrate the help system with the "quick reference" help. Currently the syntax hints only come from the quick reference, which doesn't have the exact information that I want.

The variable value mouse-over tip is planned, but probably not in the next version. There are issues with recursion when the variable is a user-defined function. This also caused some issues in zMUD and I need a better design for this in CMUD. Also, the zMUD stuff used a different parser from CMUD and the CMUD parser doesn't currently have this info.

The off-topic stuff about the web site is currently on hold until I have a chance to get my copy of Delphi4PHP. This product could really revolutionize how I can build web sites and has Ajax heavily integrated. So yes, I have lots of ideas, but they will have to wait till Delphi4PHP is released (which should be at the end of the month...plus another week or so to ship it to me). And that web work will only be done in the evenings as I have "spare time".

Anyway, back to the topic at hand. I had good luck with this tonight. I created a "proof-of-concept" container lexer for zScript. It does some basic syntax highlighting for commands, functions, variables, etc using a simple and stupid parser just to see if I could get it to work. It also includes the code-folding on the {} sections.

And it works! This is really great. It means that I can definitely use the Scintilla component and do everything that I want with it. It was very easy to create the container lexer. The code folding was a bit harder because it wasn't well documented. But I finally took a look at some of the source code for the SciTe editor that comes with Scintilla to see how they did some stuff, and I was able to figure it out.

That's a good amount of work for one day. Tomorrow I will go from writing test programs to trying to get a real version of this integrated with CMUD and write the real lexer for it.

GURU Joined: 23 Dec 2000 Posts: 4834 Location: USA

What's code folding? Is that going to mean we're going to have our nice, beautifully laid out code cluttered up with plus/minus buttons all over the place?

Posted: Fri Mar 23, 2007 3:17 am

Posted: Fri Mar 23, 2007 4:41 pm

The plus/minus icons are only displayed in a column on the left side of your code. If you have used an editor like Notepad++ then you will know exactly how it will look. And of course you'll be able to turn if off if you don't want it.

Posted: Fri Mar 23, 2007 10:53 pm

Here is a quick preview of the new syntax editor. I just grabbed the latest script from the Finished Scripts forum and cleaned up the first part of it to improve the multi-line formatting.

Here is the first picture, showing the cursor next to a } and showing the automatic brace highlighting feature:

In the above picture you can see the line numbers displayed on the left. To the right of the line numbers is the gutter where the collapsible sections are displayed with the + and - buttons. You can also see the Indentation guideline (the dim dotted lines), and you can see that the current line is slightly colored a very dim yellow. With the Brace Highlighting, notice that the indentation guideline for the current matched brace is also highlighted in yellow compared to the other indentation indicators.

In this next image, I have clicked on the second "minus" symbol on line 5 to collapse the IF block:

Notice that the line numbers are still correct even after collapsing the section. The line numbers also work correctly when lines are wrapped.

The tab key is set up to work with indentation levels. Pressing tab will indent by the current indent amount (default is 2). When you press Enter at the end of a line, the cursor is placed at the same indentation as the previous line. Pressing the Backspace key will "unindent" by the current indent amount. In fact, there are lots and lots of keyboard shortcuts already built into the Scintilla edit control. I'll be documenting all of them in the help file, and will eventually add a way for you to customize all of the shortcuts.

Currently I'm still using my "stupid" zScript lexer to do the color coding. I spent today finishing the spellchecking stuff (implementing the right-click menu for correcting spelling, and implementing the main Check Spelling dialog that checks each word in the memo field). I also made my own descendant class for the Scintilla editor to make it a bit easier to set the various color schemes and language settings. This will allow the editor to work within the Package Editor where you can select your desired scripting language from the dropdown combobox.

On Monday I will finish with the improved zScript lexer. Then I'll get this all hooked up to the new menu options in the actual package editor and get the script language selection working properly, and get it working with the Check Syntax function to set the cursor position at the error.

Anyway, I hope everyone likes the preview. It's a *huge* improvement to the current scripting editor, and it's been fun to work on.

SubAdmin Joined: 18 Nov 2001 Posts: 5182

Depending on how the Spell Checking works might it be possible for it to do some sort of spell check on scripts. Some along the lines of validating known things like commands and functions should be rather easy. However what would quite useful is the addition of checking on setting names and IDs. Not necessarily as you type, but something that would catch some of the typos. Perhaps it might even be possible to integrate it closely enough with the lexer so that it can more cleanly suggest corrections such as "#T- Wron1 status" would only check the ID of status settings for corrections.

Wizard Joined: 14 Aug 2004 Posts: 1269

That preview of the new editor looks really nice, Zugg!

Posted: Tue Mar 27, 2007 6:12 pm

Damn...I've run into a brick wall.

For some reason, the Scintilla editor doesn't work properly within the Delphi frame inheritance. Each setting editor panel (like Alias editor, Trigger editor, etc) is a frame. Each of these frames, in turn, utilize an "Editor Frame" whenever they want an instance of the code editor. This "Editor Frame" has the Scintilla Editor on it.

Whenever I place an instance of the Editor Frame onto something like the Alias frame, it resets a bunch of properties of the Scintilla editor control, instead of inheriting these properties from the Editor Frame itself. This screws up the editor and prevents it from working correctly.

I can't see anything obvious in the source code of the Scintilla editor that would cause this. However, this control *does* store a bunch of properties into the Delphi form (*.DFM) file, including some collections. I know Delphi has some problems with collections in inherited form files, so maybe that's what is corrupting everything.

Looks like I might need to create the editor component at runtime instead of design time, but there are *lots* of properties that I need to figure out how to set at run time to make this work.

I'm very annoyed that this doesn't work. It's just my luck to get this far into the implementation before being hit with something like this. Everything was working fine within normal forms...this Frame stuff in Delphi would be really nice if it actually worked, but it just seems to have so many problems. It might be a Delphi 7 issue...I'll have to see if they improved it in Delphi 2007 or not. But in any case, I need to fix this now in Delphi 7, so I'll keep looking for a work around.

The editor component was really designed to be manipulated during form design time, so I really hate to create it at runtime. Sigh...it's always something. Sure would be nice to work on *MY* code for a change instead of more 3rd party issues.

Posted: Tue Mar 27, 2007 7:39 pm

Good, I found a workaround. Seems that the Editor Frame is fine. I can put it onto a form without any problem. It's just when I put it onto another frame (like Alias Frame) that causes trouble. So, I was able to create the Editor Frame at runtime and place it onto the other frame (Alias Frame in this case). This means that instead of creating the Scintilla editor component at runtime, I'm creating the Editor Frame (which contains the Editor control) at runtime. This allows me to set the extensive properties of the Editor control at design time on the Editor frame as I wished.

So, it's getting better now.

Posted: Tue Mar 27, 2007 9:33 pm

Damn again.

Now the problem is that the Scintilla editor control doesn't properly handle itself when it's window handle gets deleted and recreated. They are trying to save the component's properties and then restore the properties when the window handle is recreated. But when the properties are restored, they get reset to their defaults when using an external lexer.

This causes a problem when the editor is used within a set of page tabs (like the Script/Compiled text tabs for aliases). When you switch tabs, window handles of the previous controls get deleted, then when you switch back to the tab the window handles are recreated.

It doesn't look like the people who wrote the Delphi wrapper properly understood how to handle this situation. Now I'm having to dig deep inside their code to try and figure out if this is fixable.

So now it's all worse again. What a pain. Can't anything be easy anymore?

Posted: Tue Mar 27, 2007 11:21 pm

OK, got it working again. The problem wasn't actually with the wrapper around the Scintilla editor, but was rather a problem with the added component called the LanguageManager which stores all of the styles for each language setting. It looks like they were in the middle of making the LanguageManager a standalone component that would hook into the editor as a proper Delphi component. But it was never finished. Looks like there hasn't been much recent development on this stuff.

In any case, I found the proper hook into the LanguageManager. Turns out it has an event that is fired when the Lexer needs to be updated. Since the LanguageManager wasn't yet a proper component, this event didn't show up in the published properties for the editor. But it is available at runtime, and once I hooked into it, I was able to get the styles to update after switching tabs finally.

I've also added the persistent options for showing line numbers and the code folding. All of this required a little bit of redesign in the Package Editor. The place to the left of the script where there was a toolbar showing a Load and Save icon (and sometimes a New Child icon) is now gone. That area of the editor is now used for the line numbers and code folding gutter. I changed the Tools menu to an "Editor" menu and placed the Spellcheck, Syntax check, Line Numbers, Code folding options there. I also added the "Load script from file" and "Save script to file" commands to that menu. These commands weren't used enough to warrant them taking up valuable editor screen space.

Turns out I also never implemented these Load and Save commands anyway, so that's something else I'll try to finish. Shouldn't be too hard.

My next step is to spend a while working on the zScript Lexer that controls the syntax highlighting and get it working better. That will probably take a couple of days.

Apprentice Joined: 29 Jun 2001 Posts: 198 Location: USA

Nice progress on the editor. I like the way it's looking so far. This should make it easier to edit long or complex scripts.

Posted: Thu Mar 29, 2007 9:41 pm

Well, I'm making progress on the Lexer for the syntax highlighting. Yesterday ended up being a waste though. Yesterday I was expanding upon the current "simple" lexer that I had written. It wasn't going very well. I kept running into so many special cases. zScript isn't a nice and clean formal language, which means the parser and the lexer interact.

For example, normally a lexer is just supposed to convert text to a list of "symbols". Then the parser takes these symbols and checks them to see if they match the parser language, and calls the code generator to generate the compiled code.

However, this can't be done with zScript. Look at the text "#SHOW". Normally, this would convert to a COMMAND symbol. However, consider the script:

Posted: Thu Mar 29, 2007 10:12 pm

Sounds like very solid development progress so far. I'll be really looking forward to the next version.

Wizard Joined: 14 Aug 2004 Posts: 1269

Posted: Fri Mar 30, 2007 5:04 pm

Seb, that's exactly right! I hadn't actually considered that, and even though I'm not making large parser changes these days, it is nice to know that any changes will automatically work properly in the syntax highlighter.

It's been more work dealing with error recovering. I've been able to make the parser try to ignore an error and continue parsing, and when it gets really messed up I've got it moving to the start of the next line and starting over at that position. That allows it to recover from syntax errors pretty well and continue syntax highlighting. Of course, this causes it to find more and more errors since starting over on the next line usually screws up how many {} are needed to close the current block, but since the errors about extra } or missing } also get ignored, it seems to actually work ok.

I wouldn't try to use this to report multiple errors in a single script, but once all of the errors are ignored, it seems to work ok for syntax highlighting.

The other good news is that I think I might have a working "auto-format" routine that will let you "pretty print" your script as in zMUD. This won't happen automatically...it will only happen when you select the Reformat Script command in the menu. But it should really help when converting those huge ugly one-line scripts from zMUD into CMUD.

Wizard Joined: 14 Aug 2004 Posts: 1269

Wahoo!