Zugg Software :: View topic - Import/Export and XML format vs zMUD format

Posted: Wed Aug 30, 2006 5:07 pm

This is a new topic to discuss the import/export in CMUD and the XML format vs the older zMUD text format. The original discussion was found in the thread: Why change and change and change???

Another related topic is Rorso's discussion of his own syntax conversion script: XML Fileformat

Please keep your discussion of Rorso's convertor in that script and reserve this thread for discussion of stuff just within CMUD.

I appreciate the work Rorso has done on his script convertor, and something more readable like that is already planned for CMUD. This is another case where some people seem to have gotten annoyed at something very early in the Beta Testing phase, and since I was out of town, I wasn't able to answer the thread (and also because it was posted hijacked to another thread that I wasn't reading anymore).

Remember that CMUD is early in beta testing and not only does it have bugs, but it is not feature complete yet. I hope Rorso hasn't put too much work into something that will be part of CMUD in the future.

The goal of the current XML format was to create a simple database dump in XML format that was "lossless". The old zMUD text format does not preserve all of the information. There are many cases where the zMUD text format fails to import something properly that was exported in text format. For example, multi-line scripts are not supported and are exported in a single-line format. This conversion to single-line format fails if there is a syntax error in the script, and the zMUD syntax error checker isn't very good.

Because of the problems with the zMUD text format, and because of the database structure of settings in CMUD, CMUD uses XML for text import/export. But this XML is not yet intended for human readability (obviously). The current priority is on making it work. I can make it more readable later. Also, I didn't write any special routines for a "CMUD XML" format. I simply used the XML import/export routines of my database system. So, what you see in the XML is the field names from the database, and not something more readable.

Because the XML is lossless and contains *all* of the information from the database, it can be used for copy/paste, export/import, and is used for the shared package library. Creating a more human readable XML format is something planned later once the basic functionality is working.

In addition, I have plans to add another "Copy for Forums" option to CMUD that will use some new bbcode features of these forums to provide a nice way to post color-highlighted scripts. But again, this isn't planned until CMUD is more stable. I'm fixing bugs first, and adding features later.

But both of these methods will still use an readable XML format (more like MUSHClient) rather than the zMUD text format. This is because of the limitations of the zMUD text format. The compatibility between CMUD and zMUD is one-way. You can import/read stuff from zMUD into CMUD, but CMUD has (and will have more) features that zMUD does not. So there is no way to expect a text export from CMUD to work in zMUD.

As an example, take a look at the nice multi-line output format Rorso used in his converter. It makes the CMUD XML file more readable, but this multi-line script format cannot be read reliably using zMUD because of the multi-line bugs in zMUD. If you are lucky you might be able to copy/paste the multi-line script into the zMUD command line, but you probably won't be able to use the Import/Ascii menu. Also, you will find that in zMUD you will have problems importing some stuff that has variable references, depending upon some of your parsing settings.

In other words, don't expect a zMUD user to be able to directly use this kind of text export from CMUD. At least unless you are willing to answer a bunch of support questions from zMUD users who can't get it to work.

As more features are added to CMUD, such as enhanced gauges, or the zApp scriptable forms, then you will run into even more stuff that zMUD doesn't understand.

So, let's not confuse these two separate issues. There is the issue of making the XML text format in CMUD readable, and there is a separate issue of trying to export stuff from CMUD back into zMUD. The readability is something that will get a lot of attention. The issue of exporting stuff from CMUD to zMUD isn't going to be directly supported.

When you are thinking about this stuff and discussing it, think about the #BUTTON command as a good example. Can anyone really remember what the 14th argument of the #BUTTON command does? (no fair cheating and looking it up in the help file).

Let's face it...for stuff like #BUTTON, the zMUD text format is horrible! It's really better to use a more readable XML format (again, MUSHclient is a good example). But just because CMUD doesn't yet have a more readable XML format doesn't mean that it's not coming. You just need to have more patience and remember that this is beta testing.

Wizard Joined: 03 Mar 2001 Posts: 1127 Location: London

Another related topic
http://forums.zuggsoft.com/phpbb/viewtopic.php?t=24295

Posted: Wed Aug 30, 2006 5:54 pm

from change change change:

Posted: Wed Aug 30, 2006 6:28 pm

Er I i hate to nit pick but

Wizard Joined: 14 Oct 2000 Posts: 1368

Actually I think the main problem here is zScript itself. We blame xml, we blame the plain text file but isn't it zScript itself that might have issues here? It is zScript that ultimately has multi-line problems which seems to be why you use XML to begin with.

While the xml format is nice it does eventually still save the script as pretty much standard text. The problem has just been postponed one level :P. What if the user would like to create an alias that automatically creates a button? E.g in zMUD:
#CLASS {buttons}
#ALIAS test {#BUTTON 1 {push!} {} {} {} {} {} {SMILEY} {Size} {84} {42} {} {} {} {121} {} {} {} "" {Explore|Inset|Top} {} {}}
#CLASS 0

As far as I know the goal of cMUD isn't to be backwards compatible with zMUD, but to fix many of the issues zMUD has. Adding better multiline support to the zScript 2.0 and clean up the syntax could be interesting.

Posted: Wed Aug 30, 2006 7:03 pm

Tech, got ya. I'm just full of mis-readings lately :(
As for a list of requirements....errr...I just wanna be able to print my scripts in a readable format much like it is in zmud.
Of course, I know, that as Rorso just said, that there are issues with zscript, and I'll assume one of those issues is the lovely way that #func turns into #var in an export. Another issue is NOT at all related to XML, but in that if I cut and paste a #func into another folder other than the main folder, the #func stops working (This again is still in zmud and not related to XML but IS related to Rorso's zscript input)

Along with being able to see the script in a Class View type window, but that's a whole other topic that I think is being talked about on another post of mine.

Tech, I really haven't posted much about the XML stuff, and I'm known to ramble on and on and on and....errr..oh, yeah, on. But if ya want, feel free to take a look at what I HAVE said and see if you can make heads or tails of it and compile something of what I mean.. Cause heck, far as I know my only main thing is being able to sit on the train, review my scripts, edit them, and go right back in and be able to change them..

Posted: Wed Aug 30, 2006 7:37 pm

Rorso: Yes, you are correct. The problem is really in zScript. zScript was never originally intended for import/export. The idea of the text export was to simulate what you would enter into the command line. But as we know, some things in the command line in zMUD get expanded *before* they get stored, which then causes export/import problems.

Things like the #BUTTON command were a kludge to provide some way to export/import button settings with all of their options. I doubt *anyone* types a full #button command on the command line themselves. I'd guess that >90% of the #button commands are from an export. It's just a lot easier to set button options in the graphical interface of the settings editor than it is to edit the #BUTTON zScript text.

But, because CMUD needs compatibility with zMUD, it still supports the #BUTTON command line syntax. I didn't want to remove something that important from CMUD. But in the future, you will see more and more stuff move towards XML-like syntax, more like zApp. For example, look at zApp and you will see syntax for creating a button within the ZML file. That's more like where CMUD is heading. But obviously the initial focus in CMUD is more on zMUD compatibility.

GURU Joined: 10 Oct 2000 Posts: 4379 Location: USA

Posted: Thu Aug 31, 2006 8:39 am

It would be really awesome aswell to be able to do that for mud output text too....

Say like Ctrl-B for "Copy to Clipboard with BB colour coding" instead of Ctrl-C....

So you can easily copy/paste things with the proper colours ?

Posted: Thu Aug 31, 2006 4:23 pm

Zugg, that would be GREAT! I'd say pretty print for it would be the best! Tho since your talking about coloring it, you prolly already have that in mind.

Namsar, that would be sweet except for those muds that use non-standard colors. Would have to have the BB colour code make a guess if that greenish yellow is to be yellow or green. But it would solve some of the color code trigger questions I guess.

Posted: Thu Aug 31, 2006 5:26 pm

It's possible that you will be able to do this with MUD output indirectly. The Export/Paste to bbcode is something that I'm adding to the zApp Memo control, which is what CMUD uses for the settings editor scripts. This same component is also used in the Editor window in CMUD. So, in theory, you could copy MUD output to the clipboard, then paste it into the editor window. Then use the Copy as BBcode (or whatever it will be called) in the editor and then paste it into the forums. So, just a two step process using the editor window.

And yes, if a MUD uses a wierd color scheme, then you will need to edit the post to "fix" the colors. I can't do anything about that.

Magician Joined: 13 Oct 2000 Posts: 463

Zugg, have you heard of/looked at YAML? It's a serialization format (similar to how you're using XML) that is designed to be easier to read by a human. It seems like a better fit than XML for your needs, though it might not have direct library support in your SQL library.

http://en.wikipedia.org/wiki/YAML
http://www.yaml.org/

In YAML, the example posted from another thread would look like this:

- record:
id: 1048708
kind: 0
parent: 2097151
pkgid: 1048708
enabled: -1
idname: ToTrain
options: ~
userval: ~
val: ~
comment: ~
name: "{*}"
subkind: 0
valkind: 0
opt: 256
owner: 2097151
userint: 0
priority: 0
flag: 128
state: 0
memref: 45037888
- record:
id: ...

Even better for readability would be to omit transient and default fields... for example, if the priority defaults to 0, omit it from the data stream. If a field is blank, like comments, omit it too. The memref field is likely transient, changing on every execution of the application... so it can be omitted as well. Other fields are not portable... for example, owner: 2097151 would not work on another person's copy of CMUD, so it can be removed also, or at least moved outside to the package spec rather than on each record.

This leaves us with a far more readable:

- record:
id: 1048708
parent: 2097151
enabled: -1
idname: ToTrain
name: "{*}"
opt: 256
flag: 128
- record:
id: ...

Compare this to the original XML, and I think you'll agree that YAML is far easier on the eyes and the keyboard when manually editing:

<record id="1048708">
<id>1048708</id>
<kind>0</kind>
<parent>2097151</parent>
<pkgid>1048708</pkgid>
<enabled>-1</enabled>
<idname>ToTrain</idname>
<options/>
<userval/>
<val/>
<comment/>
<name>{*}</name>
<subkind>0</subkind>
<valkind>0</valkind>
<opt>256</opt>
<owner>2097151</owner>
<userint>0</userint>
<priority>0</priority>
<flag>128</flag>
<state>0</state>
<memref>45037888</memref>
</record>

Wizard Joined: 14 Oct 2000 Posts: 1368

Actually yaml seems to be more or less the same as the usual .ini file. What people find tough to read in the current xml format is probably the constants. E.g flags, the kind of setting etc.

Posted: Mon Sep 04, 2006 4:47 am

Don't worry, I think I can improve the XML output well enough to please most people. You'll just have to wait for this and then give me feedback when I have something ready.

I'd really like to stick with XML because it's a known standard, and many people are already familiar with it and with similar HTML. Also, CMUD already has an XML parser in it, which is used in zApp already. And I hate to add more parsers just for something like this.

And honestly, I'm still not sure the YAML is much better. What I plan to do involves changing the name of the fields to something more understandable, and getting rid of the obscure numeric options and various things like "memref" and "parent" and stuff that doesn't really need to be in the output.

Remember, the current output is just a "dumb" database record dump in XML. There are lots of ways to improve that. But since I was (and still am) focused on functionality and not useability, I've had higher priority things to work on.

Wizard Joined: 25 Mar 2003 Posts: 1113 Location: USA

I've done some truly fun things with XML, so I was glad to hear that the format would be changing to XML. I'm definitely looking forward to more human-readable names, but having this in XML opens new possibilities.

I would like to make a small request, if possible: use slightly more general XPath expressions to parse the XML on import. Instead of forcing absolute paths (/Root/Triggers/Trigger), allow for grouping in our own tags around it and the XPath can find the settings no matter what group they're in (//Trigger). The groups wouldn't effect classes in any way, necessarily, although I would love to see triggers grouped by class tree, too.

In my experience, the most natural way to do the XML for this is to setup the settings as children of the class/module/package they belong to.

Posted: Mon Sep 04, 2006 2:25 pm

Zugg,

Trying to make the xml dump more human readable just feels wrong to me. My gut feeling is telling me that you have a potential of painting yourself into a corner, when it comes to future upgrades to the schema. The purpose of xml is for integration between disparate systems. Here's what I think will be a better long term solution to this:

1) Create a xsl stylesheet that will be the official stylesheet to viewing this file in a human readable form. This will also allow others to create their own stylesheets to conform to their idea of that is right/correct. The best way to make this painless, is to have a setting in CMUD that allows power users to change the stylesheet they want to use. I imagine that this will be a textbox where the file name can be entered. Then CMUD writes this directive into the header.

2) Allow a third party to create an export editing/viewing program. This might sound crazy at first, but think "ecosystem" here. I believe that extatic power users need to have an outlet for their creativity. I know that you will have the zApp form plugins ( *drool* ) in the future. A good way to create a vibrant ecosystem is to have a "CMUD Approved Utiltity" process. This is a for pay process, which should significantly reduce the amount of people that are not serious.

I think that these two options will reduce the amount of coding that you spend on this part of CMUD. This also empowers to the users to help themselves.

Posted: Mon Sep 04, 2006 5:18 pm

Larkin, that's pretty much the kind of syntax I'm looking at. You kind of lost me with the XPath stuff (I'm no XML expert), but your example is pretty close to what I'm considering.

BortaS: making the XML more readable still allows the things you mentioned. But as you have seen from other posts about this, there are several people who really want a more readable syntax, and making them deal with style sheets and external editors just doesn't sound like the best solution.

I'd like people to think that the settings editor within CMUD itself will be the most desireable "editor" for settings. And supporting a more readable format like Larkin has shown an example of isn't very hard. It actually helps detach the XML file from the internal database format. With the current format, a change in a database field changes the XML import/export, which isn't desireable.

And I really don't want to have to mess with style sheets.

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

I'm not overly fussed on the whole xml thing, I figure most of my stuff will be shared via a private package library for the MUD I run, however I like explaining my scripts on the noteboards which means as well as the 'paste to bbcode' it would be good if there was a 'paste to plaintext' and/or 'paste to <user defined format>' ... I could set up the editor with preferences (eg '@R' is red, @B is blue, etc) and then send it to the MUD?

And the people on the MUD could paste it into the command line and have it then put the settings in?

Posted: Mon Sep 04, 2006 11:59 pm

No Rainchild, that's the whole point and problem: there is no way to simply enter this stuff into the command line. The command line syntax just doesn't handle all of the needed options, which is the whole problem with the zMUD text import/export in the first place. Sure, *SIMPLE* stuff like Aliases, Var, etc work fine. But more complex script elements, like Buttons, Gauges, and the planned zApp Scriptable Forms cannot be easily specified with the zScript syntax.

The whole point of the XML format is to implement a loss-less Copy/Paste text format. You would Copy a setting to the clipboard, which puts the XML dump of the selected settings into the clipboard. You then paste this XML text into the forums. Someone who wants to try what you posted simply copies the XML from the forums into their clipboard and then uses the Paste option in the settings editor. The command line is NOT USED.

The Copy to BBcode just adds the needed BBcode for color syntax highlighting. But when you copy this text in the forums to the clipboard using your web browser, it just copies the plaintext version, which is what CMUD uses to paste the XML into the settings editor (the web browser also makes an HTML copy with the color codes, but CMUD ignores that and just uses the plain text in the clipboard).

The fact that people are used to copy/pasting scripts into the command line is what we need to change. The command line is for sending stuff to the MUD and NOT for entering scripts. zMUD's Import/Text option was a kludged way to import a text script. It placed the command line into a special mode that turned off some parsing. These are the kind of kludges I'm trying to get rid of.

Here is an easy example to understand this. Imagine entering the following into the command line:

Wizard Joined: 14 Aug 2004 Posts: 1269

Wizard Joined: 10 Oct 2000 Posts: 1551 Location: Australia

Ok, I wasn't following that you would be able to paste directly into the settings editor - that'll work too. They'll still be able to copy and paste from the MUD, just paste into the editor not the command line - works for me :)

Posted: Tue Sep 05, 2006 4:51 am

I think the real trick is working on the XML parser to allow a bit looser parsing. For example, I don't want to require a single "root node". I'd like it to allow something like this:

Wizard Joined: 14 Oct 2000 Posts: 1368

Posted: Tue Sep 05, 2006 2:32 pm

Wizard Joined: 25 Mar 2003 Posts: 1113 Location: USA

bortaS is correct. Any properly formatted XML file requires a single root node. You could use some standard/dummy root node name every time, and your XPath expression wouldn't even care what that node is if you use the proper syntax. W3Schools has good tutorials on XPath and lots of other stuff. Having a properly formatted XML file is very important to the power users like myself because it allows us to do some other really cool things with it using XSLT or just to edit the files in our favorite XML editor outside zMUD.

I really like the idea of having stylesheets that the power users can tweak, too, but I understand that's going a bit far in allowing customization of the XML format. It's probably one of those features that only 1-2% of the users would actually touch, so maybe not worth implementing, at least not right away.

XML is an extremely flexible format for storing data and allows easy manipulation once you learn the XSLT/XPath syntax. However, I've seen some very awful uses of XML and once you start down the wrong path it's hard to backtrack later. I'm so glad (relieved even) to see you're considering setting it up similar to the example I gave! :)