Zugg Software :: View topic

Posted: Mon Jun 07, 2010 7:38 pm

When working on the issue of existing DB variables not being loaded into the new CMUD 3.18d when they contain nested string lists, I've run into a bunch of problems. This issue of backwards compatibility is really causing some huge headaches. There are many days when I wish I didn't have to worry about backwards compatibility since it seems like I spend the majority of my time on it.

The issue is that a table in CMUD is stored internally as a complex data structure. In order to store this data structure to the database, or to export it to XML, the complex data structure needs to be converted to a simple string value. This is commonly called "serializing the data" in other languages.

In the past, CMUD has used a string format for stringlists and database variables like this:

Magician Joined: 02 Feb 2004 Posts: 364 Location: USA

How about an 'export as old format' menu item or some such?

Wizard Joined: 22 Mar 2007 Posts: 2320

Yikes. Quite a kettle of fish there. When I looked at the new format, I did realize that converting back to 3.17 would not work. I wouldn't mind terribly if that continued to be true--for now I'm using 3.18 only for testing, and once it works well enough for my scripts I'll probably switch to it for gaming. But I imagine it would bother a lot of other folks.

I can only think of two other ways to do it. The first violates your policy of not branching your versions: make a new 2.xx version (and maybe a new 3.17 version) that can convert from the 3.18 format. You have valid reasons for your policy, so this isn't viable. The other is to write an external conversion script to go from 3.18 to earlier variable formats. But that's rather awkward.

You could do it with the double format for a while, then set a drop-dead date (well after the public release) after which a next version no longer supports (and actively removes) the duplicate format. But it doesn't reduce your work any (actually adds to it a bit).

Wish I could think of an alternative.

Posted: Mon Jun 07, 2010 8:46 pm

Magician Joined: 02 Feb 2004 Posts: 364 Location: USA

I suppose another option might be to change the extension of package files; When an old one is loaded, it leaves it untouched and creates one with the new extension. This would effectively make a backup without the possibility of it being overwritten with the new format. The older CMUDs would only show the .mud and .pkg extensions so they couldn't be mixed up. The issue being is they wouldn't be able to use any changes they made in the new package (unless, once again, you could export to the old format).

Posted: Mon Jun 07, 2010 11:05 pm

OK, I've thought a lot about this today and here is what I'm going to do:

My #1 priority is to not mess up the majority of CMUD users. This means I need to preserve the old "string format" for tables and lists. I will add the json format as a separate field to both the database and the XML output.

For most users, this will have very little effect. Thus, the least side effect for the largest number of users. The only users that will notice a side effect are those user with very large database or string list variables. For those users, the XML output and the PKG output will grow to contain two copies of the data (the old string format plus the new json format). Saving a package will take slightly longer as the json format is added to the package database.

In reality, the effect is very minimal. Even dumping a huge database variable to both string format and json format just takes milliseconds. I can test this using my bench mark tests. When I run the "makedb" alias with a count of 10000 and then do a #VAR when it's done, the output of #VAR shows both the string format and the json format. And yet the output of #VAR is displayed almost instantly. So the performance decrease is minimal.

As for the size increase of the package file and XML file, disk space is cheap these days and I shouldn't be spending this kind of time trying to reduce a file size by just a few KB. The biggest annoyance will be to users with huge string lists who want to look at their XML output or edit it where they'll need to deal with the two copies of the data.

So, to summarize, this change only has a small effect on a very small number of users. Even though this might be some vocal power-users, I still think it's the right way to go. Dealing with multiple pkg versions, or pkg conversions, or some new options would just be confusing to the majority of users. Power users should be able to deal with this change.

The downside is that this will be yet another big change to the low-level code in CMUD. So this is going to put CMUD 3.19 back into Alpha stage for a few days until people can fully test it again.

What makes this stuff SO HARD is that in addition to the compatibility issues, there are big performance issues. CMUD needs to minimize the number of times the internal table is converted to and from a string format. For example, when you do this:

Enchanter Joined: 10 Oct 2000 Posts: 698

I think understand your concerns. I have a few thoughts/questions about this decision. Presumably, at some point CMUD v#.x will become obsolete and unsupported.
How far into the future will you be required to maintain backward-compatibility as it pertains to this issue? That is, will this data need to be duplicated indefinitely? My initial thought is yes but then, I'm sure that's not true at all.

As I understand it, this is an issue that I would want to view as a hurdle to jump or work through and be done with, rather than an ongoing concern. Once a stable version of CMUD is released with all the bugs ironed out relating to the new json object implementation, will we still need to (or need to be able to) create XML and PKG files that are compatible with earlier versions?

I'm likely over-thinking this but, might it be better or easier to create duplicate files by adding back the export/save code from a previous version, with the idea of eventually eliminating the duplication once you're confident that the new data structures work correctly? Do you need to create bridges extending beyond the beta versions?

Finally, are you able to force older versions of CMUD to ignore the extra data in the files? If so, perhaps just a field could be added pointing to the new-format file for subsequent versions to load.

If what I'm asking makes no sense then just say so. I might be even more clue-deficient than I know. Thanks for a wonderful product.

Wizard Joined: 22 Mar 2007 Posts: 2320

I believe what you are describing is exactly what Zugg has said he would probably be doing.

Posted: Tue Jun 08, 2010 5:00 pm

Yes, older versions will ignore any extra data in the XML output and any extra fields in the database (or extra options in the existing Options freeform field). So that's fine.

As far as supporting old versions...in the past, upgrades were free so I could just tell people they had to upgrade. Now that upgrades are paid, I plan to support the previous major version, at least as far as compatibility and support. The old version is frozen, so no bug fixes to it though.

I'm going with a "combo approach" on this. I'm going to add a preference option to "Output compatible format for string lists and database variables" as suggested above, but I'm going to make this Enabled by default. Power-users who don't want to see the duplicated old string format data in their packages can turn this off, with the understanding that then it cannot be used by players with an older version of CMUD. And TeSSH would have this option turned off because there is no sense adding this kind of compatibility mess to TeSSH.

But CMUD would keep this option enabled by default for probably a year. Then I'd just change the default value of this option to phase it out.

Magician Joined: 02 Feb 2004 Posts: 364 Location: USA

So what's the plan from here, then? Do you need to make those low-level code adjustments that you mentioned earlier, resulting in a new alpha, or is 3.18 still the focus for now? I ask because I want to know whether I should continue fumbling around with 3.18, or wait for the next release.

Posted: Tue Jun 08, 2010 8:46 pm

Yes, I still need to make the low-level changes, so you might as well wait for the next release. It might be stable enough to call a Beta, but it's going to be another "private" beta just for people in this forum. Should be out today or tomorrow.

Magician Joined: 02 Feb 2004 Posts: 364 Location: USA

Just to be clear: This was only an issue with nested databases and lists?

Posted: Wed Jun 09, 2010 4:50 pm

No, it's an issue with *any* string list or database variable. These variables are now stored internally as json tables and saved/exported via the json string format rather than the old zMUD string format. If the new option is enabled, then *both* string formats will be saved, otherwise only the new json string format will be saved.

Example:

Posted: Wed Jun 09, 2010 10:29 pm

Still fixing more low-level bugs in this stuff. I keep finding new bugs (some bugs caused by the new changes, other bugs that were still in 3.18d). Making good progress but still not ready to release a new beta yet. Hopefully tomorrow.