Forwards and Backwards Compatibility

The Dev and Release branches have brought about a complication that in retrospect should not have been unexpected.

World files and macros created in the dev branch are pretty much incompatible with the release branch.

This is not good. It fragments the potential users of a macro or TMD file; Dev Branch users are unable to share any of their TMDs with normal users. We’ve already been seeing these situations crop up on the forums; what’s worse is that WM doesn’t give very good error messages when this happens, so the user doesn’t know what’s wrong — just that WM is having some kind of error for no good reason. frustrating.

This is the crowning annoyance of a longstanding problem — WM’s file-format is quite brittle. Missing a device? Can’t load the world. Saved from a new version? Can’t load it. And sometimes “can’t load it” actually means “stomp on memory and crash the program.”

I’ve been working on a solution to this problem this week,, and I’m happy to say that the solution is, in fact, already complete.

Dance of the Tags (Implementation Details)

The easiest (and most logical) way to solve this problem is to move from a versioned binary blob file to a tag-based file format, such as used by tiff or xml in text. The new file format is still a binary format, but adopts a tag-based structure that will be a bit familiar to anyone who has done html or xml document model work. Key to WM’s new file IO tags are that each tag is a hierarchical container; it contains a size field allowing us to skip to the end of the tag if we can’t parse it. WM’s old file format had absolutely no way to recover from this kind of lack of knowledge.

Anyways. What does this change buy us?

The Dev and Release builds will be able to open each others’ files and macros

Obviously any specific new features won’t work, but files that don’t use any new features will be 100% compatible with the old version.

‘Old’ World Machine versions will be able to open future versions successfully

Once this is in place, even if you are using an older version that can’t understand some new functionality you will be able to load the file. Of course, you’ll receive a warning that you’re missing some things…

Missing device plugins will never cause the world to fail to load

Instead of failing to load the world file or just plain crashing, a placeholder “Unknown Device” will be created and wired into the network.

Of course, it won’t actually do anything and your world probably won’t work without the missing devices. But…

You may even be able to ROUND TRIP between different versions with most information intact

Devices and maybe features loaded that are not known to the current version will be kept and re-saved to disk; so that you should be able to successfully:

1. Open a “too new” file that uses some new features. You can’t use em, but you are able to make some other changes anyways and…
2. Save the changes to disk from your older edition. Then
3. Open your changed file in the newer version … and have all of its original “too new to work” data still there!

That last point is really quite rare — and hard to pull off! Especially when devices, etc change their parameters and functionality. I can’t guarantee that this will always be possible for all features, but at least with current anticipated changes it’s looking like it will be.

The new file format will roll out to the Dev Build first for testing; once it’s been tested for a bit, after that it will also be incorporated into the Release build, at which point all of the above will become possible.

PS:

There’s lots of other exciting new changes afoot.

Some of them are in R&D stage so I’m not writing much about them for now, others (like rolling out localspace support to the dev build) will be coming in the next month!