new project file format

[garfieldnate]

Follow-on to this comment: #34 (comment)

Requirements:

• Combines .dm, .vsa and comment.dm into one file
  • Fixes issues where the different files get out of sync
• Properly handle escaping of special characters (such as | inside of strings)
• Does not use semantic whitespace (so doesn't care if the file has Windows or *nix newlines)
• In practice does always choose one newline style (probably \n)
  • So that editor programs on different OS's don't keep re-writing the entire file
  • I know git enables transforming the newlines in the working copy for you, but that requires user config, meaning most probably won't do it
• Uses strings for IDs instead of integers (supporting human-readable names if a human wants to set them)
• Uses hash or random strings for new IDs
  • instead of anything sequential, which might cause two collaborators to use the same ID for a different thing
  • hash where possible so that file generation is as deterministic as possible
• Uses a ubiquitous text format, not specific to Java (so not Java preferences)
  • I'd prefer JSONC (JSON + comments and trailing commas), which is pretty human-friendly as far as text formats go
• Writes first to a temp file and then renames to the real file name
  • instead of serializing straight to the final file, which will bork the file if an Exception occurs in the middle of serialization
• Serialization should be completely deterministic
  • for example, unordered keys should be sorted during serialization
  • prevents the case of, "all I did was open the project in V-S and now git says I've made changes!"
• Format should be documented
  • advanced users can edit it if they wish
  • if the file is ever corrupted somehow, gives users an escape hatch where they can try to repair it themselves
  • helps developers add support in other libraries or editors
• Handles Unicode values
  • encode in UTF-8 (JSON specifies this already)
  • escaping Unicode chars would make the values unreadable to humans, so no escaping required!
• Not unnecessarily difficult to merge in Git
  • For example, we should consider what level of granularity should be given in each line: one whole DM node, or just one attribute? This will be the level that the user will be asked to resolve merge conflicts at. We definitely should not print the whole document on one line.
• Lines that need merging in git (starting with <<<< or >>>>>>) should be detected and the user should be alerted. For example, newlines inside of strings should be escaped so that a <<<< or >>>> at the beginning of a line cannot possibly be a legal value inside of a string in the datamap.

I suppose this would become the version 6 format. This would fix and prevent a lot of the project stability issues that we have.

Feel free to add more to the requirements list!

[PLatCIC]

I think this change is very important. Sometimes things have gotten mixed up or trashed in these .dm and .vsa files, and it's not humanly possible to make any sense of them. Having them in a format that's readable would be a great improvement.

[analog-cbarber]

I definitely recommend using a standard human-readable data language such as JSON. Non-standard variants might make sense but would then also limit present problems if there are no tools for someone's preferred language.

Ideally, you would should have a format that is git mergable. If two people check in changes from different branches, a developer should be able to merge the changes by looking at the diffs.

[garfieldnate]

Ideally, you would should have a format that is git mergable.

Thanks! I added this to the list. In the case of JSON, I think this means that V-S should save a pretty-printed document so that individual changes show up on different lines, which are easy to merge.

[garfieldnate]

Here are some requirements related to the file handling, rather than the format, as well:

• File should always be saved atomically. No partial updates should be possible. This can be accomplished by writing to a temp file and then renaming to the final filename (prevents partial writes on a network drive when connectivity is lost).
• V-S should listen for file changes and alert user, offering to reload and checking for errors. This would be important when, for example, a git pull is done, and there may or may not be merge conflicts.