Duplicate declarations

[benmccann]

@dtbullock and @tonyjayfunk reported in benmccann/xero-java-client#8 that there are duplicate declarations

@dtbullock reports that it's not a single commit which did all the damage, but changes over a series of commits. The disease seems to have started around '4cf8b534c88e0af01f94380dcdc0a46a53433961 (Removed credit note circular deps)' and 'c43766c497c2d42560a147fde485c500d5fcefa3 (Moved payments into the file)'.

Using these schemas in Java now causes numerous errors as shown in benmccann/xero-java-client#8 (comment). Here's a sample:

[ERROR] 'Invoice' is already defined
ligne 68 sur file:/Users/soufian/git/XeroAPI-Schemas/v2.00/Invoice.xsd

ERROR the first definition appears here
ligne 77 sur file:/Users/soufian/git/XeroAPI-Schemas/v2.00/CreditNote.xsd

[ERROR] 'CreditNote' is already defined
ligne 110 sur file:/Users/soufian/git/XeroAPI-Schemas/v2.00/Invoice.xsd

[dtbullock]

It is indeed puzzling why perfectly good <xs:include schemaLocation="other.xsd" /> elements were removed. I suppose it is because Microsoft's schema processor does not handle circular includes. Well, the Java tooling does ... give it a whole bunch of XSD's at once, and it untangles the circular references before proceeding. Give it the same bunch of XSD's at once, with elements defined in multiple files, and it justly complains.

C. M. Sperberg-McQueen (named as an an editor of the XSD 1.1 spec admits in this Stackoverflow post that the XSD spec neither requires nor prohibits that schema processors handle circular includes.

So we have at least two problems here.

1. Divergent behaviour of XSD processors with respect to circular includes when giving them a whole bunch of XSD's at once.

1. Changes to the XSD's in the XeroAPI-Schema project are not automatically tested against multiple XSD schema processors before pushing. I really think this needs to change. You can't be altering the XSD's for the benefit of Microsoft's tooling, and just hope that other schema processors feel the same way.

With respect to problem 1, it seems that the hack is something like 'inline the included content, run the schema processor over each XSD file separately'. However, repeating the declaration of various elements/types across multiple files is asking for an un-maintainable mess. If you really must do this to satisfy Microsoft's tooling, then (after filing the appropriate support cases and bug reports with Microsoft), you should leave the XSD in an un-hacked state, and prepare them specifically for Microsoft's tooling with a set of repeatable XSLT scripts.

With respect to problem 2, at the very least, in the case of Java, the connectifier project should work with the XSD's before they are pushed.

I would also like to see tags referencing particular releases as per http://developer.xero.com/documentation/api/v2-release-notes/

[dtbullock]

Wow, Microsof's xsd.exe tool doesn't cope with much. Give it a bunch of .xsd files and it treats multiple <xs:include/>'s appearing in those files as if we wanted it to re-delcare all the included elements for each <xs:include/> referring to the same file. Give it a single .xsd file and it will only resolve 1 level of <xs:include/>.

However, give it all the declarations in one big global file, and it seems to work a charm (albiet generating a single .cs file).

So xsd.exe is broken for all practical purposes. The main approaches would be:

1. stick with xsd.exe but use a pre-processing step to COMBINE all the XSD's into a single file;
2. use this piece of genius (or this one) to do the XSD processing from a short C# program;
3. snuffle around further in the solutions suggested here including recommending 3rd-party XSD-to-.NET tooling, because Microsoft's quite frankly, sucks.

My gut says that an XSLT to combine all the XSD files into a single file is going to be best, since this solution can be used by any binding toolset if it is needed. However, it does have the minor side-effect of declaring all the generated C# classes in a single .cs file. Which may or may not be desirable for some.

[dtbullock]

I would also just like to point out that it's not fair that I choose to use a Java stack, and wind up diagnosing issues on Microsoft's stack so that my software can run. The universe is not supposed to operate like this!

[nicholasjgreen]

Hi @dtbullock and @benmccann, sorry to hear that you're having difficulty with the schema files. They are intended to be informational and we supply them as-is plus support from the community.

For the record we are not using the Microsoft tool xsd.exe to validate them, we are using xmllint which is a widely used standard (it internally uses libxml). We have recently added Travis CI to verify them, but we don't have 100% cover yet. You can see the details in the .travis.yml file in the repo.

From reading the issue thread on connectifier it does look like the approach to XSD does produce problems for connectifier. I'm not familiar with it, perhaps someone could explain to me how it is using the XSD files? If you examine the travis script, we are comparing xml payloads against specific schema files. The errors you're reporting looks like it is trying to use all of the files together. Is that correct?

[dtbullock]

Hi @nicholasjgreen, the fact that the schema files are not actually used by Xero to do anything is actually a big part of the problem.

OK, it makes sense that you're not using .NET - no .NET user would be able to make use of the XSD in a data-binding sense, as they are, per my investigations last night.

Nor, recently, can Java users. (Note: we just excluded a large portion of the programming population from being able to make use of your XSD's in a profitable way).

In Javaland, to generate a client-side class library from a bunch of mutually-entagled XSD's, we just do this:

xjc -b bindings.xjb -d gen_src/ schemas/

and it gracefully compiles all the .xsd files in scheamas/ to Java source files in gen_src/, with hints from the bindings.xjb (in Java's case, to dis-ambiguate the 'status' attribute from the 'status' element).

This works even in the case when Invoice.xsd uses an <xs:include/> to refer to the elements/types declared in CreditNote.xsd, which in turn uses an <xs:include/> to refer to Invoice.xsd.

However, something (I guess, xmllint, or the way xmllint is being used), clearly does not cope with that arrangement for some reason, and someone at Xero has been systematically removing <xs:include/> for some files (notably CreditNote.xsd and Invoice.xsd along with others), and pasting some of the declarations in CreditNote.xsd directly into Invoice.xsd, and vice-versa. Such that these declarations (for example, <xs:complexType name="Invoice />) gets declared in both CreditNote.xsd and Invoice.xsd. (If you have a look at the commits mentioned earlier with the GIT 'history' tool, you'll get the idea).

Thus, xjc justly complains that the complex type 'Invoice' is declared more than once. (Because it's declared more than once - also creating a maintenance headache for the schemas themsevles).

[dtbullock]

On a somewhat more moving-forward note, I do notice that Travis supports both Java and .NET for testing, so if the community supply tests for these, you'll make sure the schema passes them, right?

[SidneyAllen]

I'm closing out this issue as I've spent a bit of time with JAXB - removed duplicate element declarations - the XSDs are now successfully generating classes.

[benmccann]

Wow, that's great! Thanks!