vocab xsi:type is not preserved going round-trip (possible invalid XML)

[marlontaylor]

I haven't tried you calling the cybox parser separately but since this is a cyboxCommon element, I thought it should be mentioned here. I haven't tested with all vocab xsi:types but I have provided cyboxCommon:Type as an example. Addressing #201 may be helpful in solving this.

input.xml

<stix:STIX_Package 
    xmlns:cyboxCommon="http://cybox.mitre.org/common-2"
    xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"
    xmlns:ttp="http://stix.mitre.org/TTP-1"
    xmlns:stixCommon="http://stix.mitre.org/common-1"
    xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
    xmlns:stix="http://stix.mitre.org/stix-1"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="
    http://cybox.mitre.org/common-2 http://cybox.mitre.org/XMLSchema/common/2.1/cybox_common.xsd
    http://cybox.mitre.org/default_vocabularies-2 http://cybox.mitre.org/XMLSchema/default_vocabularies/2.1/cybox_default_vocabularies.xsd
    http://stix.mitre.org/TTP-1 http://stix.mitre.org/XMLSchema/ttp/1.1.1/ttp.xsd
    http://stix.mitre.org/common-1 http://stix.mitre.org/XMLSchema/common/1.1.1/stix_common.xsd
    http://stix.mitre.org/default_vocabularies-1 http://stix.mitre.org/XMLSchema/default_vocabularies/1.1.1/stix_default_vocabularies.xsd
    http://stix.mitre.org/stix-1 http://stix.mitre.org/XMLSchema/core/1.1.1/stix_core.xsd" 
    id="INSERT_PACKAGE_ID_HERE" 
    version="1.1.1">
    <stix:TTPs>
        <stix:TTP id="INSERT_TTP_ID_HERE" xsi:type='ttp:TTPType' version="1.1.1">
            <ttp:Resources>
                <ttp:Tools>
                    <ttp:Tool>
                        <cyboxCommon:Name>TTP Resource tool</cyboxCommon:Name>
                        <cyboxCommon:Type xsi:type="cyboxVocabs:ToolTypeVocab-1.1">NIDS</cyboxCommon:Type>
                        <cyboxCommon:Type xsi:type="cyboxVocabs:ToolTypeVocab-1.1">SIM</cyboxCommon:Type>
                        <cyboxCommon:Type xsi:type="cyboxVocabs:ToolTypeVocab-1.1">Intelligence Service Platform</cyboxCommon:Type>
                        <cyboxCommon:Description>Tool Description</cyboxCommon:Description>
                        <cyboxCommon:Vendor>Tool Vendor</cyboxCommon:Vendor>
                        <cyboxCommon:Version>Tool Version</cyboxCommon:Version>
                    </ttp:Tool>
                </ttp:Tools>
            </ttp:Resources>
        </stix:TTP>
    </stix:TTPs>
</stix:STIX_Package>

script.py

#!/usr/bin/env python
# Copyright (c) 2014, The MITRE Corporation. All rights reserved.
# See LICENSE.txt for complete terms.

'''
File: ex_01.py

Description: Round-trip example. This script takes a STIX instance document from XML to
a binding object, then to a api object and then to a dictionary. That dictionary is then
converted back into an api object, which is then used to generate an XML document.
'''
import io
from pprint import pprint

from stix.core import STIXPackage


def main():
    fn = 'input.xml'
    stix_package = STIXPackage.from_xml(fn)
    stix_dict = stix_package.to_dict() # parse to dictionary
    #pprint(stix_dict)

    stix_package_two = STIXPackage.from_dict(stix_dict) # create python-stix object from dictionary
    xml = stix_package_two.to_xml() # generate xml from python-stix object
    print(xml)

if __name__ == '__main__':
    main()

output.xml (invalid)

<stix:STIX_Package 
    xmlns:cyboxCommon="http://cybox.mitre.org/common-2"
    xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"
    xmlns:example="http://example.com"
    xmlns:ttp="http://stix.mitre.org/TTP-1"
    xmlns:stixCommon="http://stix.mitre.org/common-1"
    xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
    xmlns:stix="http://stix.mitre.org/stix-1"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="
    http://cybox.mitre.org/common-2 http://cybox.mitre.org/XMLSchema/common/2.1/cybox_common.xsd
    http://cybox.mitre.org/default_vocabularies-2 http://cybox.mitre.org/XMLSchema/default_vocabularies/2.1/cybox_default_vocabularies.xsd
    http://stix.mitre.org/TTP-1 http://stix.mitre.org/XMLSchema/ttp/1.1.1/ttp.xsd
    http://stix.mitre.org/common-1 http://stix.mitre.org/XMLSchema/common/1.1.1/stix_common.xsd
    http://stix.mitre.org/default_vocabularies-1 http://stix.mitre.org/XMLSchema/default_vocabularies/1.1.1/stix_default_vocabularies.xsd
    http://stix.mitre.org/stix-1 http://stix.mitre.org/XMLSchema/core/1.1.1/stix_core.xsd" id="INSERT_PACKAGE_ID_HERE" version="1.1.1">
    <stix:TTPs>
        <stix:TTP id="INSERT_TTP_ID_HERE" xsi:type='ttp:TTPType' version="1.1.1">
            <ttp:Resources>
                <ttp:Tools>
                    <ttp:Tool>
                        <cyboxCommon:Name>TTP Resource tool</cyboxCommon:Name>
                        <cyboxCommon:Type xsi:type="cyboxVocabs:ToolTypeVocab-1.0">NIDS</cyboxCommon:Type>
                        <cyboxCommon:Type xsi:type="cyboxVocabs:ToolTypeVocab-1.0">SIM</cyboxCommon:Type>
                        <cyboxCommon:Type xsi:type="cyboxVocabs:ToolTypeVocab-1.0">Intelligence Service Platform</cyboxCommon:Type>
                        <cyboxCommon:Description>Tool description</cyboxCommon:Description>
                        <cyboxCommon:Vendor>Tool vendor</cyboxCommon:Vendor>
                        <cyboxCommon:Version>Tool version</cyboxCommon:Version>
                    </ttp:Tool>
                </ttp:Tools>
            </ttp:Resources>
        </stix:TTP>
    </stix:TTPs>
</stix:STIX_Package>

[imjonsnooow]

@gtback:

When I first tried this, I was able to reproduce.

I then updated python-stix and python-cybox to master. Now, I can't reproduce this with the given script and input doc using latest cybox and stix source; it seems as though it has already been fixed.

[gtback]

When I ran the script.py on input.xml, the result was valid (according to stix-validator). When I ran the validator on the provided output.xml, it was invalid. The issue is that "Intelligence Service Platform" is not a valid value in cyboxVocabs:ToolTypeVocab-1.0, but it is in cyboxVocabs:ToolTypeVocab-1.0. The xsi:type that is output was updated in 37cfe41.

We don't:

• validate any of the values provided to VocabStrings
• preserve the xsi:type when writing to dictionary/JSON format

You can manually set the xsi_type on an instance of VocabString to control what is output. This is preserved when reading the XML and outputting XML (it's only the dictionary format that causes problems).

[marlontaylor]

I will mention this in #201

I don't think python-cybox should try to validate user strings. I do think python-cybox should pass the xsi:type along into the dictionary/JSON and object structures so developers can decide how to handle them.

The new version handles this case; however, addressing #201 would fix the general case. Since python-cybox only uses the vocab terms it 'knows', developers will have to wait for an update in python-cybox to correctly round-trip (and/or slightly modify) a STIX document which uses the latest schema.

updated output.json

{
   "id":"INSERT_PACKAGE_ID_HERE",
   "ttps":{
      "ttps":[
         {
            "id":"INSERT_TTP_ID_HERE",
            "resources":{
               "tools":[
                  {
                     "description":"Tool Description",
                     "name":"TTP Resource tool",
                     "type":[
                        "NIDS",
                        "SIM",
                        "Intelligence Service Platform"
                     ],
                     "vendor":"Tool Vendor",
                     "version":"Tool Version"
                  }
               ]
            },
            "version":"1.1.1"
         }
      ]
   },
   "version":"1.1.1"
}

updated output.xml

<stix:STIX_Package
        xmlns:STIX_EDITOR="http://STIXDUDE.AWESOME"
        xmlns:cyboxCommon="http://cybox.mitre.org/common-2"
        xmlns:cybox="http://cybox.mitre.org/cybox-2"
        xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"
        xmlns:ttp="http://stix.mitre.org/TTP-1"
        xmlns:stixCommon="http://stix.mitre.org/common-1"
        xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
        xmlns:stix="http://stix.mitre.org/stix-1"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="
        http://cybox.mitre.org/common-2 http://cybox.mitre.org/XMLSchema/common/2.1/cybox_common.xsd
        http://cybox.mitre.org/cybox-2 http://cybox.mitre.org/XMLSchema/core/2.1/cybox_core.xsd
        http://cybox.mitre.org/default_vocabularies-2 http://cybox.mitre.org/XMLSchema/default_vocabularies/2.1/cybox_default_vocabularies.xsd
        http://stix.mitre.org/TTP-1 http://stix.mitre.org/XMLSchema/ttp/1.1.1/ttp.xsd
        http://stix.mitre.org/common-1 http://stix.mitre.org/XMLSchema/common/1.1.1/stix_common.xsd
        http://stix.mitre.org/default_vocabularies-1 http://stix.mitre.org/XMLSchema/default_vocabularies/1.1.1/stix_default_vocabularies.xsd
        http://stix.mitre.org/stix-1 http://stix.mitre.org/XMLSchema/core/1.1.1/stix_core.xsd" id="INSERT_PACKAGE_ID_HERE" version="1.1.1">
    <stix:TTPs>
        <stix:TTP id="INSERT_TTP_ID_HERE" xsi:type='ttp:TTPType' version="1.1.1">
            <ttp:Resources>
                <ttp:Tools>
                    <ttp:Tool>
                        <cyboxCommon:Name>TTP Resource tool</cyboxCommon:Name>
                        <cyboxCommon:Type xsi:type="cyboxVocabs:ToolTypeVocab-1.1">NIDS</cyboxCommon:Type>
                        <cyboxCommon:Type xsi:type="cyboxVocabs:ToolTypeVocab-1.1">SIM</cyboxCommon:Type>
                        <cyboxCommon:Type xsi:type="cyboxVocabs:ToolTypeVocab-1.1">Intelligence Service Platform</cyboxCommon:Type>
                        <cyboxCommon:Description>Tool Description</cyboxCommon:Description>
                        <cyboxCommon:Vendor>Tool Vendor</cyboxCommon:Vendor>
                        <cyboxCommon:Version>Tool Version</cyboxCommon:Version>
                    </ttp:Tool>
                </ttp:Tools>
            </ttp:Resources>
        </stix:TTP>
    </stix:TTPs>
</stix:STIX_Package>