Skip to content

Latest commit

 

History

History
610 lines (485 loc) · 26.7 KB

Developers.md

File metadata and controls

610 lines (485 loc) · 26.7 KB
Raw

Introduction

EDDN is a zeromq service which allows players of the game Elite Dangerous, published by Frontier Developments, to upload game data so that interested listeners can receive a copy.

EDDN accepts HTTP POST uploads in a defined format representing this game data and then passes it on to any interested listeners.

Sources

There are two sources of game data, both provided by the publisher of the game, Frontier Developments. They are both explicitly approved for use by third-party software.

Journal Files

On the PC version of the game, "Journal files" are written during any game session. These are in newline-delimited JSON format, with each line representing a single JSON object. Frontier Developments publishes documentation for the various events in their Player Tools & API Discussions forum.

In general the documentation is made available in a file named something like:

Journal_Manual-v<version>

as both a MicroSoft word .doc file, or a .pdf file. Historically the use of _ versus - in those filenames has varied.

Consult the latest of these for documentation on individual events.
However, be aware that sometimes the documentation is in error, possibly due to not having been updated after a game client change.

Companion API (CAPI) data

Frontier Developments provides an API to retrieve certain game data, even without the game running. Historically this was for use by its short-lived iOS "Companion" app, and was only intended to be used by that app. There was no public documentation, or even admission of its existence.

Eventually, after some enterprising players had snooped the connections and figured out the login method and endpoints, Frontier Developments allowed general use of this .

Originally the API authentication required being supplied with the email and password as used to login to the game (but at least this was over HTTPS).

In late 2018 Frontier switched the authentication to using an oAuth2 flow, meaning players no longer need to supply their email and password to third-party sites and clients.

As of October 2021 there has still never been any official documentation about the available endpoints and how they work. There is some third-party documentation by Athanasius.

It is not recommended to use CAPI data as the source as it's fraught with additional issues. EDMarketConnector does so in order to facilitate obtaining station data without the player needing to, e.g. open the commodities screen.

Detecting CAPI data lag

When using the Companion API please be aware that the server that supplies this data sometimes lags behind the game - usually by a few seconds, sometimes by minutes. You MUST check in the data from the CAPI that the Cmdr is docked, and that the station and system names match those reported from the Journal before using the data for the commodity, outfitting and shipyard Schemas:

  1. Retrieve the commander data from the /profile CAPI endpoint.
  2. Check that commander['docked'] is true. If not, abort.
  3. Retrieve the data from the /market and /shipyard CAPI endpoints.
  4. Compare the system and station name from the CAPI market data, ["lastStarport"]["name"] and ["lastSystem"]["name"], to that from the last Docked or Location journal event. If either does not match then you MUST abort. This likely indicates that the CAPI data is lagging behind the game client state and thus should not be used.

Uploading messages

Send only live data to the live Schemas

You MUST NOT send information from any non-live (e.g. alpha or beta) version of the game to the main Schemas on this URL.

You MAY send such to this URL so long as you append /test to the $schemaRef value, e.g.

"$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2/test",

You MUST also utilise these test forms of the Schemas whenever you are testing your EDDN-handling code, be that new code or changes to existing code.

As well as the Live service there are also beta and dev endpoints which might be available from time to time as necessary, e.g. for testing new Schemas or changes to existing ones. Ask on the #eddn channel of the EDCD Discord (check at the bottom for the invite link).

Alternatively you could attempt running your own test instance of EDDN.

Sending data

Messages sent to EDDN MUST:

  • Use the URL: https://eddn.edcd.io:4430/upload/
    • Note the non-standard port 4430.
    • Yes, the trailing / is required.
    • Note the use of TLS-encrypted HTTPS. A plain HTTP request will elicit a 400 Bad Request response.
  • Use the HTTP 1.1 protocol. HTTP/2 is not supported at this time.
  • Use a POST request, with the body containing the EDDN message. No query parameters in the URL are supported or necessary.

The body of an EDDN message is a JSON object in UTF-8 encoding. If you do not compress this body then you MUST set a Content-Type header of applicaton/json.

You MAY use gzip compression on the body of the message, but it is not required. If you do compress the body then you *MUST send a Content-Encoding header of gzip.

Due to issues when messages are compressed, form-encoded data is NO LONGER SUPPORTED as of 2022-06-16.

You should be prepared to handle all scenarios where sending of a message fails:

  1. Connection refused.
  2. Connection timed out.
  3. Other possible responses as documented in Server responses.

Carefully consider whether you should queue a 'failed' message for later retry. In particular, you should ensure that one 'bad' message does not block other messages from being successfully sent.

You MUST wait some reasonable time (minimum 1 minute) before retrying any failed message.

You MUST NOT retry any message that received a HTTP 400 or 426 code. An exception can be made if, and only if, you have manually verified that you have fixed the issues with it (i.e. updated the Schema/version to a currently supported one and adjusted the data to fit that Schema/version).

You MAY retry a message that initially received a 413 response (in the hopes that the EDDN service admins decided to increase the maximum allowed request size), but should not do so too quickly or in perpetuity.

In general:

  • No data is better than bad data.
  • Delayed good data is better than degrading the EDDN service for others.

Format of uploaded messages

Each message is a JSON object in UTF-8 encoding with contents that comply with the relevant Schema. There is an outline in the general EDDN Message README.

For example, a shipyard message, version 2, might look like:

{
  "$schemaRef": "https://eddn.edcd.io/schemas/shipyard/2",
  "header": {
    "uploaderID": "Bill",
    "gameversion": "4.0.0.1451", 
    "gamebuild": "r286916/r0 ",
    "softwareName": "My excellent app",
    "softwareVersion": "0.0.1"
  },
  "message": {
    "systemName": "Munfayl",
    "stationName": "Samson",
    "marketId": 128023552,
    "horizons": true,
    "odyssey": true,  
    "timestamp": "2022-09-27T06:39:43Z",
    "ships": [
      "anaconda",
      "dolphin",
      "eagle",
      "ferdelance",
      "hauler",
      "krait_light",
      "krait_mkii",
      "mamba",
      "python",
      "sidewinder"
    ]
  }
}

Contents of header

You MUST send the header component of the message.

uploaderID

The EDDN Relay will obfuscate the uploaderID value to prevent long-term tracking of individual players. Please DO send a sensible uploaderID value, preferably simply the relevant in-game Commander name.

softwareName

You MUST set a unique, and self-consistent, value of softwareName so that you can be easily identified should any issues be found with the messages you send.

softwareVersion

You MUST set a pertinent value for softwareVersion. We would recommend using Semantic Versionining in your project.

Listeners MAY make decisions on whether to accept data, or to treat it differently, based on this. As such you MUST increment your version number if you make any changes to the content of messages your software sends to EDDN.

gameversions and gamebuild

To ensure that Listeners can make decisions on how to handle data based on the client and feature set it came from there are two mandatory fields in the headers of EDDN messages. NB: Initially the schemas do not actually make these mandatory, but all Senders MUST make every effort to include them ASAP.

Where present in the data source the gameversion value MUST come from the field of that name in the data source, i.e. from either Fileheader or LoadGame as outlined below.

  1. If you are using Journal files directly then you MUST use the value from theFileheader event. Values from LoadGame are also acceptable, but there are some events before that which might be relevant to EDDN messages, especially as the ordering of 'startup'/login events can change with game patches.

  2. If you are using the CAPI /journal endpoint to retrieve and process Journal events then:

    1. You will not have Fileheader available.
    2. If the field is present in the LoadGame event, as in 4.0 clients, use its value.
    3. If LoadGame does not have the field, as with 3.8 Horizons clients (up to at least 3.8.0.1400), you SHOULD set the value to "CAPI-Legacy-journal". If, for reasons of code architecture, you are unable to determine that data was CAPI-sourced then you MAY set it to "" instead, but this is only a fallback and you should endeavour to send a more useful value.

  3. If you are sourcing data from other CAPI endpoints, i.e. for commodity, shipyard, outfitting or fcmaterials_capi messages, then you SHOULD set the values appropriately as per the CAPI host and endpoint the data came from. You MUST NOT use the value from any Journal you are also reading for the user. The general form is "CAPI-(Live|Legacy)-<endpoint>".

    1. If it's a commodity message from the Live CAPI host, then use "CAPI-Live-market". If it was from the Legacy CAPI host then use "CAPI-Legacy-market".
    2. If it's a shipyard message, then use "CAPI-Live-shipyard" or "CAPI-Legacy-shipyard".
    3. If it's an oufitting message, then also use "CAPI-Live-shipyard" or "CAPI-Legacy-shipyard".
    4. If it's an fcmaterials_capi message, then use "CAPI-Live-market" or "CAPI-Legacy-market", as the data comes from that endpoint.

    Again, if your code architecture doesn't allow for signalling that the data source was CAPI, then you MAY set it to "" instead, but this is only a fallback and you should endeavour to send a more useful value.

For gamebuild you MUST use the value of the build field in the data source, following the same logic as for gameversion above, else send as "". An alternative to the latter is to mirror the gameversion value, but only where that has not come from a Journal value, i.e. CAPI-... strings. Do NOT strip any leading or trailing white space, pass it through as-is.

For emphasis, if you cannot set a data-source value, or an appropriate "CAPI-..." value then you MUST still send the field with an empty string value.

Examples of valid values:

Data Source Data Type Game Galaxy gameversion gamebuild
Journal Journal Live "4.0.0.1475" "r289563/r0 "
Journal Journal Legacy "3.8.0.1400" "r289583/r0 "
CAPI /journal Live1 "4.0.0.1475" "r289563/r0 "
CAPI /journal Legacy2 "CAPI-Legacy-journal" ""
CAPI /journal Legacy2 "CAPI-Legacy-journal" "CAPI-Legacy-journal"
CAPI /market3 Live "CAPI-Live-market" ""
CAPI /market3 Live "CAPI-Live-market" "CAPI-Live-market"
CAPI /market3 Legacy "CAPI-Legacy-market" ""
CAPI /market3 Legacy "CAPI-Legacy-market" "CAPI-Legacy-market"

Contents of message

Every message MUST comply with the Schema its $schemaRef value cites. Each Schema file should have a matching <schema>-README.md file in the project root schemas/ directory . Always consult this so that you're aware of any Schema-specific requirements.

The Schema file, <schema>-v<version>.json, is considered the authority on the format of messages for that Schema. If anything in the accompanying documentation is in disagreement with this then please open an issue on GitHub so that we can investigate and correct, or clarify, as necessary.

Apart from short time windows during deployment of a new version the live EDDN service should always be using the Schemas as present in the live branch. So, be sure you're checking the live versions and not, e.g. those in the master or other branches.

Please consult the general README for Schemas for more detailed information about a message's content.

EDDN is intended to transport generic data not specific to any particular Cmdr and to reflect only the data that every player would see in-game in station services or the local map. To that end:

  1. Uploading applications MUST ensure that messages do not contain any Cmdr-specific data (other than "uploaderID", the "horizons" flag, and the "odyssey" flag).
  2. Uploading applications MUST remove any data where the name of the relevant key has a _Localised suffix.

The individual Schemas will instruct you on various elisions (removals) to be made to comply with this.

Because the first versions of some Schemas were defined when only the CAPI data was available, before Journal files existed, many of the key names chosen in the Schemas are based on the equivalent in CAPI data, not Journal events. This means you MUST rename many of the keys from Journal events to match the Schemas. Consult the relevant Schema for details.

Some of these requirements are also enforced by the Schemas, and some things the Schemas enforce might not be explicitly called out in documentation. So, do check what you're sending against the relevant Schema(s) when making any changes to your code.

It is also advisable to Watch this repository on GitHub so that you are aware of any changes to Schemas.

horizons and odyssey flags

Where the Schema allows for them, horizons and odyssey keys MUST be added with appropriate boolean values. null is not allowed in the values, so if you cannot determine a value do not include that key at all.

To emphasise that, in the case where there is no Odyssey boolean in the LoadGame event DO NOT INCLUDE IT IN THE EDDN MESSAGE. No, not with a false value. DO NOT INCLUDE IT.

The only source of these is the LoadGame event from journals. It's present both in the PC local files and the CAPI journal data. If you're composing a shipyard or outfitting message from CAPI data then it is possible to synthesise the horizons flag. For now consult the function capi_is_horizons() in EDMarketConnector:plugins/eddn.py for a method to achieve this.

As of 2022-09-27 the following was observed for the LoadGame events as present in CAPI-sourced Journal files (which were confirmed to match the PC-local files for these events):

  • PC Odyssey Client, game version 4.0.0.1450:

    { "timestamp":"2022-09-27T09:47:35Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, "Odyssey":true, ...

  • PC Horizons 4.0 Client, game version 4.0.0.1450:

    { "timestamp":"2022-09-27T11:25:45Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, "Odyssey":false, ...

  • PC Horizons Client, game version 3.8.0.407, no Odyssey key was present:

    { "timestamp":"2022-09-27T11:28:53Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":true, ...

  • PC 'base' Client, game version 3.8.0.407, no Odyssey key was present:

    { "timestamp":"2022-09-27T11:31:32Z", "event":"LoadGame", "FID":"<elided>", "Commander":"<elided>", "Horizons":false, ...

Do not attempt to use the value(s) from a Fileheader event as the semantics are different. With clients 3.8.0.407 and 4.0.0.1450 the following was observed:

Game Client Fileheader LoadGame
Base "Odyssey":false "Horizons":false
Horizons 3.8 "Odyssey":false "Horizons":true
Horizons 4.0 "Odyssey":true "Horizons":true, "Odyssey":false
Odyssey "Odyssey":true "Horizons":true, "Odyssey":true

NB: The 'Base' client appears to simply be the Horizons client with any Horizons-only features disabled.

  • In the Fileheader event the Odyssey flag is indicating whether it's a 4.0 game client.
  • In the LoadGame event the Horizons and Odyssey flags indicate if those features are active, but in the 3.8 game client case you only get the Horizons boolean.

Other data Augmentations

Some schemas mandate that extra data be added, beyond what is in the source data, to aid Listeners.

This is usually related to specifying which system an event took place in, and usually means ensuring there is the full set of:

  1. StarSystem - the name of the system.
  2. SystemAddress - the game's unique numerical identifier for the system.
  3. StarPos - The system's co-ordinates.

Whilst it can be argued that any Listener should see preceding event(s) that give any missing information where at least the system name or SystemAddress is already in the event data, this might not always be true. So Senders MUST add this data where required. It helps to fill out basic system information (name, SystemAddress and co-ordinates).

However, there is a known game bug that can result in it stopping writing to the game journal, and some observed behaviour implies that it might then later resume writing to that file, but with events missing. This means any Sender that blindly assumes it knows the current system/location and uses that for these Augmentations might send erroneous data.

  1. Senders MUST cross-check available event data with prior 'location' event(s) to be sure the correct extra data is being added.
  2. Listeners SHOULD realise that any data added as an Augmentation might be in error.

For Senders, if the source data only has SystemAddress then you MUST check that it matches that from the prior Location, FSDJump or CarrierJump event before adding StarSystem and StarPos data to a message. Drop the message entirely if it does not match. Apply similar logic if it is only the system's name that is already present in data. Do not blindly add SystemAddress or StarPos. Likewise, do not blindly add StarPos if the other data is already in the source, without cross-checking the system name and SystemAddress.

Listeners might be able to apply their own cross-check on received messages, and use any mismatch with respect to what they already know to make a decision whether to trust the augmented data. Flagging it for manual review is probably wise.

Server responses

There are three possible sources of HTTP responses when sending an upload to EDDN.

  1. The reverse proxy that initially accepts the request.
  2. The python bottle module that the Gateway uses to process the forwarded requests. This might object to a message before the actual EDDN code gets to process it at all.
  3. The actual EDDN Gateway code.

Once a message has cleared the EDDN Gateway then there is no mechanism for any further issue (such as a message being detected as a duplicate in the Monitor downstream of the Gateway) to be reported back to the sender.

To state the obvious, if there are no issues with a request then an HTTP 200 response will be received by the sender. The body of the response should be the string OK.

Reverse Proxy responses

In addition to generic "you typoed the URL" and other such "you just didn't make a valid request" responses you might experience the following:

  1. 408 - Request Timed Out - the sender took too long to make/complete its request and the reverse proxy rejected it as a result.
  2. 503 - Service Unavailable - the EDDN Gateway process is either not running, or not responding.
  3. 400 - Bad Request - if you attempt to use plain HTTP, rather than HTTPS.

bottle responses

  1. 413 - Payload Too Large - bottle enforces a maximum request size and the request exceeds that. As of 2022-01-07 the limit is 1MiB, which is versus the compressed size of the body, if compression is used. Thus compression will allow for sending approximately 10x larger messages. To verify the current limit check for the line that looks like:

    bottle.BaseRequest.MEMFILE_MAX = 1024 * 1024 # 1MiB, default is/was 100KiB

    in src/eddn/Gateway.py, as added in commit 0e80c76cb564771465f61825e694227dcc3be312.

EDDN Gateway responses

For all failures the response body will contain text that begins FAIL: . Currently two different HTTP status codes are utilised:

  1. 400 - Bad Request - This indicates something wrong with the request body. Possibly due to a format issue (compression, form encoding), or the actual content of the EDDN message:

    1. FAIL: zlib.error: <detail> - A failure to decompress a message that claimed to be compressed.

    2. FAIL: Malformed Upload: <detail> - the message appeared to be form-encoded, but either the format was bad or there was no data key.

    3. FAIL: JSON parsing: <detail> - the message couldn't be parsed as valid JSON. e.g.

    FAIL: JSON parsing: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
    1. FAIL: Schema Validation: <detail> - the message failed to validate against the cited Schema. e.g.
    FAIL: Schema Validation: [<ValidationError: "'StarPos' is a required property">]

    The exact detail will be very much dependent on both the Schema the message cited and the contents of the message that failed to pass the validation.

    In particular, if the message contains a key that is tagged 'disallowed' in the Schema, then the response will look like:

    FAIL: Schema Validation: "[<ValidationError: "{'type': ['array', 'boolean', 'integer', 'number', 'null', 'object', 'string']} is not allowed for 'BadVALUE'">]"

    This is due to the use of a JSON schema stanza that says "don't allow any valid type for the value of this key" to trigger the error for such disallowed keys.

    Note how the value for the disallowed key is cited, not the key name itself. This is a limitation of how the jsonschema python module reports errors, and we are hoping to improve this.

  2. 426 - Upgrade Required - This indicates that the cited Schema, or version thereof, is outdated. The body of the response will be:

    FAIL: Oudated Schema: The schema you have used is no longer supported. Please check for an updated version of your application.

    The wording here is aimed at users of applications that send messages over EDDN. If you're the developer of such an application then obviously you need to update your code to use a currently supported Schema and version thereof.

There shouldn't be any other variants of a 'FAIL' message. If you find any then please open an issue on GitHub with as much detail as possible so that we can update this documentation.

Receiving messages

EDDN provides a continuous stream of information from uploaders. To use this data you'll need to connect to the stream using ZeroMQ (a library is probably available for your language of choice).

The URL for the live Relay is:

tcp://eddn.edcd.io:9500

Depending on the programming language and library used, you might need to explicitly subscribe to an empty topic, '', in order to receive anything.

Unfortunately at this time we're using an old version of ZeroMQ which does not support server-side subscription filtering. So by all means specify a more specific topic filter if it suits your needs, but realise that the dropping of not-matching messages will happen at your end, not the server.

Once you've connected to that you will receive messages. To access the data you will first need to zlib-decompress each message. Then you will have a textual JSON object as per the Schemas.

In general, check the guidance for Uploading messages for the expected format of the messages. Pay particular attention to any schema-specific Augmentations. Whilst Senders MUST make every effort to ensure such data is correct it is possible that bugs in either their code, or the game itself, could mean it is incorrect. Listeners use such data at their own risk.

Consumers can utilise the $schemaRef value to determine which Schema a particular message is for. There is no need to validate the messages against the Schemas yourself, as that is performed on the EDDN Gateway.
Messages that do not pass the schema validation there are not forwarded to receivers.

There is example code available for a variety of programming languages to help you get started.

Footnotes

  1. Live is a 4.0 client and has gameversion in LoadGame which is present in CAPI-sourced journals. ↩

  2. Legacy is a 3.8 client and last tested still doesn't have gameversion in LoadGame, and CAPI-sourced journals don't have Fileheader. ↩ ↩2

  3. And similarly for other CAPI endpoints, e.g. shipyard. ↩ ↩2 ↩3 ↩4