Release/5.8.0

As Windows 8.1 is now End-Of-Life we no longer explicitly support it, but for the time being it will likely continue to work. This is dependent on future Python releases not dropping support for Windows 8.1 in a manner that prevents it working. Any bug report made against Windows 8.1 use may be ignored unless reproduced on a supported OS.

---

NB: THERE WAS A SLIGHT BUILD ERROR IN 5.8.0-rc3 WHICH MEANS ITS UPDATE CHECKS ARE NON-FUNCTIONAL. IF YOU INSTALLED THAT VERSION THEN PLEASE MANUALLY DOWNLOAD AND INSTALL THIS ONE TO ENSURE YOU HAVE WORKING UPDATE CHECKS.
Release 5.8.0

This release is essentially the same as 5.8.0-rc3 with only the version and this changelog updated.

It brings a new feature related to Fleetcarrier data, some convenience for Linux users, some fixes, and otherwise some internal changes that should not adversely affect either users or third-party plugins. For the latter, read below for some new/changed things that could benefit you.

• This release, and all future ones, now create two additional archive files in the GitHub release:

  1. EDMarketConnector-release-<version>.zip
  2. EDMarketConnector-release-<version>.tar.gz

  The advantage of these over the GitHub auto-generated ones is that they have been hand-crafted to contain all the necessary files, and only those files.

  If you use the application from source, and not via a git clone, then we highly recommend you use one of these archives, not the GitHub auto-generated ones.

  Anyone installing on Windows should continue to use the EDMarketConnector_win_<version>.msi files as before.

• New Feature - You can now have the application query the /fleetcarrier CAPI endpoint for data about your Fleet Carrier. The data will then be passed to interested plugins.

  Note that there are some caveats:

  1. This feature defaults to Off. The option is on the Configuration tab of Settings as "Enable Fleetcarrier CAPI Queries". It is advised to only enable this if you know you utilise plugins that make use of the data.

  2. These queries are only triggered by CarrierBuy and CarrierStats Journal events, i.e. upon buying a Fleetcarrier or opening the Carrier Management UI in-game. NB: There is a 15 minute cooldown between queries.

  3. If you have Fleetcarrier cargo which got into the cargo hold through a lot of individual transactions, or if you have a lot of separate buy/sell orders then these queries can take a long time to complete.

     If this happens with your game account then all other CAPI queries will be blocked until the /fleetcarrier query completes. 'Other CAPI queries' means those usually triggered upon docking to gather station market, shipyard and outfitting data. To ameliorate the effects of this there is currently a timeout of 60 seconds on /fleetcarrier queries, and will not occur more often than every 15 minutes.

     We plan to address this by moving the /fleetcarrier queries into their own separate thread in the future.

• The code for choosing the 'Output' folder is now simply the tkinter function for such a dialogue, rather than a special case on Windows. In the past the former had issues with Unicode characters, but in testing no such issue was observed (on a supported OS).

• There are two new items on the "Help" menu:

  1. Troubleshooting -> Wiki:Troubleshooting
  2. Report A Bug -> Issues - New Bug Report

• Translations have been updated. Thanks again to our volunteer translators!

• If we ever activate any functionality killswitches, the popup denoting which are active has been made more readable.

• There's a new section in Contributing.md - "Python Environment". This should aid any new developers in getting things set up.

Linux Users

We now ship an io.edcd.EDMarketConnector.desktop file. To make use of this you should run scripts/linux-setup.sh once. This will:

1. Check that you have $HOME/bin in your PATH. If not, it will abort.

2. Create a shell script edmarketconnector in $HOME/bin to launch the application.

   NB: This relies on the filesystem location you placed the source in not changing. So if you move the source you will need to re-run the script.

3. Copy the .desktop and .icon files into appropriate locations. The .desktop file utilises the shell script created in step 2, and thus relies on it existing and on it being in a directory that is in your PATH.

Once this has been completed any XDG-compliant desktops should have an entry for "E:D Market Connector" in their "Games" menu.

Fixes

• The tracking of a Cmdr's location that was being performed by the core EDDN plugin has been moved into the Journal monitoring code. This results in the tracking being correct upon application (re)start, reflecting the state from the latest Journal file, rather than only picking up with any subsequent new Journal events.

  This change should remove instances of "Wrong System! Missed Jump ?" and similar sanity-check "errors" when continuing to play after a user restarts the application whilst the game is running.

  Plugin developers, see below for how this change can positively affect you.

• The name of the files written by "File" > "Save Raw Data" now have a . between the system and station names.

• Use of CAPI data in EDMC.exe when invoked with either -s or -n arguments hadn't been updated for prior changes, causing such invocations to fail. This has been fixed.

Plugin Developers

• Each plugin is now handed its own sub-frame as the parent parameter passed to plugin_app() instead of the actual main UI frame. These new Frames are placed in the position that plugin UI would have gone into. This should have no side effects on well-behaved plugins.

  However, if you have code that attempts to do things like parent.children() or the like in your plugin_app() implementation, this might have stopped working. You shouldn't be trying to do anything with any of the UI outside your plugin anyway, but if you definitely have a need then look things up using .nametowidget(). There are examples in the core plugins (which DO have good reason, due to maintaining main UI label values).

  All of the plugins listed on our Wiki were given perfunctory testing and no issues from this change were observed.

  This is a necessary first step to some pending plugin/UI work:

  • UI: Alllow for re-ordering third-party plugins' UIs
  • UI: Allow configuration of number of UI columns.
  • Re-work how Plugins' Settings are accessed

• New - capi_fleetcarrier() function to receive the data from a CAPI /fleetcarrier query. See PLUGINS.md for details.

• It was found that the ShutDown event (note the capitalisation, this is distinct from the actual Journal Shutdown event) synthesized for plugins when it is detected that the game has exited was never actually being delivered. Instead this was erroneously replaced with a synthesized StartUp event. This has been fixed.

• As the location tracking has been moved out of the core EDDN plugin, and into monitor.py all of it is now available as members of the state dictionary which is passed to journal_entry().

  This both means that no plugin should need to perform such location state tracking itself and they can take advantage of it being fully up to date when a user restarts the application with the game running.

  A reminder: When performing 'catch up' on the newest Journal file found at startup, the application does not pass any events to the journal_entry() method in plugins. This is to avoid spamming with data/state that has possibly already been handled, and in the case of the Cmdr moving around will end up not being relevant by the time the end of the file is reached. This limitation was also why the core EDDN plugin couldn't properly initiate its location tracking state in this scenario.

  See PLUGINS.md for details of the new state members. Pay particular attention to the footnote that details the caveats around Body tracking.

  Careful testing has been done for only the following. So, if you make use of any of the other new state values and spot a bug, please report it:

  1. SystemName
  2. SystemAddress
  3. Body (Name)
  4. BodyID
  5. BodyType
  6. StationName
  7. StationType
  8. (Station) MarketID

• There is an additional property request_cmdr on CAPIData objects, which records the name of the Cmdr the request was made for.

• FDevIDs files are their latest versions at time of this version's build.

• examples\plugintest - dropped the "pre-5.0.0 config" code, as it's long since irrelevant.

Developers

• If you utilise a git clone of the source code, you should also ensure the sub-modules are initialised and synchronised. wiki:Running from source has been updated to include the necessary commands.

• The coriolis-data git sub-module now uses an HTTPS, not "git" URL, so won't require authentication for a simple git pull.

• If you have a dump directory in CWD when running EDMarketConnector.py under a debugger you will get files in that location when CAPI queries complete. This will now include files with names of the form FleetCarrier.<callsign>.<timstamp>.json for /fleetcarrier data.

• All the main UI tk widgets are no...

Release/5.8.0-rc3

Pre-Release 5.8.0-rc3

A further issue with building the Windows executable has been found and fixed.
This release is otherwise identical to 5.8.0-rc1 (-rc2 was used up in only fixing one of the two problems).

Release/5.8.0-rc1

The Windows build of this was broken, please see 5.8.0-rc3 for the working installer.

Pre-Release 5.8.0-rc1

• This release, and all future ones, now create two additional archive files in the GitHub release:

  1. EDMarketConnector-release-<version>.zip
  2. EDMarketConnector-release-<version>.tar.gz

  The advantage of these over the GitHub auto-generated ones is that they have been hand-crafted to contain all the necessary files, and only those files.

  If you use the application from source, and not via a git clone, then we highly recommend you use one of these archives, not the GitHub auto-generated ones.

  Anyone installing on Windows should continue to use the EDMarketConnector_win_<version>.msi files as before.

• New Feature - You can now have the application query the /fleetcarrier CAPI endpoint for data about your Fleet Carrier. The data will then be passed to interested plugins.

  Note that there are some caveats:

  1. This feature defaults to Off. The option is on the Configuration tab of Settings as "Enable Fleetcarrier CAPI Queries". It is advised to only enable this if you know you utilise plugins that make use of the data.

  2. These queries are only triggered by CarrierBuy and CarrierStats Journal events, i.e. upon buying a Fleetcarrier or opening the Carrier Management UI in-game. NB: There is a 15 minute cooldown between queries.

  3. If you have Fleetcarrier cargo which got into the cargo hold through a lot of individual transactions, or if you have a lot of separate buy/sell orders then these queries can take a long time to complete.

     If this happens with your game account then all other CAPI queries will be blocked until the /fleetcarrier query completes. 'Other CAPI queries' means those usually triggered upon docking to gather station market, shipyard and outfitting data. To ameliorate the effects of this there is currently a timeout of 60 seconds on /fleetcarrier queries, and will not occur more often than every 15 minutes.

     We plan to address this by moving the /fleetcarrier queries into their own separate thread in the future.

• The code for choosing the 'Output' folder is now simply the tkinter function for such a dialogue, rather than a special case on Windows. In the past the former had issues with Unicode characters, but in testing no such issue was observed (on a supported OS).

• There are two new items on the "Help" menu:

  1. Troubleshooting -> Wiki:Troubleshooting
  2. Report A Bug -> Issues - New Bug Report

• Translations have been updated. Thanks again to our volunteer translators!

• If we ever activate any functionality killswitches, the popup denoting which are active has been made more readable.

• There's a new section in Contributing.md - "Python Environment". This should aid any new developers in getting things set up.

Linux Users

We now ship an io.edcd.EDMarketConnector.desktop file. To make use of this you should run scripts/linux-setup.sh once. This will:

1. Check that you have $HOME/bin in your PATH. If not, it will abort.

2. Create a shell script edmarketconnector in $HOME/bin to launch the application.

   NB: This relies on the filesystem location you placed the source in not changing. So if you move the source you will need to re-run the script.

3. Copy the .desktop and .icon files into appropriate locations. The .desktop file utilises the shell script created in step 2, and thus relies on it existing and on it being in a directory that is in your PATH.

Once this has been completed any XDG-compliant desktops should have an entry for "E:D Market Connector" in their "Games" menu.

Fixes

• The tracking of a Cmdr's location that was being performed by the core EDDN plugin has been moved into the Journal monitoring code. This results in the tracking being correct upon application (re)start, reflecting the state from the latest Journal file, rather than only picking up with any subsequent new Journal events.

  This change should remove instances of "Wrong System! Missed Jump ?" and similar sanity-check "errors" when continuing to play after a user restarts the application whilst the game is running.

  Plugin developers, see below for how this change can positively affect you.

• The name of the files written by "File" > "Save Raw Data" now have a . between the system and station names.

• Use of CAPI data in EDMC.exe when invoked with either -s or -n arguments hadn't been updated for prior changes, causing such invocations to fail. This has been fixed.

Plugin Developers

• Each plugin is now handed its own sub-frame as the parent parameter passed to plugin_app() instead of the actual main UI frame. These new Frames are placed in the position that plugin UI would have gone into. This should have no side effects on well-behaved plugins.

  However, if you have code that attempts to do things like parent.children() or the like in your plugin_app() implementation, this might have stopped working. You shouldn't be trying to do anything with any of the UI outside your plugin anyway, but if you definitely have a need then look things up using .nametowidget(). There are examples in the core plugins (which DO have good reason, due to maintaining main UI label values).

  All of the plugins listed on our Wiki were given perfunctory testing and no issues from this change were observed.

  This is a necessary first step to some pending plugin/UI work:

  • UI: Alllow for re-ordering third-party plugins' UIs
  • UI: Allow configuration of number of UI columns.
  • Re-work how Plugins' Settings are accessed

• New - capi_fleetcarrier() function to receive the data from a CAPI /fleetcarrier query. See PLUGINS.md for details.

• It was found that the ShutDown event (note the capitalisation, this is distinct from the actual Journal Shutdown event) synthesized for plugins when it is detected that the game has exited was never actually being delivered. Instead this was erroneously replaced with a synthesized StartUp event. This has been fixed.

• As the location tracking has been moved out of the core EDDN plugin, and into monitor.py all of it is now available as members of the state dictionary which is passed to journal_entry().

  This both means that no plugin should need to perform such location state tracking itself and they can take advantage of it being fully up to date when a user restarts the application with the game running.

  A reminder: When performing 'catch up' on the newest Journal file found at startup, the application does not pass any events to the journal_entry() method in plugins. This is to avoid spamming with data/state that has possibly already been handled, and in the case of the Cmdr moving around will end up not being relevant by the time the end of the file is reached. This limitation was also why the core EDDN plugin couldn't properly initiate its location tracking state in this scenario.

  See PLUGINS.md for details of the new state members. Pay particular attention to the footnote that details the caveats around Body tracking.

  Careful testing has been done for only the following. So, if you make use of any of the other new state values and spot a bug, please report it:

  1. SystemName
  2. SystemAddress
  3. Body (Name)
  4. BodyID
  5. BodyType
  6. StationName
  7. StationType
  8. (Station) MarketID

• There is an additional property request_cmdr on CAPIData objects, which records the name of the Cmdr the request was made for.

• FDevIDs files are their latest versions at time of this version's build.

• examples\plugintest - dropped the "pre-5.0.0 config" code, as it's long since irrelevant.

Developers

• If you utilise a git clone of the source code, you should also ensure the sub-modules are initialised and synchronised. wiki:Running from source has been updated to include the necessary commands.

• The coriolis-data git sub-module now uses an HTTPS, not "git" URL, so won't require authentication for a simple git pull.

• If you have a dump directory in CWD when running EDMarketConnector.py under a debugger you will get files in that location when CAPI queries complete. This will now include files with names of the form FleetCarrier.<callsign>.<timstamp>.json for /fleetcarrier data.

• All the main UI tk widgets are now properly named. This might make things easier if debugging UI widgets as you'll no longer see a bunch of !label1, !frame1 and the like.

  Each plugin's separator is named as per the scheme plugin_hr_<X>, and when a plugin has UI its new container Frame is named plugin_X. Both of these start with 1, not 0.

Pre-Release 5.7.1-alpha1

As well as some other changes, as below, this is a 64-bit release for plugin developers to test against. Moving forwards we're looking into producing both 32-bit and 64-bit build/installers. See also https://github.com/EDCD/EDMarketConnector/releases/tag/Release%2F5.5.1-alpha0

• New Feature - Perform a CAPI /fleetcarrier query when you open the Carrier Management UI in-game. It will also trigger when you buy an FC (Fleet Carrier).

  1. You will need to enable this using the new option on the Configuration tab of Settings.
  2. There will be a slight delay before the query is even initiated.
  3. These queries can take a long time, depending on the number of separate 'lots' of cargo, and orders, you have in your FC. In the meantime the normal 'station' CAPI queries will be blocked. Thus, it is advised to only enable this feature if you know you have an active plugin that utilises the data.
     However, currently the timeout for such queries is set at one minute, so any query taking longer than that will fail. This timeout will be tweaked based on user feedback. Ultimately we will look into making these queries independent of the 'station' CAPI queries.
  4. Due to the last point, there is an enforced 15 minute delay between the last such query completing and a new one being initiated.

• If we ever activate any functionality killswitches, the popup denoting which are active has been made more readable.

• There's a new section in Contributing.md - "Python Environment". This should aid any new developers in getting things set up.

• The code for choosing the 'Output' folder is now simply the tkinter function for such a dialogue, rather than a special case on Windows. In the past the former had issues with Unicode characters, but in testing no such issue was observed (on a supported OS).

Bug Fixes

• Use of CAPI data in EDMC.exe when invoked with either -s or -n arguments hadn't been updated for prior changes, causing such invocations to fail. This has been fixed.

Plugin Developers

• In order to make use of the data from a /fleetcarrier query you will need to implement the new function capi_fleetcarrier(). See PLUGINS.md for details.

• FDevIDs files are their latest versions at time of this version's build.

• examples\plugintest - dropped the "pre-5.0.0 config" code, as it's long since irrelevant.

NB: The EDMarketConnector_win_5.7.1-alpha1.msi file attached here was built on Athanasius' machine, not on GitHub. Here's what VirusTotal thinks.

Release/5.7.0

Release 5.7.0

This release re-enables CAPI queries for Legacy players. As a result, the 'Update' button functionality is now restored for Legacy players, along with "Automatically update on docking" functionality.

• We now test against, and package with, Python 3.11.1, 32-bit.

• This release is functionally identical to 5.7.0-rc1, as no problems were reported with that.

• As noted above, Legacy players now have CAPI functionality once more. Plugin developers check below for how you can determine the source galaxy of such data.

• Due to a bug it turned out that a workaround for "old browsers don't support very long URLs" had been inactive since late 2019. As no-one has noticed or complained we've now removed the defunct code in favour of the simple webbrowser.open(<url>).

  Testing showed that all of Firefox, Chrome and Chrome-based Edge worked with very long URLs without issues.

• EDMC.exe -n had been broken for a while, it now functions once more.

• Some output related to detecting and parsing gameversion from Journals has been moved from INFO to DEBUG. This returns the output of any EDMC.exe command to the former, quieter, version.

Bugs

• A corner case of "game not running" and "user presses 'Update' button" would result in an empty uploaderID string being sent to EDDN. Such messages are still accepted by the EDDN Gateway, and the Relay then obfuscates this field anyway. So, at worse, this would make it look like the same uploader was in lots of different places. This has been fixed.

• The message about converting legacy replay.jsonl was being emitted even when there was no file to convert. This has been fixed.

Plugin Developers

• An erroneous statement about "all of Python stdlib" in PLUGINS.md has been corrected. We don't/can't easily include all of this. Ask if any part of it you require is missing.

• In order to not pass Legacy data to plugins without them being aware of it there is now a new function cmdr_data_legacy(), which mirrors the functionality of cmdr_data(), but for Legacy data only. See PLUGINS.md for more details.

• The data passed to cmdr_data() and cmdr_data_legacy() is now correctly typed as CAPIData. This is a sub-class of UserDict, so you can continue to use it as such. However, it also has one extra property, source_host, which can be used to determine if the data was from the Live or Legacy CAPI endpoint host. See PLUGINS.md for more details.

• If any plugin had been attempting to make use of config.get_int('theme'), then be aware that we've finally moved from hard-coded values to actual defined constants. Example use would be as in:

  from config import config
  from theme import theme
  
  active_theme = config.get_int('theme')
  if active_theme == theme.THEME_DARK:
      ...
  elif active_theme == theme.THEME_TRANSPARENT:
      ...
  elif active_theme == theme.THEME_DEFAULT:
      ...
  else:
      ...

  But remember that all tkinter widgets in plugins will inherit the main UI current theme colours anyway.

• The contents of NavRoute.json will now be loaded during 'catch-up' when EDMarketConnector is (re-)started. The synthetic StartUp (note the capitalisation) event that is emitted after the catch-up ends will have state['NavRoute'] containing this data.

  However, the Fileheader event from detecting a subsequent new Journal file will blank this data again. Thus, if you're interested in "last plotted route" on startup you should react to the StartUp event. Also, note that the contents will indicate a NavRouteClear if that was the last such event.

  PLUGINS.md has been updated to reflect this.

• If you've ever been in the habit of running our develop branch, please don't. Whilst we try to ensure that any code merged into this branch doesn't contain bugs, it hasn't at that point undergone more thorough testing. Please use the stable branch unless otherwise directed.

• Some small updates have been made in edmc_data as a part of reviewing the latest update to coriolis-data.
  We make no guarantee about keeping these parts of edmc_data up to date.
  Any plugins attempting to use that data should look at alternatives, such as FDevIDs/outfitting.csv.

  A future update might remove those maps, or at least fully deprecate their use by plugins. Please contact us now if you actually make use of this data.

VirusTotal

As always we have pre-emptively uploaded EDMarketConnector_win_5.7.0.msi to VirusTotal. Check the report and Troubleshooting/viruses if you have concerns about a malware detection.

Release/5.7.0-rc1

Pre-Release 5.7.0-rc1

This release allows CAPI queries for Legacy players. This means that the 'Update' button functionality is restored, along with "Automatically update on docking".

• We now test against, and package with, Python 3.11.1, 32-bit.

• As noted above, Legacy players now have CAPI functionality once more. Plugin developers check below for how you can determine the source galaxy of such data.

• Due to a bug it turned out that a workaround for "old browsers don't support very long URLs" had been inactive since late 2019. As no-one has noticed or complained we've now removed the defunct code in favour of the simple webbrowser.open(<url>).

  Testing showed that all of Firefox, Chrome and Chrome-based Edge worked with very long URLs without issues.

• EDMC.exe -n had been broken for a while, it now functions once more.

• Some output related to detecting and parsing gameversion from Journals has been moved from INFO to DEBUG. This returns the output of any EDMC.exe command to the former, quieter, version.

Bugs

• A corner case of "game not running" and "user presses 'Update' button" would result in an empty uploaderID string being sent to EDDN. Such messages are still accepted by the EDDN Gateway, and the Relay then obfuscates this field anyway. So, at worse, this would make it look like the same uploader was in lots of different places. This has been fixed.

• The message about converting legacy replay.jsonl was being emitted even when there was no file to convert. This has been fixed.

Plugin Developers

• An erroneous statement about "all of Python stdlib" in PLUGINS.md has been corrected. We don't/can't easily include all of this. Ask if any part of it you require is missing.

• In order to not pass Legacy data to plugins without them being aware of it there is now a new function cmdr_data_legacy(), which mirrors the functionality of cmdr_data(), but for Legacy data only. See PLUGINS.md for more details.

• The data passed to cmdr_data() and cmdr_data_legacy() is now correctly typed as CAPIData. This is a sub-class of UserDict, so you can continue to use it as such. However, it also has one extra property, source_host, which can be used to determine if the data was from the Live or Legacy CAPI endpoint host. See PLUGINS.md for more details.

• If any plugin had been attempting to make use of config.get_int('theme'), then be aware that we've finally moved from hard-coded values to actual defined constants. Example use would be as in:

  from config import config
  from theme import theme
  
  active_theme = config.get_int('theme')
  if active_theme == theme.THEME_DARK:
      ...
  elif active_theme == theme.THEME_TRANSPARENT:
      ...
  elif active_theme == theme.THEME_DEFAULT:
      ...
  else:
      ...

  But remember that all tkinter widgets in plugins will inherit the main UI current theme colours anyway.

• The contents of NavRoute.json will now be loaded during 'catch-up' when EDMarketConnector is (re-)started. The synthetic StartUp (note the capitalisation) event that is emitted after the catch-up ends will have state['NavRoute'] containing this data.

  However, the Fileheader event from detecting a subsequent new Journal file will blank this data again. Thus, if you're interested in "last plotted route" on startup you should react to the StartUp event. Also, note that the contents will indicate a NavRouteClear if that was the last such event.

  PLUGINS.md has been updated to reflect this.

• If you've ever been in the habit of running our develop branch, please don't. Whilst we try to ensure that any code merged into this branch doesn't contain bugs, it hasn't at that point undergone more thorough testing. Please use the stable branch unless otherwise directed.

• Some small updates have been made in edmc_data as a part of reviewing the latest update to coriolis-data.
  We make no guarantee about keeping these parts of edmc_data up to date. Any plugins attempting to use that data should look at alternatives, such as FDevIDs/outfitting.csv.

  A future update might remove those maps, or at least fully deprecate their use by plugins. Please contact us now if you actually make use of this data.

Release/5.6.1

Release 5.6.1

This release addresses some minor bugs and annoyances with v5.6.0, especially for Legacy galaxy players.

In general, at this early stage of the galaxy split, we prefer to continue to warn Legacy users who have 'send data' options active for sites that only accept Live data. In the future this might be reviewed and such warnings removed such that the functionality fails silently. This might be of use to users who actively play in both galaxies.

• CAPI queries will now only be attempted for Live galaxy players This is a stop-gap whilst the functionality is implemented for Legacy galaxy players. Doing so prevents using Live galaxy data whilst playing Legacy galaxy, which would be increasingly wrong and misleading.

  1. 'Automatic update on docking' will do nothing for Legacy players.
  2. Pressing the 'Update' button whilst playing Legacy will result in a status line message "CAPI for Legacy not yet supported", and otherwise achieve nothing. The only function of this button is to query CAPI data and pass it to plugins, which does not include Inara and EDSM.
  3. A Legacy player trying to use "File" > "Status" will get the message "Status: No CAPI data yet" due to depending on CAPI data.

  It is hoped to implement CAPI data retrieval and use for Legacy players soon, although this will likely entail extending the plugins API to include a new function specifically for this. Thus only updated plugins would support this.

• EDDN: Where data has been sourced from the CAPI this application now sends a header->gameversion in the format "CAPI-(Live|Legacy)-<endpoint>" as per the updated documentation.

  1. As this version only queries CAPI for Live players that will only be "CAPI-Live-<endpoint>" for the time being.

  2. If, somehow, the CAPI host queried matches neither the current Live host, the Legacy host, nor the past beta host, you will see "CAPI-UNKNOWN-<endpoint>".

  3. As that statement implies, this application will also signal 'Live' if pts-companion.orerve.net has been used, due to detecting an alpha or beta version of the game. However, in that case the /test schemas will be used.

  Closes #1734.

• Inara: Only warn about Legacy data if sending is enabled in Settings > Inara.

  Closes #1730.

• Inara: Handling of some events has had a sanity check added so that the Inara API doesn't complain about empty strings being sent. In these cases the event will simply not be sent.

  Closes #1732.

• EDSM: EDSM has decided to accept only Live data on its API. Thus, this application will only attempt to send data for Live galaxy players.

  If a Legacy galaxy player has the Settings > EDSM > "Send flight log and Cmdr status to EDSM" option active then they will receive an error about this at most once every 5 minutes. Disabling that option will prevent the warning.

Plugin Developers

• PLUGINS.md has been updated to make it clear that the only use of imports from the config module are for setting/getting/removing a plugin's own configuration, or detecting application shutdown in progress.
• PLUGINS.md has also been updated to add a note about how the data passed to a plugin cmdr_data() is, strictly speaking, an instance of CAPIData, which is an extension of UserDict. It has some extra properties on it, but these are for internal use only and no plugin should rely on them.
• As noted above, implementing CAPI data for Legacy players will likely entail an additional function in the API provided to plugins. See #1728 for discussion about this.

VirusTotal

Pre-emptive upload to VirusTotal for EDMarketConnector_win_5.6.1.msi

Release/5.6.0

Release 5.6.0

Tha major reason for this release is to address the Live versus Legacy galaxy split coming in Update 14 of the game.
See the section "Update 14 and the Galaxy Split" below for how this might impact you.

Changes

• We now test against, and package with, Python 3.10.8.

• The code for sending data to EDDN has been reworked. This changes the 'replay log' from utilising an internal array, backed by a flat file (replay.jsonl), to an sqlite3 database.

  As a result:

  1. Any messages stored in the old replay.jsonl are converted at startup, if that file is present, and then the file removed.
  2. All new messages are stored in this new sqlite3 queue before any attempt is made to send them. An immediate attempt is then made to send any message not affected by "Delay sending until docked".
  3. Sending of queued messages will be attempted every 5 minutes, unless "Delay sending until docked" is active and the Cmdr is not docked in their own ship. This is in case a message failed to send due to an issue communicating with the EDDN Gateway.
  4. When you dock in your own ship an immediate attempt to send all queued messages will be initiated.
  5. When processing queued messages the same 0.4-second inter-message delay as with the old code has been implemented. This serves to not suddenly flood the EDDN Gateway. If any message fails to send for Gateway reasons, i.e. not a bad message, then this processing is abandoned to wait for the next invocation.

  The 5-minute timer in point 3 differs from the old code, where almost any new message sending attempt could initiate processing of the queue. At application startup this delay is only 10 seconds.

  Currently, the feedback of "Sending data to EDDN..." in the UI status line has been removed.

  If you do not have "Delay sending until docked" active, then the only messages that will be at all delayed will be where there was a communication problem with the EDDN Gateway, or it otherwise indicated a problem other than 'your message is bad'.

• As a result of this EDDN rework this application now sends appropriate gameversion and gamebuild strings in EDDN message headers.
  The rework was necessary in order to enable this, in case of any queued or delayed messages which did not contain this information in the legacy replay.jsonl format.

• For EDSM there is a very unlikely set of circumstances that could, in theory lead to some events not being sent. This is so as to safeguard against sending a batch with a gameversion/build claimed that does not match for all of the events in that batch.

  It would take a combination of "communications with EDSM are slow", more events (the ones that would be lost), a game client crash, and starting a new game client before the 'more events' are sent.

Update 14 and the Galaxy Split

Due to the galaxy split announced by Frontier there are some changes to the major third-party websites and tools.

• Inara has chosen to only accept Live galaxy data on its API.

  This application will not even process Journal data for Inara after 2022-11-29T09:00:00+00:00 unless the gameversion indicates a Live client. This explicitly checks that the game's version is semantically equal to or greater than '4.0.0'.

  If a Live client is not detected, then there is an INFO level logging message "Inara only accepts Live galaxy data", which is also set as the main UI status line. This message will repeat, at most, every 5 minutes.

  If you continue to play in the Legacy galaxy only then you probably want to just disable the Inara plugin with the checkbox on Settings > Inara.

• All batches of events sent to EDSM will be tagged with a gameversion, in a similar manner to the EDDN header.

  Ref: EDSM api-journal-v1

• All EDDN messages will now have appropriate gameversion and gamebuild fields in the header as per EDDN/docs/Developers.md.

  As a result of this you can expect third-party sites to choose to filter data based on that.

  Look for announcements by individual sites/tools as to what they have chosen to do.

Known Bugs

In testing if it had been broken at all due to 5.5.0 -> 5.6.0 changes it has come to light that EDMC.EXE -n, to send data to EDDN, was already broken in 5.5.0.

In addition, there is now some extra 'INFO' logging output which will be produced by any invocation of EDMC.EXE. This might break third-party use of it, e.g. Trade Computer Extension Mk.II.
This will be fixed as soon as the dust settles from Update 14, with emphasis being on ensuring the GUI EDMarketConnector.exe functions properly.

Notes for EDDN Listeners

• Where EDMC sourced data from the Journal files it will set gameversion and gamebuild as per their values in Fileheader or LoadGame, whichever was more recent (there are some events that occur between these).

• If any message was already delayed such that it did not have the EDDN header recorded, then the gameversion and gamebuild will be empty strings. In order to indicate this the softwareName will have (legacy replay) appended to it, e.g. E:D Market Connector Connector [Windows] (legacy replay). In general this indicates that the message was queued up using a version of EDMC prior to this one. If you're only interested in Live galaxy data then you might want to ignore such messages.

• Where EDMC sourced data from a CAPI endpoint, the resulting EDDN message will have a gameversion of CAPI-<endpoint> set, e.g. CAPI-market. At this time it is not 100% certain which galaxy this data will be for, so all listeners are advised to ignore/queue such data until this is clarified.

  gamebuild will be an empty string for all CAPI-sourced data.

Plugin Developers

• There is a new flag in state passed to plugins, IsDocked. See PLUGINS.md for details.

VirusTotal

Pre-emptive upload to VirusTotal for EDMarketConnector_win.5.6.0.msi.

Release/5.6.0-beta1

Pre-Release 5.6.0-beta1

• We now test against, and package with, Python 3.10.8.

• The code for sending data to EDDN has been reworked. This changes the 'replay log' from utilising an internal array, backed by a flat file (replay.jsonl), to an sqlite3 database.

  As a result:

  1. Any messages stored in the old replay.jsonl are converted at startup, if that file is present, and then the file removed.
  2. All new messages are stored in this new sqlite3 queue before any attempt is made to send them. An immediate attempt is then made to send any message not affected by "Delay sending until docked".
  3. Sending of queued messages will be attempted every 5 minutes, unless "Delay sending until docked" is active and the Cmdr is not docked in their own ship. This is in case a message failed to send due to an issue communicating with the EDDN Gateway.
  4. When you dock in your own ship an immediate attempt to send all queued messages will be initiated.
  5. When processing queued messages the same 0.4-second inter-message delay as with the old code has been implemented. This serves to not suddenly flood the EDDN Gateway. If any message fails to send for Gateway reasons, i.e. not a bad message, then this processing is abandoned to wait for the next invocation.

  The 5-minute timer in point 2 differs from the old code, where almost any new message sending attempt could initiate processing of the queue. At application startup this delay is only 10 seconds.

  Currently, the feedback of "Sending data to EDDN..." in the UI status line has been removed.

• As a result of this EDDN rework this application now sends appropriate gameversion and gamebuild strings in EDDN message headers.
  The rework was necessary in order to enable this, in case of any queued or delayed messages which did not contain this information in the legacy replay.jsonl format.

  EDDN Listeners - If any message was already delayed such that it did not have the EDDN header recorded, then the gameversion and gamebuild will be empty strings. In order to indicate this the softwareName will have (legacy replay) appended to it, e.g. E:D Market Connector Connector [Windows] (legacy replay). In general this indicates that the message was queued up using a version of EDMC prior to this one. If you're only interested in Live galaxy data then you might want to ignore such messages.

Update 14 and the Galaxy Split

Due to the galaxy split announced by Frontier there are some changes to the major third-party websites and tools.

• Inara has chosen to only accept Live galaxy data on its API. As such EDMarketConnector will not even process Journal data for Inara after 2022-11-29T09:00:00+00:00 unless the gameversion indicates a Live client. This explicitly checks that the game's version is semantically equal to or greater than '4.0.0'.

  If a Live client is not detected, then there is an INFO level logging message "Inara only accepts Live galaxy data", which is also set as the main UI status line. This message will repeat, at most, every 5 minutes.

  If you continue to play in the Legacy galaxy only then you probably want to just disable the Inara plugin with the checkbox on Settings > Inara.

• As this application now sends gameversion in all EDDN message headers you can expect other third-party sites to choose to filter data based on that.

  Look for announcements by individual sites/tools as to what they have chosen to do.

Plugin Developers

• There is a new flag in state passed to plugins, IsDocked. See PLUGINS.md for details.

Release/5.5.1-alpha0

** See https://github.com/EDCD/EDMarketConnector/releases/tag/Release%2F5.7.1-alpha1 instead for a 64-bit test version.**

• We now test against, and package with, Python 3.10.8 64-bit. In part this change to 64-bit is because py2exe, the program we employ to create windows executables, is no longer warranting that their 32-bit versions will continue to work (they no longer run 32-bit tests).

  Another advantage to switching to 64-bit is in case any data-hungry plugins attempt to use around/more than 2GiB of RAM at once. With 64-bit Python as the basis the only limit will be your system RAM and 'page file' space.

• ctypes code (for Windows API calls) has been updated where necessary to work with 64-bit Python.

• The ctypes code to invoke a "choose a folder" dialog for Settings > Output tab > File location > Browser has been removed in favour of always using tkinter.filedialog().

  • This was tested as still working under Windows 10 21H2 (Build 19044.2311), even when forcing a UTF-8 locale to not be selected and selecting a folder containing Unicode characters.

  • The Windows UTF-8 code page has been available since Windows 10 Build 1903, and even that version is no longer supported, so any supported version of Windows 10 shouldn't have a problem with this.

  • The last Windows 8.1 build is still supported until 'Jan 10, 2023' and might have an issue with this. But as that 'end of life' date is very much in sight we don't plan to hold back the change just for that OS.

Those statements about (un)supported OS versions not withstanding, please do test this version and that specific functionality on any and all Windows systems which ran EDMarketConnector 5.5.0 without issues.

NB: If you make use of any third-party plugins that utilise extra modules which themselves use native libraries then you might find that plugin no longer works. The plugin author will likely need to make a new release which contains the 64-bit versions of any libraries, rather than the 32-bit version they would have been supplying up until now. Please report the issue to them, not us.

Plugin Developers

As noted above, if any third-party plugin is shipped with extra python modules that include native libraries then you will need to switch to shipping the 64-bit versions of those libraries.

There shouldn't be any other impact on plugins, as no API interfaces have changed. Although if your own code utilises any ctypes calls you might also find that it needs updating or replacing.

The key thing to look out for there is where the naive way to call a ctypes-mediated Windows API function defaults the return type to int. On 32-bit Python this is OK if the actual return type is meant to be a pointer, as the two types are of the same size and that pointer will continue to work if passed through to another ctypes function. However on 64-bit python the size of int is still 32 bits, but the pointers are 64 bits. Thus any returned pointer gets erroneously cast and truncated to 32 bits unless you properly declare the function prototype first.

See https://docs.python.org/3/library/ctypes.html#ctypes.WINFUNCTYPE for how to properly declare a function prototype, including return value.

Due to this move to 64-bit Python being a potentially breaking change for some plugins we will be bumping our major version in the final release, i.e. to 6.0.0. We're holding off on that for these alpha releases in case we need to backout/delay this change, not wanting to have testers on an alpha version that is considered newer than a release 5.6.0 or later 5.x.y version.