Basher package #163
Assignees
Labels
A-gui
Area: GUI (The wx wrapper classes in the gui package)
A-wx
Area: wxPython interactions (the gui package, balt.py and the basher package)
C-goal
Category: Long term goal. May be code-related or a meta goal.
C-refactoring
Category: Refactoring. A purely internal refactoring, with no user-facing changes
Milestone
Comments
Utumno
added
the
C-goal
|
Ok - basher split - commit 3000 ;) Review the merge commit above - pushed https://github.com/wrye-bash/wrye-bash/tree/zzz-utumno-basher-nitpick (merged as 'utumno-basher-package) where the commits appear before I squashed them together - to be deleted. Will go on in https://github.com/wrye-bash/wrye-bash/tree/utumno-issue-163-basher-refactoring - I need to remove the Link classes from there - this is related to the refresh work too - basher is difficult to work with as it is. |
Utumno
added a commit
that referenced
this issue
Nov 21, 2014
See: #163 3fcbe3f (needed review!) introduced: Traceback (most recent call last): File "<string>", line 11205, in Execute File "bash\barb.py", line 352, in Apply self.WarnRestart() File "bash\barb.py", line 411, in WarnRestart bash.bashFrame.Restart() AttributeError: 'NoneType' object has no attribute 'Restart' Turns out bashFrame is initialized in basher.BashApp.onInit(self): def Init(self): # not OnInit(), we need to initialize _after_ the app # has been instanced There frame = BashFrame() is called and in there `BashFrame = self` is set. Change: now basher.BashApp.onInit() returns the bashFrame !
Utumno
added a commit
that referenced
this issue
Dec 8, 2014
Under #163. The Links classes are ripped to modules. Their internal API changed. New Link subclasses to factor out the UI code/logic. AppendToMenu calls are confined in balt. wx usages went down by 785: dev: 2251 usages (balt: 381, basher/: 1457) merge: 1466 usages (balt: 395, basher/: 832) dev at 43e359a Link 'data' attribute renamed to 'selected': data is a nono for a name plus it had a different meaning in different Link subclasses. That's towards bringing closer (read eliminate) Tank and List. Introduced module globals as intermediate step to be able to rip the classes. These must go - maybe static fields of BashFrame. Used existing balt wrappers - they must be revisited to prune the id parameter (partly done). Ids belong to wx code ie balt.
Utumno
added a commit
that referenced
this issue
Jan 12, 2015
Under #163. Dropped self.checkboxes in List (now UIList/List.icons) Inverted meaning of sort indicators in mod list to match explorer style: ^ == ascending - if reverse: self.list.SetColumnImage(self.colDict[col], self.sm_up) - else: self.list.SetColumnImage(self.colDict[col], self.sm_dn) + if reverse: self.list.SetColumnImage(self.colDict[col], self.sm_dn) + else: self.list.SetColumnImage(self.colDict[col], self.sm_up)
Utumno
added a commit
that referenced
this issue
Jan 12, 2015
Under #163. UIList common superclass of Tank and List and bugfixes.
This was referenced Jan 16, 2015
Closed
Utumno
added a commit
that referenced
this issue
Jan 29, 2015
Under #163, #174. Seems we are going towards a package per panel rather than a links package - the specific link init methods and the links belong together as is shown by the ugly `__all__` directives. `import *` is _evil_ and in combination with our loader _chaotic_ evil. Mopy/bash/basher/app_buttons.py Renamed Oblivion_Button to more accurate Game_Button. Note pickle cruft.
Utumno
added a commit
that referenced
this issue
Jan 29, 2015
I wanted to rip as much out of bosh as I could. Turns out ModInfos uses the isMergeable stuff which in turns uses modInfos global. I could go with in-function imports but preferred to not sweep this under the carpet: bosh needs splitting, no escaping from this (and mergeability stuff stays there to remind us). configIsCBash() is another beast: the iron rule is "no module imports basher" (apart bash). configIsCBash() belongs to the (GUI) configs but those are used outside of basher. Stuck it right in the top of patcher for now, for everyone to see. Closes #3, at very long last. Concerns #163, #178 also - we must decide on GUI patchers config handling.
Utumno
added a commit
that referenced
this issue
Feb 9, 2015
Under #163. Fixes also Installers, People status count not updated on deletion. self.panel = parent.GetParent().GetParent() is still ugly but iniPanel = self.GetParent().GetParent().GetParent() was uglier and all over. if hasattr(Link.Frame, 'notebook'): we were being called via onShow callback in BashNotebook.__init__ where Link.Frame.notebook was not yet set. Now I register the callback _after_ `self.SetSelection(pageIndex)` so this check is not needded - note ShowPanel() is manually called in RefreshData() callback. Renamed _currentTab to currentPage - not exactly private plus we already have Panel and Page - although what we really see is tabs... Note also BashNotebook.OnShowPage callback was not fired when pageIndex was 0 - verified by stepping in with the debugger. This added to my confusion and caused even more weirdness - a wx bug ? Welcome to event programming - don't forget your pills.
Utumno
added a commit
that referenced
this issue
Feb 9, 2015
Under #163. Deglobalization: more work needed but on its way. Globals are relics of the once 20k lines basher - and a code smell. Moved to BashFrame where they belong - and used for inter panel communication. Links API nearing beta - check Link docs. Did not want to hit master with a half baked Link API - still work to be done (Idlist is evil - should give its place to OO tools completely). wx id is now manipulated only by ChoiceLink local classes. Contains also some de crufting, a fix for unicode names on settings files, and some work on bash.main().
Utumno
added a commit
that referenced
this issue
Feb 23, 2015
Under #163. Note that `'bash.installers.colAligns': {'Order': 'RIGHT',...` results in: Traceback (most recent call last): File "bash\basher\__init__.py", line 4136, in __init__ item = panel(self) File "bash\basher\__init__.py", line 2862, in __init__ self.uiList = InstallersList(left, data, self.keyPrefix, details=self) File "bash\basher\__init__.py", line 2494, in __init__ dndList=True, dndFiles=True, dndColumns=['Order']) File "bash\balt.py", line 1931, in __init__ self.UpdateColumns() File "bash\balt.py", line 1995, in UpdateColumns colAlign = wxListAligns[self.colAligns.get(colKey,0)] TypeError: list indices must be integers, not str defaultParam did not do anything in those keys as they were already set in settingsDefaults (colWidths and colAligns)
Utumno
added a commit
that referenced
this issue
Feb 23, 2015
Under #163. Added prompt if trying to open more than 7 files. Signed-off-by: MrD <[email protected]>
Utumno
added a commit
that referenced
this issue
Feb 23, 2015
…ned: Under #163. To see the sizeHints in action unbind the OnSize() callback in UIList. UIList displays in size hints size IIUC - and can't be resized. Sizes/EVENT_SIZE handling needs reworking from the ground up - for now I just make the UI more consistent (one of the first rules of UI design). Plus those are really class level attributes - yes, those are all essentially singleton classes (the UILists). minimumSize sets the panels isVertical ? height : width. Maybe 256 is a lot but installers was more (the sash pos was at 530) plus the UI design is post refactoring concern.
Utumno
added a commit
that referenced
this issue
Feb 28, 2015
Under #163. I guess I have to implement _appendable_ metaclass to call this beta. IdList finally gone. MenuLink: made 'name' init param optional. Also adds extension skipping when renaming and "sort by selected" in master lists. Bunch of warnings fixed.
Utumno
added a commit
that referenced
this issue
Mar 17, 2015
Under #163. After 4 months of work I am finally able to move this beast to base. This opens the way to control refresh from a central point. Needless to say this will end up in UIList - but now we are at the heart of Tank vs List dichotomy and those two won't give up without a fight. Renamed RefreshUI of other classes - RefreshUI is very central and ini controls and Panel refreshes must not share the same name. The Tank UpdateItem(s) win hands down as they really permit more flexible (read focused) updates (methinks) - check the (naive) attempt to only refresh renamed saves - blows with: Traceback (most recent call last): File "bash\basher\__init__.py", line 1990, in OnLabelEdited self.RefreshUI(renamed) ### (ut)TEST File "bash\basher\__init__.py", line 1993, in RefreshUI super(SaveList, self).RefreshUI(files, detail) File "bash\basher\__init__.py", line 399, in RefreshUI self.PopulateItem(file_, selected=selected) File "bash\basher\__init__.py", line 2001, in PopulateItem itemDex = self.items.index(itemDex) ValueError: bolt.Path(u'C:\\Users\\MrD\\Documents\\My Games\\Oblivion\\Saves\\SI\\oio___.ess') is not in list Passing a string _or_ a list _or_ an item is totally non OO and will be squashed. Replaced all instances passing with single items lists. RefreshUI is ugly - better to be reminded of this fact when using it. EDIT: notice I needed to call SortItems() separately - this is to be amended. Signed-off-by: MrD <[email protected]>
Utumno
added a commit
that referenced
this issue
Mar 17, 2015
Remaining globals: modDetails, saveDetails, under #163. Also replaced Link.Frame.statusBar with property to use with (Un)HideButton, UpdateIconSizes, PopupMenu... Encapsulated another wx call - GetStatusBar() - and binned now unused SetText() - took the timer for the ride. Did not seem to work, plus events are _evil_ - keep just the necessary part of the evil. Mopy/bash/basher/mod_links.py: modDetails local renamed.
Utumno
added a commit
that referenced
this issue
Mar 17, 2015
Under #163. The only place this was set was in _checkUncheckMod. I really do not get why it reversed the current sort order: self.colReverse[self.sort] = not self.colReverse.get(self.sort,0) It _did_ reverse the sort order (test with, say, 'File' column) on (de)selecting a mod - which seemed very much like a bug anyway. It probably at some point only sorted `if self.selectedFirst` - this would be laudable, however SortItems is much cheaper now, and it would sort irrespective of sortDirty anyways. RefreshUI will call sort items and this will sort newly (de)selected mods as needed - avoiding unnecessary sorts is on the TODOs. Signed-off-by: MrD <[email protected]>
Utumno
added a commit
that referenced
this issue
Mar 17, 2015
…otes: Under #163. getColumns() would ideally be an array of lambdas as _sort_keys. This would help also define in ONE place the columns - and get rid of 'bash.*.allCols' - allCols@ could return _columns.keys() - _columns should be an ordered dict of sorts (['File': (sortKey, columnLabel),...]). Reworked also PopulateColumns(), had some regressions (colDict was not updated properly). Still it is too complex - note in particular the creation of a column in _two_ branches and the self._colDict attribute I'd like to see gone. # This is the 2nd commit message: Introducing List.setUI: This must eventually be melded with Tank api (getGuiKeys(), getMouseText()).
Utumno
added a commit
that referenced
this issue
Oct 22, 2016
I really struggled to get this right - well turns out all
that's needed was:
@@ -3366,2 +3366,3 @@ def __init__(self,parent):
left.SetSizer(vSizer((self.uiList,1,wx.GROW)))
+ right.SetSizer(vSizer((self.detailsPanel,1,wx.GROW)))
I dropped wx.LayoutAlgorithm() calls - they do not seem to make
a difference.
Note I inverted the order the details vs the panel ClosePanel was called
@@ -231,5 +231,5 @@ def RefreshUIColors(self):
def ClosePanel(self, destroy=False):
- super(_DetailsViewMixin, self).ClosePanel(destroy)
if self.detailsPanel:
self.detailsPanel.ClosePanel(destroy)
+ super(_DetailsViewMixin, self).ClosePanel(destroy)
The current order resulted in the code for installers and people to
fail saving the comments/text entered before shutting Bash down.
At long last SashTankPanel is gone, gone, gone - time to fix the Panel's
hierarchy.
Under #163, #253
Utumno
removed
the
M-backburner
Utumno
added a commit
that referenced
this issue
Dec 1, 2016
At long last: - cures remaining glitches in refresh - closes #293 for good - fixes the ini tab glitch, however the left panel can be minimized out of view, that's another wx mystery - see #197 - bins _select override - cleans up the ini panel API: RefreshIniDetails, RefreshPanel, SelectTweak - see also next commit. - Leverage ShowPanel API that is now ready to be fine tuned, cleaned up and commented. - and so much more really - this at long last finalizes the UI panels refactoring and opens the way to fine tune refresh, applying ini tweaks etc etc Next commit I kept separate but actually belongs with this one. Under #326 Under #163
Utumno
added a commit
that referenced
this issue
Dec 1, 2016
Triggered by refresh fixups ( 665da9c ) that however needed that I finally wrote the details ini panel. Under #326. Settles #293 finally. Ghost of #163, now all Bash tabs are BashTab instances, yey! Signed-off-by: MrD <[email protected]>
Utumno
pushed a commit
that referenced
this issue
Jan 4, 2019
The goal here is to replace unnecessary overrides of Link._initData with overrides of a new property, ItemLink.menu_help. To that end, this commit and the one after it renames ItemLink.help to _help to bring it in line with the other internal ItemLink variable, _text. This commit runs PyCharm's rename refactoring as a starting point for manual renaming of everything it missed. Rename ItemLink.help to _help, Part 2 See the previous commit for Part 1 and the explanation for this change. This commit fixes all (hopefully) remaining usages of the old ItemLink.help in Wrye Bash that were missed in the automatic refactoring in Part 1. Removed MenuLink._help MenuLink doesn't implement ItemLink, so that assignment did absolutely nothing (and even had a value of 'UNUSED') Utumno: I squashed the commits by @Infernio (Thanks!) cause if they were kept separate they would break the menus. Note I wanted to rename help to _help because (like text) is a very bad name for an attribute, too generic, used in build in params (like argparse) it _is_ a builtin and whatnot. Ideal name would be _menu_help but that would cause quite a few lines to pass the 79 chars limit. Before this commit there were 0 occurrences of _help in the codebase and now `_help` must ONLY be used for the menus attribute. I edited this commit to wrap a couple lines that passed the 79 chars limit. Having a property will help us add more elaborate help messages (for instance automatically adding the shortcut) and simplifies further the menus. Oh here are the next two commits that needed be squashed: Created ItemLink.menu_help Also refactored ItemLink.AppendToMenu to use it. This property will allow us to replace several custom help definitions in _initData's with overrides of menu_help - see the next commit for that. Changed _ProfileLink.help to menu_help: This class and its subclasses were using their own property called 'help', which overrode the "help" class variable. Utumno: Pycharm's automated rename missed instance attributes and properties overrides, we should report it to them (or have I already?) Under #174, #163, the eternal
Infernio
added
the
M-backburner
Infernio
pushed a commit
that referenced
this issue
Mar 23, 2020
Huge merge introducing a gui package that decouples the wx library from the rest of the Bash code for real. Explicit wx usages (imports and wx.* usages) are almost gone from unrelated files, implicit usages (e.g. event parameters and methods that directly go into wx code) have gone down significantly. Howver, the most important part is that we now wrap the native control and export an API that should stay the same even if we switch the graphics library at some point (already of use for the wx4 update). Contains also some much needed app_buttons.py refactoring, GUI work and much more detailed docs. Another nice side benefit is that the debugger no longer shows a jungle of random variables and proprties for GUI objects. This is based on initial work by @nycz with imortant additions by @Infernio including the events handling API. Final work on wrapping windows, frames, dialogs and ListCtrl was done by @Utumno. Leaves us with 218 uses of wx outside gui - that's amazing compared to where we started, where wx was everywhere. Pros: - Another level of abstraction™ - so you do not bind events anymore but subscribe behavior, you do not add to sizers, you define layouts etc - internal API is oriented into grouping code according these usecases - cutting on very nasty repeated logic (see With***Events and co) - much more adequate naming, again modeled according to the usecases. Naming is one of the WIP parts of this merge to be finalized in pt2 - see commit history for some renames in the branch - this gives us the opportunity to revisit the GUI from the ground up - quite a few bugs were fixed and we can more easily centralize common patterns. Cons: - 36 files changed, 5157 insertions(+), 3540 deletions(-), see Pros Limits: - wx classes that use wx objects - see belt with Wizard and WizardPage - from an architectural endpoint - turns out wx is built with inheritance as the main way of using it (look at about 100% of the examples on the web). Wrapping is not as flexible - at least it should allow wrapping an _Acomponent - or not, as this may prove tricky - event handling is as tricky as ever - especially concerning the events that are automatically fired from wx methods - see: https://wxpython.org/Phoenix/docs/html/events_overview.html#user-generated-events-vs-programmatically-generated-events Under #190 and, by extension, #163
Infernio
added a commit
that referenced
this issue
Sep 16, 2020
307 is a gigantic release that has been cooking for almost five years
at this point. In fact, it is even bigger than 306:
305 -> 306: 48,748 additions and 44,482 deletions
306 -> 307: 172,916 additions and 111,125 deletions
Where 306 was mostly about refactoring the codebase to save it from
becoming inoperable, 307 is all about using the newly refactored
codebase to design new features, support more games and, of course,
continue the everlasting refactoring war.
The following notes are an attempt to summarize every important thing
that's happened in 307, grouped as topics in a semi-chronological
order. Do note that this is mostly a futile effort due to the 2000+
commits and 280000+ lines worth of changes that make up 307's
development (and the fact that I only joined the project two thirds of
the way through 307 :P).
It is also highly recommended to read this on GitHub, with gitk open
in the background. That way you can click on issue numbers to open
them, while pasting commit SHAs into gitk, where the individual
commits that make up a merge can be seen (GitHub does not expose
this at all).
All mentioned commits are authored by @Utumno or @Infernio unless
otherwise mentioned.
-----------------------------------------------------------------------
### The bosh Split (#201)
The first goal of 307 was to split bosh into a package. After the
patchers had been split out (#163) and basher turned into a
package (#3), bosh was clearly the next big target.
1bdc253e362857dc8a2a484be60bbcccd7891d82 began the splitting, then
it was continued by splitting out the 'messages' code backing the
PM Archives tab in 906858fe14c730ba797711a855933d996d573e2f.
As a small interlude, 5b14b499e17c48917a472ee9ed426b3c7afd8985 then
removed the PM Archives tab entirely (#221) due to the maintenance
burden it had turned into.
682d2134c5e9f69b057cc9fb6e69e13767f5e7de continued the bosh split by
ripping out:
- The face transfer code (used for moving NPC/player faces between
saves and mods) into faces.py
- The OMODs code (used for unpacking OMODs, an old, ugly mod format
used by OBMM) into omods.py
- And the BAIN converter code (used for BCFs, aka BAIN Conversion
Files - small files containing instructions that can be used to
tell BAIN how to repackage an archive into a format it can
recognize) into converters.py
318e66a9ad3a096c316decc751bc76d6c3b5b858 and
75c48e654c21feccd52cd851f2b502fd7c41c45f moved some bosh contents into
parsers.py and bass.py - most importantly, the very commonly used
'dirs' and 'settings' constants.
ea242af573952328168759b9c9c9ff1c01ac681c moved various miscellaneous
things mostly related to plugins and LOOT into a new mods_metadata.py
file.
Coinciding with the INI refactoring (see 'INIs' section below),
6d2a1fa7bcb15f3d5fe787b5475e904e79809253 moved the INI handling code
into bosh/ini_files.py. Similarly,
afb7058a8e0d8a4567d43ec8f6f8828cafc34b83 moved BAIN code into
bosh/bain.py as part of BAIN refactoring (see 'BAIN' section below).
A last few commits splitting out the saves and mergeability checking
into their own files (bosh/_mergeability.py, bosh/_saves.py and
bosh/save_headers.py) were made in
347c552c8bbd3506a5bacd6c678a92e1275c8e06 and
aa82a7af6fbcc6ce24ac1a940d3c2d4f665d626b, closing the issue for now.
bosh/__init__.py is still quite big (3427 lines), but splitting further
is far from trivial.
-----------------------------------------------------------------------
### Fallout 4 (#251)
On November 10 of 2015, Fallout 4 was released - and less than three
weeks later a branch by @lojack5 adding basic support for it was merged
in e85506af4fbd28c4617b46c9ef0832c6c6570ecb0. Unfortunately, not all
game merges in 307 were this speedy ;)
Some more improvements and fixes landed in
134d7c63839de3daa52108f7ce9cc02fcc928df8 by @Sharlikran.
Fallout 4 support was barebones then and is still barebones now. The
real followup work here will come in 308/309 (see #525 and #482).
-----------------------------------------------------------------------
### Dropped Support For Older Settings Files (#253)
Various bits of backwards compatibility cruft had accumulated over the
course of 306 (and even earlier) to keep pre-306 settings working. 307
dropped all this and instead shows an error message if pre-306 settings
are loaded in 307, telling the user to resave them in 306 first so that
they can be upgraded to the new format. All this was done in a single
merge, 8ceef20f269aedb18dbffe17063fac1cb7ccc0f4.
With how big 307 has become, it is probably no surprise that it too has
by now managed to accumulate a metric ton of backwards compatibility
hacks. Cleanup for those will follow in 308/309.
-----------------------------------------------------------------------
### env.py (#258)
In the ongoing quest for native Linux support (#243), encapsulating
OS-specific code (i.e. Windows-specific code) in its own module is
an important step. This was done in
646df2ca063c2dff03cb2185e7e2969fdea065a5. We're still not really there
yet and env.py is, for the most part, still a Windows-specific module.
The end goal would be to have two separate backends from env.py, so
that it imports from one of them based on the OS it is running on.
-----------------------------------------------------------------------
### BAIN (#219)
307 also marks the beginning of the grueling task that is refactoring
BAIN. Still nowhere near done, but it has become manageable. Much more
work to be done in 308/309. Some of the important targets were:
refreshSizeCrcDate was a big method that handled scanning directories
for their contents. The problem was that it was basically two methods
in one and implemented things that only make sense in one specific
directory, namely the Data folder (e.g. empty directory removal).
20e41b3cfc38c71e8baae6e34778b964ca809533 refactored refreshSizeCrcDate
to prepare it for 5ecf4000f10383e3018b4a4febbf8a8f13164a9e, where it
was split into two methods (_refresh_from_project_dir and
_refresh_from_data_dir).
refreshDataSizeCrc is one of the central parts of BAIN. It scans a
package for its contents, applies skips, remaps documentation to the
'Docs' folder, queries and caches CRCs, etc.
ea79bb4fdfe2318d84952ba63fec2fb9b2a7e33a and
5fef1e3924bce2edcf19ff41e469c2a846d851bb tackled this behemoth,
reducing its size significantly. Various later commits like
54844fbe1ea4ddfd48128e61f4232439431aac45 and
c48112237b9953effb74b4206707394f225f035b touched this one up even
further, bringing its final size in 307 to 210 lines (down from 348
lines in 306).
5d435bb05b9e09340eef9a71a402ac018abcb573 and
5ebbf4e8e3646efa8019b9cbccb144be9bc0ead9 focused on centralizing and
refactoring BAIN refreshes, most notably the irefresh method.
b109a4d08f21e023a30d5c0bf4ed45def0ae26e1 made BAIN use the modInfos
cache to avoid recalculating CRCs that we already calculated for the
Mods tab, giving a minor speedup - but mostly the idea here is we want
to read the Data folder once and then delegate the files to the various
FileInfos based on name and/or extension. This is a first step in that
direction (see also #353 and #265).
b1711b1b52fe76f0463b47fab825058f0f3bc41e was an important merge
addressing case insensitive string comparisons. BAIN makes heavy use of
this to keep track of files in the Data folder, each package, external
changes, etc. So a central dictionary that made case-insensitive
comparisons of its string keys was necessary - which is exactly what
was introduced in this merge, as bolt.CIstr and bolt.LowerDict.
545cad09d29cf4b17ed470caba974829f5876d7f refactored the conflict
detection and reporting algorithms to, well, separate them in the first
place. They sat in a single big method called getConflictReport which
came in at 120 lines pre-refactoring. In addition, the fact that this
method returned a single string made it impossible to improve the GUI
for conflicts (a goal in 309) without parsing the string - the string
we just constructed from in-memory objects. The new getConflictReport
is 55 lines long, much more readable and makes use of a new
find_conflicts method that can be used to retrieve the actual conflicts
as in-memory objects to work on.
-----------------------------------------------------------------------
### basher Package Followups (#163)
Began in 306, finished here. daafdd5e76746833afe4eba496aa4afac41ff439
dropped the ancient Tank class for good, curing the flickering on the
Mods tab in the process (#179).
More work done in b7ffca085feb4afe93e48cdafb68521fd9402899.
a28fcfce1f59482d1e701504b738a8df26a9bfe4 is a joint merge by @Utumno
and @DianaNites refactoring the mod export/import links, which were
pasta-filled mess. More prerequisite work landed in
10703a9e2982ae28db60af72146de8d8dca9c324
The topic was finally laid to rest with
fb37a8bf110b6953c07a61a470d05b73ab03eb17 and
3bf6006d842942214694547a3e4c2b2fccb71dee. Every tab now has neatly
separated tab panels, UILists and details panels. The API could still
be better (especially considering how many new tabs we have planned
for 308+ - see e.g. #233, #456, #50, etc.), but it's a far cry from
the situation in 305 and 306.
-----------------------------------------------------------------------
### Skyrim Patchers (#151)
Another principal goal of 307 is getting Skyrim patcher support as
close to Oblivion as possible.
Some prerequisites were merged in
3a2d396b2a9cf2ecd2525006a4e9960f3b4c85ff and
1e18dad44b97bf80b115de0c7175d14df07f2c3f (mostly records).
The first real patcher porting then happened in
9599368dee429f4b63f116d25945beae73d838db.
-----------------------------------------------------------------------
### Load Order (#295 and #309)
Load order handling is a complex beast, to say the least. Not only is
handling all the edge cases difficult, but there are three different
methods that the games use for implementing load order (four if you
count Morrowind, which we don't support - yet ;)).
Wrye Bash originally used the BOSS API for managing load order. This
was changed in 1bf84f3e3b246195b93d9715e2d1891decc47354 to instead use
the dedicated libloadorder. Even libloadorder itself proved problematic
however:
- Adding support for new games was tough and quickly turned into
'adding more clauses into if-else chains'.
- The API generally made no effort to keep actives order and load
order together - which became untenable in Fallout 4, where the two
are inextricably linked.
- Wrye Bash needs to read all the plugins anyways (e.g. CRC, ESM flag,
etc.) and keeps that information cached in bosh.modInfos.
libloadorder was reading it all again which, on top of being
inelegant, was thrown away performance - syscalls are *not* cheap!
- It's a DLL - bad for git and won't work on Linux (#243)
So, in one big merge, 66d7b4d695289f6dc29142f94a6004494e3306bd replaced
libloadorder entirely with a new API written in pure Python. This
enabled many new features such as locking the load order in all games
(bc7e5de47f3ecf31d1290d436940a37a7b6eb0cf), automatic backups whenever
we make a fix to the LO (38817bb5ad6e222c4fdbde270d10341b2371bdb5),
etc.
-----------------------------------------------------------------------
### Patchers (#312 and #461)
Porting patchers to newer games is going to become harder and harder
unless we refactor them to make it easier. Patcher code wasn't
*terrible* per se and was fairly isolated from the rest of the code
(apart from its tight connection to records code - see 'Records
refactoring' section below), but a lot of it was copy-pasted and hence
difficult to understand and expand - plus bugs often had to be fixed in
upwards of 10 places. Adding a new patcher easily required hundreds of
lines of pasta. This is still an ongoing goal, with much more to come
in 308.
30eda2dd5c987648a11fbe01b8ee1b6c56d7c1ac was an early merge, containing
refactoring on other more or less related things.
122784f5cf4d737bbb5943e9bd395d2bfc53c43a then moved config handling
(i.e. which patchers are active and which sources they are operating
on) to basher. The idea is to have the patchers operating only on a
list of sources - not only for elegance and simplicity, but also to
make them testable (see the 'CI & Tests' section below).
Heavyweight refactoring began in
66d7b4ef3f8959e180f0029874c73e40de2a5a52 and
f8bfdc4c5894ca2832aa5aef6515d70f52df110f, which introduced the
_SimpleImporter class to deduplicate a lot of copy-pasted
implementation code. Keep your eyes on this class, it proved to be a
good idea ;)
eb110497c3ce7b9d72df5a6565a0c7efffbc9a28 and
604ebd31d2f20d26c06ab2d043f114f7e0abc26b went in a different direction,
by using the work that had already been done on refactoring the
patchers to port many of the Oblivion-specific patchers over to FO3,
FNV and Skyrim, as well as add some entirely new patchers that had
been commonly requested. Of course, this involved a bunch of
refactoring too, mostly moving implicit constants from all over the
patcher code into game/*/constants.py, which will make it much easier
to port the patchers in the future (e.g. to Fallout 4, see #482).
10e680cbb347b203f935c042145a5fc308b5f4c8 returned back to good
old-fashioned refactoring by decoupling the config/GUI side of the
patchers entirely from the model side (i.e. the code that actually
implements patcher behavior). Previously, patchers were linked
together by importing the patcher implementations in the GUI and using
them as mixins. This led to lots of weird, hard-to-debug code that
crippled the IDE's ability to perform static analysis. For example, the
implementation of the leveled list patcher would use
if not self.remove_empty_sublists: return
to skip the 'empty sublist removal' part of the patcher if the checkbox
for this was not checked in the GUI. However, the IDE had no way of
knowing that that variable actually existed, since it came from the GUI
side of the code via a mixin. After many failed attempts to devise a
base class for both the model and GUI side of patchers, @Utumno instead
realized that a much cleaner design would be to have no mixins.
Instead, each patcher's GUI panel now has a class variable called
patcher_type, which it sets to the model class that gets imported from
the `patcher` package. This allowed us to drop tons of boilerplate code
and make the resulting code much cleaner and easier to understand, but
most importantly it acted as a springboard for further refactoring.
Most notably, 9697b64abc00beaed04e950af20c8116db15151a split our
importers.py file into four files: _cbash_importers.py, _shared.py,
mergers.py and preservers.py. The key insight here was that we can
split our importers nicely into two types: preservers simply carry
forward the last value(s) from a tagged mod, while mergers merge values
from all tagged mods based on the tags those mods have applied.
This resulted in several hundred lines of duplicate patcher code being
chopped off due to us absorbing many preservers back into the base
class. It will also make it much easier to drop the CBash patchers,
since they are now in a separate file altogether (see the 'CBash
Deprecation' section below for more information on this).
One last merge worth mentioning here is
426db77ee71c939bb785b94eab7d4281f0f1fa26. It is a highly WIP attempt to
tame the mess that is parsers.py by devising a proper base class. The
savings so far do look promising, but CSV reading and writing are ugly
warts that still stand in the way. Plus the base class might be too
complicated - right now it has six different knobs that tweak its
behavior, and it's not even clear if using it for the all parsers is
feasible. Still, we had to merge since previous betas came out with
the plugin export/import commands this merge ports to Skyrim.
Much more will follow here in 308 - most notably an upcoming
refactoring of tweaks that will make them *much* faster and drop ~1200
lines of duplicate code.
-----------------------------------------------------------------------
### INIs (#247 and #326)
INI handling was spotty at best. Random unicode tracebacks kept showing
up, the INI Edits tab was one of the last big performance hogs and the
default INI tweaks being files in the Mopy folder led to confusion (at
least one mod had a 'Mopy\INI Tweaks' structure and was supposed to be
installed *into the Mopy folder*).
The first step was df2fcc5f95bc7bc2613a14cb996671f3b82dd2db, a series
of smaller refactorings and fixups to make the tab's code more
manageable.
1d4c23a037f6845f5dfebaf1e5e005f8f121a63b began the work on performance
by introducing the proof of concept for a cache, while
58c47d562f43773c5b017531176b8869138869bf attacked the refresh APIs used
by the INI Edits tab and significantly reduced the number of syscalls
it made.
The default INI tweaks were finally dealt with in
656127c645070d35d31423e014b692621ff94015 by hardcoding them into
Mopy/bash/game/*/default_tweaks.py. No more tampering with the tweaks,
no more Mopy\INI Tweaks folder to confuse users, fewer loose files
packaged, simplified INI refresh, better performance due to fewer
syscalls...
ef21c5bb0a4f23d737dde3b22af36dcaf4f9b58b contained some more work on
centralizing the 'apply a tweak' logic and fixing a longstanding issue
where INIs would have Unix line endings written out, even on Windows.
The LowerDict introduced during BAIN refactoring (see 'BAIN' section
above) also turned out to be very useful for INIs:
a9112d6761e47b1fc41aca53df1add5bdd41ad79 used it to rewrite core parts
of ini_files.py for performance and readability.
bb6c8bd2e7ac22d1218cbad90e404b9739dba7bc reworked the handling of INI
encodings based on a central principle: work with unicode and stripped
newlines internally, encode/decode and add/remove newlines at IO
boundaries.
-----------------------------------------------------------------------
### wxPython (#190, #15 and #488)
A war that started before living memory and will continue until long
after we're all gone - or will it? Actually, we're very close to
winning this conflict for good!
wxPython was all over the place in 305. 306 improved the situation
significantly, but 307 puts even those efforts to shame:
305: 2260 usages (balt: 381, basher: 1586)
306: 1112 usages (balt: 418, basher: 494)
307: 207 usages (balt: 191, basher: 8)
*8* direct usages in basher, down from 494! Let's see how we got there:
- 114c83729aa50045cac4f381912007826d8048bf: Utumno vs wx. Utumno lost,
of course, but wx usages did go down from 1075 to 923. Mostly
accomplished by moving common code to balt.
- c7f5f5085acf1ec6e55f0048c256f7a0ebc3f367: Down to 902, and some
progress was made towards wxPython 3.
- 69c7f9f679f48df8cf7562442e80c9129f10924a: wx.lib.iewin was an ugly
beast that was binding Wrye Bash to the comtypes dependency.
Unfortunately, dropping it required upgrading to wxPython 3 (see
below), so this merge simply centralized the iewin import for a
future removal.
- 531679d37d6c50cfc030b9c462f44250bcfab7a0: A small merge containing
backwards-compatible changes that brought us closer to wxPython 3.
- 050391ca22d7c8451390cbff6fd150ab0b9bcabd: The upgrade to wxPython 3.
Introduced several significant architectural achievements:
- Dropping the comtypes dependency by rewriting our HTML rendering
code to use WebView instead of wx.lib.iewin.
- Removing bolt's locale-related behavior on import that made
importing it dangerous - it's been encapsulated in a new
top-level module, localize.py.
- Rewriting a lot of the very early boot process - see also the
'Boot' section below.
- f9e46eed670b3d2805b1570fc62daf9cca12ce79: The big one. An absolutely
enormous joint merge by @Utumno, @Infernio and @nycz introducing a
new package, gui, that truly encapsulates wxPython. nycz wrote the
first version of the code back in 2017, most importantly the layouts
code that encapsulates wxPython's sizers in a declarative API. We
then devised an event handling framework that enabled us to hunt
down a lot of *implicit* wxPython usages. These are much harder and
nastier to track because they can't simply be regexed. Thankfully
PEP8 will be able to help us here, since all of wxPython uses
PascalCase for its methods, while we're using snake_case for all new
code. The result is a reduction down to 218 direct wx usages outside
of gui.
- 22de7ff9e804b2bcaa8a819922f4b5100b86d80f: After upgrading to
wxPython 3, the next goal was upgrading to wxPython 4. This is also
the first release of wxPython that supports Python 3, making this a
significant step in the direction of py3 support (#460).
- eb86a4cb35fc24b58288a6b3e917ed9350a02e23: In preparation for
finalizing and merging the FOMOD support (see 'FOMODs' section
below), a bit more de-wx'ing happened, mostly on radio buttons and
the splash screen.
-----------------------------------------------------------------------
### BSAs (#339 and #338)
Same story as libloadorder. We were using a binary, libbsa, to do it.
This was thrown away performance (we already read and cached the BSAs
in bosh.bsaInfos), had no Linux support, etc.
Its replacement, bsa_files.py, was introduced in
b199a7bd5eec69ec0852f86d6e3629784e6b90ce, then used to support strings
files packed into BSAs in c4f12d56d1273f40096abe45b6803d9265b1e75e and
finished in 8ce81bfc9b5dc4643988e54471fab3e9b6a1f72d.
With BSA handling code now taking shape in the form of bsa_files.py,
having a second class arbitrarily handling a few things with entirely
different (and much uglier) code would be a bad idea - so
d69f3e82c3afe4b6461b6ccce49a210bedd28dd6 dropped the ancient BsaFile.
There were still some unimplemented parts of the BSA format:
- TES3 format: Added in 4ed5bd8c4ed9a67f872f109c660cf0afeb6ddca5,
also in preparation of the POC Morrowind support we have in 307 (see
the 'Morrowind' section below).
- Compressed BSAs: Added in a6e11c4601d788e0e25c17361d92eaa659e59768,
including both lz4-compressed ones for SSE and zlib-compressed ones
for all previous games.
- FO4 DX10 format: Added in 903b6d11d1451855951cce608c0ce8fe6a23743f.
Currently unused, since we only use BSA extraction for strings
files, which the DX10 format can't contain (it may only contain
textures, for which it is specifically built and optimized).
- Writing: We can read and extract everything from Morrowind to FO76
now, but have no support for altering and writing out BSAs yet.
This will be a goal in 308/309 (see note below).
All this work on BSAs acts as a prerequisite for the BSAs tab we want
to enable and expand in 308/309 (it already exists in the codebase, but
is very unfinished at the moment) - see #233.
-----------------------------------------------------------------------
### FileInfo(s) (#336)
At the heart of each tab sits a DataStore subclass. This provides the
UIList (i.e. what you see on the left side of each tab) with the data
it should show. Most tabs (all but People and Installers) then have
TableFileInfos in the hierarchy, and all but INIInfos (which backs the
INI Edits tab, unsurprisingly) then have FileInfos in its hierarchy.
These FileInfos use FileInfo classes to represent the files that are
going to be shown in the list.
These APIs are not *bad*, but they're not *good* either. Refactoring
this is an ongoing goal, with lots of work done in 307.
b199a7bd5eec69ec0852f86d6e3629784e6b90ce devised a common API for
representing a tracked file with caching called AFile and used it for
the new BSA API (see 'BSAs' section above). Some further work on
freezing the AFile API and making FileInfo use it happened in
11f6769f641e80552b6f2d6e3c25d49a85e66c63,
e3064942119c504786cbe6befafd8a0aa38bec2e and
a0ae08c42fdf47688870387a6f6296de0551bfc2.
Cosaves got a lot of work done in 307, bringing them from pre-alpha at
best to a solid beta API (see the 'Cosaves' section below). Screenshots
also got reworked to use the FileInfo(s) APIs in
e007a5308d509fcd5e123d566b2b3c24e228edaa and
9a317f62cc8f7f807581a0a5f5453637d5d8b5ff.
-----------------------------------------------------------------------
### Skyrim SE (#347)
In a join merge by @Arthmoor, @Utumno and @Sharlikran, Wrye Bash got
SSE support added: 76b8abd5437042dd5b1f1e4505a651211d5523e8.
See the 'ESLs' section below for some of the following challenges with
SSE support. Of course, patcher support was spotty at first too - we
ported all Skyrim patchers to it in
c076e9fe1f5a27746a11b91ba9e0e1b43b80a915.
One more merge worth mentioning here is
064d5021e068260cc99225be29d8d651b0761c61, which sped up startup in all
games, but most notably in SSE. Our reference setup we used for testing
(with ~200 saves) went from 10s down to 2s.
-----------------------------------------------------------------------
### Boot (#373 and #390)
The boot phase was nothing short of a mess. There was no clear guiding
principle of what is initialized when, leading to hard to debug
problems. Unexpected errors could take down Wrye Bash for good without
any way to tell what the problem even was. Restoring settings wasn't
working at all. This is still somewhat in flux just due to how complex
the boot phase is, but it's definitely gotten better.
0e3ef608e2906afb3405b009c622c632caa21dab began the process by
centralizing the wxPython import. In
6060a157be13372ff2a8e09c8de9878c16190f2a, @D4id4los rewrote core parts
of the boot procedure to gracefully handle and show errors, even when
not in debug mode - making fixing startup errors encountered by users
much easier.
Restoring settings was addressed in
17f2266525e5095f8c068804fe3cc3471c9bac1a. In short, when restoring
settings, we would override the settings we just tried to restore due
to our atexit hook firing immediately afterwards. Instead of hacking
away at it with monkey patches, @Utumno carefully reworked the boot
sequence to clearly lay out what gets initialized when, breaking barb's
dependency on the rest of Wrye Bash (balt, bosh, bush, etc.) in the
process and adding a new top-level module, initialization.py, to better
encapsulate init procedures.
The locale mess that bolt did on import was addressed during the
wxPython 3 upgrade in 050391ca22d7c8451390cbff6fd150ab0b9bcabd. This
also resulted in a much more well-documented early boot process,
including setting up the BashBugDump and bolt.depring much earlier,
allowing us to use it to consistently log during the entire boot phase.
-----------------------------------------------------------------------
### Cosaves (#437)
xSE (i.e. the script extenders - OBSE, SKSE, etc.) create cosaves for
each save you make. Wrye Bash originally only needed these for its
master remapping feature to not break things, but over the course of
307 we've come to use them to display save masters with ESLs in them
(since those saves store two separate lists, an accurate master list is
only possible by looking at the PLGN chunk in the cosave). They present
a unique challenge in that each cosave is attached to a regular save,
and all operations on that save need to respect the cosave. That means
renaming, deleting, backing up, etc. need to not just apply to the
main save, but also to its cosave, if it has one.
0a300af01a85bb3535d3194203fd34cc0d3e9b29 introduced the initial API for
this, bosh/cosaves.py. It was mostly just a collection of code from
various parts of bosh (mostly _saves.py), and as such was difficult to
understand, maintain and extend.
822c0bd16e5788d1811449810f34edfee16d77a1 then refactored it (the commit
looks like a rewrite, but was actually a gigantic refactoring
comprising 100+ commits that had to get squashed down to a single one
in order to not break dev) for maintainability and to bring Pluggy
(an ancient cosave format in Oblivion) support into the cosave
hierarchy. It also added support for saves with ESL masters, as
mentioned above.
Finally, e4dc76995703f16ee97fb07d495b6db635a2c6a8 reworked our handling
of cosaves to be much more robust
-----------------------------------------------------------------------
### ESLs (#382 and #429)
ESLs are a new type of plugin file introduced with SSE and FO4. They
present a unique challenge in that they can bypass the usual 256
plugins limit, and as such stress-test many central assumption in any
tool that tries to support them. Our APIs stood well to the test
however, with d507111773d459e41468ac835955fe88915e622e only having to
make minimal changes to add initial support.
Due to the limited understanding of ESLs at the time, we were very
conservative in what we allowed users to do with them. There was no way
to verify ESL flags, add and remove them, and the Bashed Patch excluded
ESLs completely. That was fixed in
547565a32f8d1d4a2944984c8c29092834f4fff6, a join merge by @Sharlikran,
@Utumno and @Infernio. With it, Wrye Bash gained the ability to add the
ESL flag to ESL-capable mods, importing from ESLs into the Bashed Patch
was reenabled and load order operations for ESL-flagged ESPs were
fixed.
Once again, we were quite conservative in implementing the ESL flagging
in Wrye Bash. For all record types we had not decoded yet, we simply
failed the verification and told people to use xEdit to check instead.
c137405418360063b6d767e7179c31fa94936cbc changed that to use a generic
method that does not rely on our record definitions, since the only
thing we actually have to care about are the headers of all records in
the file. The contents of those records do not matter. This made
ESL-flagging both faster and completely accurate for all record types
in both SSE and FO4.
-----------------------------------------------------------------------
### Game Handling (#358)
Along the way, especially after adding initial ESL suport (see 'ESLs'
section above), it became clear that Wrye Bash's game handling would
become a big issue that needed addressing. Adding support for a new
game involved dozens of edits all over the codebase due to fsName
checks and copy-pasting and editing a big constants.template file.
This cripped the IDE's static analysis, since it couldn't check that
any given game constant existed, let alone had the right type. We were
also importing way too much for each game (e.g. all the constants, the
default tweaks, the vanilla files, etc.), when we should really just
import them for the one game we're actually managing. Thrown away
performance and memory, plus just plain inelegant.
In a joint merge by @Utumno and @GandaG,
11fa0f6a71ca8420071e76357b89f5d3e221c904, the game constants were moved
from module-level into classes, allowing them to be inherited. This got
rid of tons of duplicate code (1229 insertions(+), 1824 deletions(-))
and made adding both a new game and a new game constant easier and less
error-prone.
Some more work to move game-specific constants out of random files and
into the game/*/constants.py files they belong into happened in
fa74a1b7a61d9b3150f0d2b171145e171f2d27e5, along with some fixes in
06d6a6bdaa925f379ffc1f05d0fa5977057ac739 and
57d3a621a08f4852dc5d5cc36878db9286351579.
4050e60bd372494c46860c85fa15c1701abcc5ae devised a way for us to avoid
importing the constants, default tweaks, etc. for every game, while
ddda9393d9b08a132c4a346fbdd8cc84454bccbd and
0fecde47b73d3204735c28b37260b7af8a01f700 finally finished off the last
few constants outside game/*/constants.py, closing this issue for the
time being.
-----------------------------------------------------------------------
### Fallout 3 & New Vegas (#150 and #468)
Not originally planned to be part of 307, but after it was accidentally
included in Beta 3, we had to merge it:
0f06e4fd306684aafccd2764b47a66d2205d10fc
@valda originally ported Wrye Bash to FO3/FNV as Wrye Flash. Efforts to
backport the changes to WB had been dragging along since forever, so
the main thing we learned from this merge was that leaving games to rot
around in branches is a *terrible* idea. Better to have the WIP code on
dev without explicitly providing support, as leaving it on a branch
makes it accumulate subtle bugs from refactoring extremely quickly.
Some work on synchronizing the FO3 and FNV versions happened in the
form of 54b8e614acd0841460f550d33d23340824f06e31 - since FNV is a
vastly more popular game when it comes to modding, many of the
improvements that valda made to the NV version of Wrye Flash did not
make it back to the FO3 version. With us being based on a single
codebase, doing that is much easier - eventually culminating in the FNV
constants being entirely deduplicated in
3f96076501d5c260af856dac1f6475c21aa53a6e,
e68b7af1eb9a3eed96b72cda067be82d86e067c6 and
dccd28c70ffb857ccc70265a5ca22ca718d8e442 so that they are based on the
FO3 ones, meaning that adding e.g. a new patcher or bash tag to FO3
will automatically add it to FNV as well.
We're almost at feature parity with valda's version now - only the race
patcher is missing from our version. This will be addressed in 308/309.
On the other hand, we support several patchers and tags that valda's
version doesn't, on top of tons of other features and bugfixes (see,
for example, the rest of this commit message ;)).
-----------------------------------------------------------------------
### Readmes (#432 and #464)
307 includes a significant reworking of the readmes, courtesy of
@FelesNoctis in 493c76b38c760d31757657a5b9bcf70f196d1dcd. It was later
followed up with more edits for maintainability and to update the
screenshots: see 18969116f7547148bf7b66196f91accd1225b465 and TODO
-----------------------------------------------------------------------
### Wizards (#446, #445, #444, #436 and #189)
Several issues related to wizards were fixed (see e.g.
169d8347c1e4f3a3f6d696d4a660f89987f6bffc,
1f71bf335b85276566c12db43b53097842e05981 and
30f698520be938cd3cb6aa950cf979bc5468edb6).
We also refactored the code quite a bit and deprecated the old 'Espm'
versions of keywords and functions in favor of new 'Plugin' versions
(more intuitive, easier to spell and remember, and more accurate with
the advent of ESLs) in 3426384083bd5d7c61d42730ed7fa9c5629bc2db.
Finally, 9244f8536ff0e4b780e3190cc40a032771310f4c added a new wizard
function that had been requested a long time ago, enabling wizards to
alter their behavior based on a plugin's load order.
There is still a lot to do on wizards. For a start, the format is not
formally defined - and the parser that acts as a reference is quite
buggy (e.g. `Note thisIsAString` will print out 'thisIsAString',
because the parser gets confused about its token states and
accidentally treats 'thisIsAString' as a string). See
https://github.com/Infernio/wizparse for my POC attempt at defining a
formal grammar based on ANTLR that other mod managers will also be able
to use. This is low priority, but will be continued in 309.
Additionally, the wizard GUI presents a significant challenge in
de-wx'ing (see 'wxPython' section above) and has a lot of duplication
with the new FOMOD GUI (see 'FOMODs' section below).
-----------------------------------------------------------------------
### Python 3 (#460)
We officially started the process of porting Wrye Bash over to Python 3
in September 2019, seeing as Python 2 has reached its end of life. This
has turned out to be nothing short of a giant can of worms. Wrye Bash
makes heavy use of bytestrings, so simply letting 2to3 run over the
codebase would be disastrous - we'd be fixing unicode/bytes tracebacks
for the next few months and getting no actual work done. In addition,
our policy of having no breaking commits on dev means that an eventual
Python 3 port will have to be a single commit, which makes bisecting
useless. So the result is that we need that py3 commit to be as small
as possible. With that goal in mind, a lot of prerequisite work that
brings us closer to a py3 port without breaking py2 has landed:
- 660ecbb81f49d478bdc8e4a8905e328d1daf9dca: py3 has no 'ur' prefix for
strings since the one in py2 wasn't actually a 'raw' prefix:
>>> print(ur'\u03B3')
γ
So dropping this one from the codebase was necessary. This commit
just dropped all usages in strings that didn't actually have
backslashes or were autogenerated paths (i.e. vanilla_files).
- d43ad244170e2110a6daca7d5febed4020550247: This commit by @syntaxaire
finished off the 'ur' removal mentioned above.
- 5a98eb4c025651f4e9366db2a7d488ec2068f1fc: cmp and __cmp__ do not
exist in py3. For the most part, we just had to implement rich
comparisons.
- fa74a1b7a61d9b3150f0d2b171145e171f2d27e5,
cae844b9dde8af014b09a1cb24af2348d5620058 and
6db5b8b59e28bc46a9d42e966d31007e113c59e6: Changing old-style classes
to new-style ones work fine, except when the class is used as a
mixin with a new-style one that uses __slots__. That can lead to
nasty layout conflicts, as seen in the first of these three commits.
- 8e201c49bbc809da89b1bda1d269f4cb7619dfc0: Our codebase included an
ancient version of chardet (1.0.1 from 2008) due to a single manual
edit that was needed to make it avoid returning the EUC-TW encodings
that Python doesn't support. We dropped it in favor of the PyPI
version, and addressed the EUC-TW problem in
60d0c29dbae91c12c1f7825df9f4e8e243ca09d2.
- d8d03ca39e1e9f85250fd014cabcc2a65945e5e7,
197b6a2de78acd723f9d747fd6751fd2c68cf944 and
659e5b696be5083b9bef0d39356acc30ab46b5a4: Long integers don't exist
in py3 due to its int type having no max size. So we needed to drop
all 'L' postfixes and usages of sys.maxint.
- 664f1722a53c91794f192e343936dfd34b8e86a8: Fixes for various issues
encountered during an experimental run of 2to3 by @lojack5.
- 33eac7624971ecd22e1f65ff5e47bc71ca175dbc: Merge by @GandaG
addressing various py3 issues like print, moved stdlib modules, old
exception syntax and usage of local absolute imports.
- 050391ca22d7c8451390cbff6fd150ab0b9bcabd and
22de7ff9e804b2bcaa8a819922f4b5100b86d80f: We were stuck on wxPython
2.8 for a long time, but the first version of wxPython that actually
has py3 support is wxPython 4. These two commits (as well as tons of
prerequisite refactoring, see the 'wxPython' section above) cleared
that blocker for good.
The Python 3 port is one of the primary 308 goals, along with patcher
refactoring (#312) and records refactoring (#480), on which it is
blocked (due to the aforementioned heavy bytestrings usage, which
those refactorings will help us isolate and encapsulate).
-----------------------------------------------------------------------
### Build Scripts (#415)
An enormous productivity gain for developers came in the form of
@GandaG's reworking of build scripts in
a1b5bfaa40fdbe04549ba3775106ffdff471e62e and
5f31a2adf39a607db282f49bd43e66c992a1baad. The ancient
package_for_release.py has been replaced with a sleek new build.py
script that does everything you need to do to build Wrye Bash in a
single invocation. On top of that, Ganda also dropped tons of weird
legacy things the build scripts did, like using ResHacker.exe to set
the Wrye Bash icon - more binaries gone <3
-----------------------------------------------------------------------
### Enderal (#433)
Support for the Steam release of Enderal: Forgotten Stories, a total
conversion mod for Skyrim was added in
c2d73965fba4d0a82bb95f7cbe13b7f5dbcc0155. This was a fairly simple game
to suport since it is pretty much just a pre-modded version of Skyrim
LE. Of course, the work on refactoring game handling is the reason why
we had so few problems with this merge. Once again, we left this game
too long on a branch, meaning it began to accumulate bugs. That
necessitated a fixup commit almost immediately in
3afa217d987c8c4ab86341ed8a8ec906b099767a. For all future game merges,
we resolved to merge more quickly, as long as adding support for the
game doesn't break any other code.
-----------------------------------------------------------------------
### Records (#480)
After all the above, there were a few spots left in the codebase that
needed *heavy* refactoring: records, patchers, saves (*not* save
headers, the Oblivion-specific save editing code) and BAIN. Since
the records and patchers code are very closely intertwined, they need
to be attacked in tandem (refactoring the patchers is sort of a
'top-down' approach, while refactoring the records is a 'bottom-up'
approach to the same problem). See the 'Patchers' section above for
more information on that refactoring.
The whole shebang began in ecac15d01dc5c89463ad47aab74260abfbea4167 and
3a3e9c935f5c1a798211eb0eaed0a0dd76a9af24, which were mostly just random
commits improving some record definitions.
Heavyweight refactoring began in
134433fde71534fc09e357ad64c696194f51a8eb, which moved an awful lot of
records code into brec by creating new tools for defining record
definitions. The result is a massive reduction in code size:
6787 insertions(+), 9581 deletions(-)
..and a very nice situation where almost the entire *implementation* of
PBash sits in brec, while the (almost) purely declarative definitions
sit in game/*/records.py. Unfortunately, this bloated brec to 3000+
lines and made it much harder to tell which classes belonged together.
This was addressed in 28c11cb934056790e2a07703a9a09b4a5de8aa48, a huge
merge that split brec into a package, added OBME support to PBash, sped
up plugin loading by using AOT construction of struct.Struct instances
instead of struct.pack/unpack and implemented merging of all record
types in Oblivion. That's right, PBash can now merge everything CBash
can. See the 'CBash Deprecation' section below for more information.
After reading both the 'Python 3' and 'Patchers' sections, you should
already know what's coming here: much more in 308. There's already a
large refactoring ('part 2.5' of #480) that just needs some testing
before it can be merged, and @Utumno is working on a 'part 3' of #480
that will seriously turn some parts of the records code on its head.
-----------------------------------------------------------------------
### Usability and Accessibility
Wrye Bash has a (not entirely undeserved ;)) reputation for being
difficult for newcomers to get started with (in UX terms, we'd say its
out-of-box experience is bad). While this is obviously a big goal that
we're nowhere close to solving today (really, someone with actual UX
experience would be needed on the team), 307 does include some work
towards both this goal and the goal of making Wrye Bash accessible to
everyone, regardless of disabilities.
- 1bf488a10a4c9fedb2737e4b2eee86c484f7b93d: The ability to jump to a
plugin's matching installer from the Mods tab has been added (#53).
- 3d7b9816ec0e2b7d666a7bead0cd45db5347aed7 by @fireundubh added the
ability to jump to the matching plugin when a master is
double-clicked in a masterlist (#311).
- 261a1029a78e2cae773386c4e41f496a05f018c0 and
f4987d1d0db38e4379f6470f67c37b982192817f by @BeermotorWB and
@MacSplody trimmed the jungle that was the package context menu on
the Installers tab by moving the more rarely used commands into
submenus.
- f65112bea111157558f78f056b550a7f689752a3 added the ability to jump
to a plugin from the Plugin Filter on the Installers tab.
- 92691409567ce020c6958325ee8c4bd8c9e820cc made the 'Sort By',
'Columns' and 'File' submenus on each tab consistent by putting them
in the same places (they were in seemingly random positions on the
context menus before).
- 206ffbd9b09a0b8107ac5dacd9b0b3f9c2d1bfc2 by @warmfrost85 allowed
users to get a preview of what Clean Data and Sync From Data are
going to do, as well as the ability to use that preview to change
which files the commands will affect.
- e5685f3e87abd0bce199fe619509a9648ce2db89, also by @warmfrost85,
allowed Sync From Data to work with archives. Previously if you
wanted to, say, clean a plugin and sync it back into its archive,
you would have to either do it manually in 7zip or use the
workaround of unpacking the archive to a project, then using Sync
From Data and finally packing the project back into an archive. Now
you can just do it all in one go by using Sync From Data directly
on the archive.
- b427bbd383688a2f0fd55266b040cdc6ddf7c590 and
a2297331b6571b6989bf1ea3b1c253a85c834448 increased the contrast on
all our checkbox images to pass WCAG AAA guidelines, to make it much
easier for people with weak eyesight to use Wrye Bash. In the
future, we want to allow people to customize the colors of the
checkboxes so that colorblind people can make use of them too
(see #511).
- edbe5e51d294ed0706478a9f8896d1e170d76058 added a menubar to improve
discoverability of Wrye Bash's column context menus, as well as
making it possible to access them purely using a keyboard.
Previously you would have had to right click one of the columns to
access these commands and options, which is a bit arcane and doesn't
work for people who can't use a mouse at all. The same merge also
reworked our half-baked settings menu that was implemented as a list
of popup options when the tiny gear icon is clicked to instead be a
proper settings dialog. Not only is this much easier to navigate, it
will scale far better for our future needs (e.g. extensions).
-----------------------------------------------------------------------
### Morrowind (#479)
Morrowind is not officially supported in 307. We've added some (very)
WIP code in 45ed499de6b5564756867ce4b586ed8f4acb6c1f, enough to install
mods and manage load order, but nowhere close to the featureset that
Wrye Mash sports. The main purpose of this code is to act as a sort of
regression test, making sure that we won't introduce anything breaking
Morrowind support in the future (e.g. during a refactoring).
However, adding full Morrowind support on par with Wrye Mash won't be a
goal for quite a while (309+ at least). There's a lot of research and
refactoring to get into before we can approach that. Thankfully,
Elminster has began adding Morrowind support to xEdit as well, so we
may soon have all the tools and documentation we need to write a modern
version of Wrye Mash's features.
-----------------------------------------------------------------------
### Skyrim VR & Fallout 4 VR (#401 and #454)
The situation here is the same as with Morrowind, but for different
reasons. None of us developers own VR hardware, so we have no way to
actually test if Wrye Bash isn't breaking everything on these games.
Which is why there is no official support yet. Still, @nallar and
@nephatrine contributed some initial code in
8273f37e431dfebcfbf114d2fee8cf8365dec889 and
b2418eb47007a110fb3e255190d79304837a85f1 that seems to be working fine,
judging by the fact that we're apparently included on at least one
Skyrim VR guide already. And again, including this semi-supported game
in our codebase means we are much less likely to break it in the
future (not to mention that each game we add generally leads us down a
rabbit hole of refactoring to make Wrye Bash more flexible and
game-agnostic, which is definitely a good thing - one of our long-term
goals is adding support for games outside the 'Bethesda sphere').
-----------------------------------------------------------------------
### FOMODs (#380)
One of the most commonly requested features in Wrye Bash's entire
history. FOMODs are a fairly terrible mod format due to their
incredibly loose nature. You can place files pretty much anywhere in
the package, and there is no specification. That means mod managers are
free to implement whatever they want - as long as it vaguely works like
NMM's installer did, it's an FOMOD installer. Still, this format has
become very common in Skyrim and Fallout 4, so we should support it.
Huge thanks to @GandaG for contributing the initial version in
fa48b1d26bcaf3e9a49b96376df56e2c5b425146, including a full FOMOD
parser, which was a showstopper before. While all other parts of
FOMOD support (including the GUI, the command itself and BAIN
integration) have been rewritten since then (see below), the parser is
still pretty much just as Ganda left it.
Due to BAIN being built around mods being, well, *structured*, it
wasn't a great fit for FOMOD support at first. We eventually came up
with a way to hide all the ugliness of FOMODs from BAIN and just
present it with the clearly structured resulting list of files to
install in a big merge (54844fbe1ea4ddfd48128e61f4232439431aac45) that
also contained a bunch of GUI improvements.
It turns out that BAIN wasn't *actually* that bad a fit for FOMODs
after all: it already had machinery in place to map one path in an
package to a different path in the Data folder, since that's necessary
for its 'root heuristic', remapping of docs into the `Docs` folder and
the rarely used plugin remapping feature of the Plugin Filter. By
hooking into the right part of refreshDataSizeCrc, we were able to use
this feature to map the ugly mess that an FOMOD can be to a final list
of neat paths. BAIN can now work with this regular list of paths, and
when it does need to deal with the real package it will look them back
up in the mapping. Of course all the BAIN refactoring that went into
307 (see 'BAIN' section above) is responsible for making this
possible :)
We still don't recognize all FOMODs, but to get any better would
require extensive BAIN refactoring - which just so happens to be a 309
goal.
-----------------------------------------------------------------------
### CI & Tests (#474, #508)
After two of our betas needed point releases immediately afterwards to
correct blatantly obvious errors, we decided it was time to seriously
push for a CI service that will build and test each Wrye Bash commit.
40285cbab414e3b7fcbcffc33c0be816e68d13f1 added the first version of
this using GitHub Actions.
The test suite is still very limited (only cosaves and a few parts of
bolt are extensively tested) and there are tons of open questions - for
example, we're currently using a very hacky way to change bush.game to
fake having restarted Wrye Bash with a different game selected. While
this works fine for the few tests we have right now, it *will not*
scale, especially not once we get to testing bosh and patcher (which
are the ones we really *want* to test). Expanding on this will be
another important goal in 308/309.
-----------------------------------------------------------------------
### Nehrim (#514)
Support for the Steam release of Nehrim: At Fate's Edge was added in
1bdcd9df2cd9cee164645b74beec1f513aa24129. Unlike Enderal, Nehrim is
not installed as a separate game. Instead, the launcher backs up your
Oblivion installation and allows you to switch between Nehrim and
Oblivion by simply swapping the two folders around. That means we
instead check which game is currently installed in your Oblivion folder
and launch Wrye Bash for that.
That meant a lot of refactoring, including some ugly warts that will
need profiles (#250) to fully resolve, but it also means we were able
to drop our hacky half-baked support for the manual Nehrim version. One
particular thing about this merge that is worth highlighting is that we
merged it *instantly*. As soon as the branch was finished and somewhat
tested, it landed on dev. The reasoning was twofold:
1. We wanted to merge a huge records refactoring branch afterwards,
and did not want to put others through resolving conflicts for it
all the time.
2. FO3, FNV and Enderal have taught us that letting games sit around
on branches is a terrible idea, since they quickly accumulate
subtle bugs.
-----------------------------------------------------------------------
### CBash Deprecation (#520)
With all the refactoring that has happened in 307, CBash has been
completely left behind in the dust. PBash can now merge all record
types (#516), has functional OBME support (#515) and supports several
tags and patchers that CBash does not (#461). Additionally, CBash is
starting to become both a maintenance burden and a roadblock on the way
to refactoring and the patchers and therefore the upgrade to Python 3.
209d884469581248d8ca97954bcb4d05c8ef0d61 officially deprecates CBash.
In fact, the whole reason we are putting out the 307 release *right
now* is so we can get on with removing CBash for good in 308 ;)
-----------------------------------------------------------------------
Massive thanks to everyone who contributed to this release, including:
@Utumno, @Infernio, @Sharlikran, @GandaG, @lojack5, @nycz,
@BeermotorWB, @leandor, @syntaxaire, @fireundubh, @Ortham,
@warmfrost85, @Arthmoor, @D4id4los, @MacSplody, @saebel, @nephatrine,
@nallar, @llde, @FelesNoctis, @DianaNites, @valda and many more that
GitHub's contribution tracker doesn't list.
Infernio
added a commit
that referenced
this issue
Sep 16, 2020
…nity members]
307 is a gigantic release that has been cooking for almost five years
at this point. In fact, it is even bigger than 306:
305 -> 306: 48,748 additions and 44,482 deletions
306 -> 307: 172,916 additions and 111,125 deletions
Where 306 was mostly about refactoring the codebase to save it from
becoming inoperable, 307 is all about using the newly refactored
codebase to design new features, support more games and, of course,
continue the everlasting refactoring war.
The following notes are an attempt to summarize every important thing
that's happened in 307, grouped as topics in a semi-chronological
order. Do note that this is mostly a futile effort due to the 2000+
commits and 280000+ lines worth of changes that make up 307's
development (and the fact that I only joined the project two thirds of
the way through 307 :P).
It is also highly recommended to read this on GitHub, with gitk open
in the background. That way you can click on issue numbers to open
them, while pasting commit SHAs into gitk, where the individual
commits that make up a merge can be seen (GitHub does not expose
this at all).
All mentioned commits are authored by @Utumno or @Infernio unless
otherwise mentioned.
-----------------------------------------------------------------------
### The bosh Split (#201)
The first goal of 307 was to split bosh into a package. After the
patchers had been split out (#163) and basher turned into a
package (#3), bosh was clearly the next big target.
1bdc253e362857dc8a2a484be60bbcccd7891d82 began the splitting, then
it was continued by splitting out the 'messages' code backing the
PM Archives tab in 906858fe14c730ba797711a855933d996d573e2f.
As a small interlude, 5b14b499e17c48917a472ee9ed426b3c7afd8985 then
removed the PM Archives tab entirely (#221) due to the maintenance
burden it had turned into.
682d2134c5e9f69b057cc9fb6e69e13767f5e7de continued the bosh split by
ripping out:
- The face transfer code (used for moving NPC/player faces between
saves and mods) into faces.py
- The OMODs code (used for unpacking OMODs, an old, ugly mod format
used by OBMM) into omods.py
- And the BAIN converter code (used for BCFs, aka BAIN Conversion
Files - small files containing instructions that can be used to
tell BAIN how to repackage an archive into a format it can
recognize) into converters.py
318e66a9ad3a096c316decc751bc76d6c3b5b858 and
75c48e654c21feccd52cd851f2b502fd7c41c45f moved some bosh contents into
parsers.py and bass.py - most importantly, the very commonly used
'dirs' and 'settings' constants.
ea242af573952328168759b9c9c9ff1c01ac681c moved various miscellaneous
things mostly related to plugins and LOOT into a new mods_metadata.py
file.
Coinciding with the INI refactoring (see 'INIs' section below),
6d2a1fa7bcb15f3d5fe787b5475e904e79809253 moved the INI handling code
into bosh/ini_files.py. Similarly,
afb7058a8e0d8a4567d43ec8f6f8828cafc34b83 moved BAIN code into
bosh/bain.py as part of BAIN refactoring (see 'BAIN' section below).
A last few commits splitting out the saves and mergeability checking
into their own files (bosh/_mergeability.py, bosh/_saves.py and
bosh/save_headers.py) were made in
347c552c8bbd3506a5bacd6c678a92e1275c8e06 and
aa82a7af6fbcc6ce24ac1a940d3c2d4f665d626b, closing the issue for now.
bosh/__init__.py is still quite big (3427 lines), but splitting further
is far from trivial.
-----------------------------------------------------------------------
### Fallout 4 (#251)
On November 10 of 2015, Fallout 4 was released - and less than three
weeks later a branch by @lojack5 adding basic support for it was merged
in e85506af4fbd28c4617b46c9ef0832c6c6570ecb0. Unfortunately, not all
game merges in 307 were this speedy ;)
Some more improvements and fixes landed in
134d7c63839de3daa52108f7ce9cc02fcc928df8 by @Sharlikran.
Fallout 4 support was barebones then and is still barebones now. The
real followup work here will come in 308/309 (see #525 and #482).
-----------------------------------------------------------------------
### Dropped Support For Older Settings Files (#253)
Various bits of backwards compatibility cruft had accumulated over the
course of 306 (and even earlier) to keep pre-306 settings working. 307
dropped all this and instead shows an error message if pre-306 settings
are loaded in 307, telling the user to resave them in 306 first so that
they can be upgraded to the new format. All this was done in a single
merge, 8ceef20f269aedb18dbffe17063fac1cb7ccc0f4.
With how big 307 has become, it is probably no surprise that it too has
by now managed to accumulate a metric ton of backwards compatibility
hacks. Cleanup for those will follow in 308/309.
-----------------------------------------------------------------------
### env.py (#258)
In the ongoing quest for native Linux support (#243), encapsulating
OS-specific code (i.e. Windows-specific code) in its own module is
an important step. This was done in
646df2ca063c2dff03cb2185e7e2969fdea065a5. We're still not really there
yet and env.py is, for the most part, still a Windows-specific module.
The end goal would be to have two separate backends from env.py, so
that it imports from one of them based on the OS it is running on.
-----------------------------------------------------------------------
### BAIN (#219)
307 also marks the beginning of the grueling task that is refactoring
BAIN. Still nowhere near done, but it has become manageable. Much more
work to be done in 308/309. Some of the important targets were:
refreshSizeCrcDate was a big method that handled scanning directories
for their contents. The problem was that it was basically two methods
in one and implemented things that only make sense in one specific
directory, namely the Data folder (e.g. empty directory removal).
20e41b3cfc38c71e8baae6e34778b964ca809533 refactored refreshSizeCrcDate
to prepare it for 5ecf4000f10383e3018b4a4febbf8a8f13164a9e, where it
was split into two methods (_refresh_from_project_dir and
_refresh_from_data_dir).
refreshDataSizeCrc is one of the central parts of BAIN. It scans a
package for its contents, applies skips, remaps documentation to the
'Docs' folder, queries and caches CRCs, etc.
ea79bb4fdfe2318d84952ba63fec2fb9b2a7e33a and
5fef1e3924bce2edcf19ff41e469c2a846d851bb tackled this behemoth,
reducing its size significantly. Various later commits like
54844fbe1ea4ddfd48128e61f4232439431aac45 and
c48112237b9953effb74b4206707394f225f035b touched this one up even
further, bringing its final size in 307 to 210 lines (down from 348
lines in 306).
5d435bb05b9e09340eef9a71a402ac018abcb573 and
5ebbf4e8e3646efa8019b9cbccb144be9bc0ead9 focused on centralizing and
refactoring BAIN refreshes, most notably the irefresh method.
b109a4d08f21e023a30d5c0bf4ed45def0ae26e1 made BAIN use the modInfos
cache to avoid recalculating CRCs that we already calculated for the
Mods tab, giving a minor speedup - but mostly the idea here is we want
to read the Data folder once and then delegate the files to the various
FileInfos based on name and/or extension. This is a first step in that
direction (see also #353 and #265).
b1711b1b52fe76f0463b47fab825058f0f3bc41e was an important merge
addressing case insensitive string comparisons. BAIN makes heavy use of
this to keep track of files in the Data folder, each package, external
changes, etc. So a central dictionary that made case-insensitive
comparisons of its string keys was necessary - which is exactly what
was introduced in this merge, as bolt.CIstr and bolt.LowerDict.
545cad09d29cf4b17ed470caba974829f5876d7f refactored the conflict
detection and reporting algorithms to, well, separate them in the first
place. They sat in a single big method called getConflictReport which
came in at 120 lines pre-refactoring. In addition, the fact that this
method returned a single string made it impossible to improve the GUI
for conflicts (a goal in 309) without parsing the string - the string
we just constructed from in-memory objects. The new getConflictReport
is 55 lines long, much more readable and makes use of a new
find_conflicts method that can be used to retrieve the actual conflicts
as in-memory objects to work on.
-----------------------------------------------------------------------
### basher Package Followups (#163)
Began in 306, finished here. daafdd5e76746833afe4eba496aa4afac41ff439
dropped the ancient Tank class for good, curing the flickering on the
Mods tab in the process (#179).
More work done in b7ffca085feb4afe93e48cdafb68521fd9402899.
a28fcfce1f59482d1e701504b738a8df26a9bfe4 is a joint merge by @Utumno
and @DianaNites refactoring the mod export/import links, which were
pasta-filled mess. More prerequisite work landed in
10703a9e2982ae28db60af72146de8d8dca9c324
The topic was finally laid to rest with
fb37a8bf110b6953c07a61a470d05b73ab03eb17 and
3bf6006d842942214694547a3e4c2b2fccb71dee. Every tab now has neatly
separated tab panels, UILists and details panels. The API could still
be better (especially considering how many new tabs we have planned
for 308+ - see e.g. #233, #456, #50, etc.), but it's a far cry from
the situation in 305 and 306.
-----------------------------------------------------------------------
### Skyrim Patchers (#151)
Another principal goal of 307 is getting Skyrim patcher support as
close to Oblivion as possible.
Some prerequisites were merged in
3a2d396b2a9cf2ecd2525006a4e9960f3b4c85ff and
1e18dad44b97bf80b115de0c7175d14df07f2c3f (mostly records).
The first real patcher porting then happened in
9599368dee429f4b63f116d25945beae73d838db.
-----------------------------------------------------------------------
### Load Order (#295 and #309)
Load order handling is a complex beast, to say the least. Not only is
handling all the edge cases difficult, but there are three different
methods that the games use for implementing load order (four if you
count Morrowind, which we don't support - yet ;)).
Wrye Bash originally used the BOSS API for managing load order. This
was changed in 1bf84f3e3b246195b93d9715e2d1891decc47354 to instead use
the dedicated libloadorder. Even libloadorder itself proved problematic
however:
- Adding support for new games was tough and quickly turned into
'adding more clauses into if-else chains'.
- The API generally made no effort to keep actives order and load
order together - which became untenable in Fallout 4, where the two
are inextricably linked.
- Wrye Bash needs to read all the plugins anyways (e.g. CRC, ESM flag,
etc.) and keeps that information cached in bosh.modInfos.
libloadorder was reading it all again which, on top of being
inelegant, was thrown away performance - syscalls are *not* cheap!
- It's a DLL - bad for git and won't work on Linux (#243)
So, in one big merge, 66d7b4d695289f6dc29142f94a6004494e3306bd replaced
libloadorder entirely with a new API written in pure Python. This
enabled many new features such as locking the load order in all games
(bc7e5de47f3ecf31d1290d436940a37a7b6eb0cf), automatic backups whenever
we make a fix to the LO (38817bb5ad6e222c4fdbde270d10341b2371bdb5),
etc.
-----------------------------------------------------------------------
### Patchers (#312 and #461)
Porting patchers to newer games is going to become harder and harder
unless we refactor them to make it easier. Patcher code wasn't
*terrible* per se and was fairly isolated from the rest of the code
(apart from its tight connection to records code - see 'Records
refactoring' section below), but a lot of it was copy-pasted and hence
difficult to understand and expand - plus bugs often had to be fixed in
upwards of 10 places. Adding a new patcher easily required hundreds of
lines of pasta. This is still an ongoing goal, with much more to come
in 308.
30eda2dd5c987648a11fbe01b8ee1b6c56d7c1ac was an early merge, containing
refactoring on other more or less related things.
122784f5cf4d737bbb5943e9bd395d2bfc53c43a then moved config handling
(i.e. which patchers are active and which sources they are operating
on) to basher. The idea is to have the patchers operating only on a
list of sources - not only for elegance and simplicity, but also to
make them testable (see the 'CI & Tests' section below).
Heavyweight refactoring began in
66d7b4ef3f8959e180f0029874c73e40de2a5a52 and
f8bfdc4c5894ca2832aa5aef6515d70f52df110f, which introduced the
_SimpleImporter class to deduplicate a lot of copy-pasted
implementation code. Keep your eyes on this class, it proved to be a
good idea ;)
eb110497c3ce7b9d72df5a6565a0c7efffbc9a28 and
604ebd31d2f20d26c06ab2d043f114f7e0abc26b went in a different direction,
by using the work that had already been done on refactoring the
patchers to port many of the Oblivion-specific patchers over to FO3,
FNV and Skyrim, as well as add some entirely new patchers that had
been commonly requested. Of course, this involved a bunch of
refactoring too, mostly moving implicit constants from all over the
patcher code into game/*/constants.py, which will make it much easier
to port the patchers in the future (e.g. to Fallout 4, see #482).
10e680cbb347b203f935c042145a5fc308b5f4c8 returned back to good
old-fashioned refactoring by decoupling the config/GUI side of the
patchers entirely from the model side (i.e. the code that actually
implements patcher behavior). Previously, patchers were linked
together by importing the patcher implementations in the GUI and using
them as mixins. This led to lots of weird, hard-to-debug code that
crippled the IDE's ability to perform static analysis. For example, the
implementation of the leveled list patcher would use
if not self.remove_empty_sublists: return
to skip the 'empty sublist removal' part of the patcher if the checkbox
for this was not checked in the GUI. However, the IDE had no way of
knowing that that variable actually existed, since it came from the GUI
side of the code via a mixin. After many failed attempts to devise a
base class for both the model and GUI side of patchers, @Utumno instead
realized that a much cleaner design would be to have no mixins.
Instead, each patcher's GUI panel now has a class variable called
patcher_type, which it sets to the model class that gets imported from
the `patcher` package. This allowed us to drop tons of boilerplate code
and make the resulting code much cleaner and easier to understand, but
most importantly it acted as a springboard for further refactoring.
Most notably, 9697b64abc00beaed04e950af20c8116db15151a split our
importers.py file into four files: _cbash_importers.py, _shared.py,
mergers.py and preservers.py. The key insight here was that we can
split our importers nicely into two types: preservers simply carry
forward the last value(s) from a tagged mod, while mergers merge values
from all tagged mods based on the tags those mods have applied.
This resulted in several hundred lines of duplicate patcher code being
chopped off due to us absorbing many preservers back into the base
class. It will also make it much easier to drop the CBash patchers,
since they are now in a separate file altogether (see the 'CBash
Deprecation' section below for more information on this).
One last merge worth mentioning here is
426db77ee71c939bb785b94eab7d4281f0f1fa26. It is a highly WIP attempt to
tame the mess that is parsers.py by devising a proper base class. The
savings so far do look promising, but CSV reading and writing are ugly
warts that still stand in the way. Plus the base class might be too
complicated - right now it has six different knobs that tweak its
behavior, and it's not even clear if using it for the all parsers is
feasible. Still, we had to merge since previous betas came out with
the plugin export/import commands this merge ports to Skyrim.
Much more will follow here in 308 - most notably an upcoming
refactoring of tweaks that will make them *much* faster and drop ~1200
lines of duplicate code.
-----------------------------------------------------------------------
### INIs (#247 and #326)
INI handling was spotty at best. Random unicode tracebacks kept showing
up, the INI Edits tab was one of the last big performance hogs and the
default INI tweaks being files in the Mopy folder led to confusion (at
least one mod had a 'Mopy\INI Tweaks' structure and was supposed to be
installed *into the Mopy folder*).
The first step was df2fcc5f95bc7bc2613a14cb996671f3b82dd2db, a series
of smaller refactorings and fixups to make the tab's code more
manageable.
1d4c23a037f6845f5dfebaf1e5e005f8f121a63b began the work on performance
by introducing the proof of concept for a cache, while
58c47d562f43773c5b017531176b8869138869bf attacked the refresh APIs used
by the INI Edits tab and significantly reduced the number of syscalls
it made.
The default INI tweaks were finally dealt with in
656127c645070d35d31423e014b692621ff94015 by hardcoding them into
Mopy/bash/game/*/default_tweaks.py. No more tampering with the tweaks,
no more Mopy\INI Tweaks folder to confuse users, fewer loose files
packaged, simplified INI refresh, better performance due to fewer
syscalls...
ef21c5bb0a4f23d737dde3b22af36dcaf4f9b58b contained some more work on
centralizing the 'apply a tweak' logic and fixing a longstanding issue
where INIs would have Unix line endings written out, even on Windows.
The LowerDict introduced during BAIN refactoring (see 'BAIN' section
above) also turned out to be very useful for INIs:
a9112d6761e47b1fc41aca53df1add5bdd41ad79 used it to rewrite core parts
of ini_files.py for performance and readability.
bb6c8bd2e7ac22d1218cbad90e404b9739dba7bc reworked the handling of INI
encodings based on a central principle: work with unicode and stripped
newlines internally, encode/decode and add/remove newlines at IO
boundaries.
-----------------------------------------------------------------------
### wxPython (#190, #15 and #488)
A war that started before living memory and will continue until long
after we're all gone - or will it? Actually, we're very close to
winning this conflict for good!
wxPython was all over the place in 305. 306 improved the situation
significantly, but 307 puts even those efforts to shame:
305: 2260 usages (balt: 381, basher: 1586)
306: 1112 usages (balt: 418, basher: 494)
307: 207 usages (balt: 191, basher: 8)
*8* direct usages in basher, down from 494! Let's see how we got there:
- 114c83729aa50045cac4f381912007826d8048bf: Utumno vs wx. Utumno lost,
of course, but wx usages did go down from 1075 to 923. Mostly
accomplished by moving common code to balt.
- c7f5f5085acf1ec6e55f0048c256f7a0ebc3f367: Down to 902, and some
progress was made towards wxPython 3.
- 69c7f9f679f48df8cf7562442e80c9129f10924a: wx.lib.iewin was an ugly
beast that was binding Wrye Bash to the comtypes dependency.
Unfortunately, dropping it required upgrading to wxPython 3 (see
below), so this merge simply centralized the iewin import for a
future removal.
- 531679d37d6c50cfc030b9c462f44250bcfab7a0: A small merge containing
backwards-compatible changes that brought us closer to wxPython 3.
- 050391ca22d7c8451390cbff6fd150ab0b9bcabd: The upgrade to wxPython 3.
Introduced several significant architectural achievements:
- Dropping the comtypes dependency by rewriting our HTML rendering
code to use WebView instead of wx.lib.iewin.
- Removing bolt's locale-related behavior on import that made
importing it dangerous - it's been encapsulated in a new
top-level module, localize.py.
- Rewriting a lot of the very early boot process - see also the
'Boot' section below.
- f9e46eed670b3d2805b1570fc62daf9cca12ce79: The big one. An absolutely
enormous joint merge by @Utumno, @Infernio and @nycz introducing a
new package, gui, that truly encapsulates wxPython. nycz wrote the
first version of the code back in 2017, most importantly the layouts
code that encapsulates wxPython's sizers in a declarative API. We
then devised an event handling framework that enabled us to hunt
down a lot of *implicit* wxPython usages. These are much harder and
nastier to track because they can't simply be regexed. Thankfully
PEP8 will be able to help us here, since all of wxPython uses
PascalCase for its methods, while we're using snake_case for all new
code. The result is a reduction down to 218 direct wx usages outside
of gui.
- 22de7ff9e804b2bcaa8a819922f4b5100b86d80f: After upgrading to
wxPython 3, the next goal was upgrading to wxPython 4. This is also
the first release of wxPython that supports Python 3, making this a
significant step in the direction of py3 support (#460).
- eb86a4cb35fc24b58288a6b3e917ed9350a02e23: In preparation for
finalizing and merging the FOMOD support (see 'FOMODs' section
below), a bit more de-wx'ing happened, mostly on radio buttons and
the splash screen.
-----------------------------------------------------------------------
### BSAs (#339 and #338)
Same story as libloadorder. We were using a binary, libbsa, to do it.
This was thrown away performance (we already read and cached the BSAs
in bosh.bsaInfos), had no Linux support, etc.
Its replacement, bsa_files.py, was introduced in
b199a7bd5eec69ec0852f86d6e3629784e6b90ce, then used to support strings
files packed into BSAs in c4f12d56d1273f40096abe45b6803d9265b1e75e and
finished in 8ce81bfc9b5dc4643988e54471fab3e9b6a1f72d.
With BSA handling code now taking shape in the form of bsa_files.py,
having a second class arbitrarily handling a few things with entirely
different (and much uglier) code would be a bad idea - so
d69f3e82c3afe4b6461b6ccce49a210bedd28dd6 dropped the ancient BsaFile.
There were still some unimplemented parts of the BSA format:
- TES3 format: Added in 4ed5bd8c4ed9a67f872f109c660cf0afeb6ddca5,
also in preparation of the POC Morrowind support we have in 307 (see
the 'Morrowind' section below).
- Compressed BSAs: Added in a6e11c4601d788e0e25c17361d92eaa659e59768,
including both lz4-compressed ones for SSE and zlib-compressed ones
for all previous games.
- FO4 DX10 format: Added in 903b6d11d1451855951cce608c0ce8fe6a23743f.
Currently unused, since we only use BSA extraction for strings
files, which the DX10 format can't contain (it may only contain
textures, for which it is specifically built and optimized).
- Writing: We can read and extract everything from Morrowind to FO76
now, but have no support for altering and writing out BSAs yet.
This will be a goal in 308/309 (see note below).
All this work on BSAs acts as a prerequisite for the BSAs tab we want
to enable and expand in 308/309 (it already exists in the codebase, but
is very unfinished at the moment) - see #233.
-----------------------------------------------------------------------
### FileInfo(s) (#336)
At the heart of each tab sits a DataStore subclass. This provides the
UIList (i.e. what you see on the left side of each tab) with the data
it should show. Most tabs (all but People and Installers) then have
TableFileInfos in the hierarchy, and all but INIInfos (which backs the
INI Edits tab, unsurprisingly) then have FileInfos in its hierarchy.
These FileInfos use FileInfo classes to represent the files that are
going to be shown in the list.
These APIs are not *bad*, but they're not *good* either. Refactoring
this is an ongoing goal, with lots of work done in 307.
b199a7bd5eec69ec0852f86d6e3629784e6b90ce devised a common API for
representing a tracked file with caching called AFile and used it for
the new BSA API (see 'BSAs' section above). Some further work on
freezing the AFile API and making FileInfo use it happened in
11f6769f641e80552b6f2d6e3c25d49a85e66c63,
e3064942119c504786cbe6befafd8a0aa38bec2e and
a0ae08c42fdf47688870387a6f6296de0551bfc2.
Cosaves got a lot of work done in 307, bringing them from pre-alpha at
best to a solid beta API (see the 'Cosaves' section below). Screenshots
also got reworked to use the FileInfo(s) APIs in
e007a5308d509fcd5e123d566b2b3c24e228edaa and
9a317f62cc8f7f807581a0a5f5453637d5d8b5ff.
-----------------------------------------------------------------------
### Skyrim SE (#347)
In a join merge by @Arthmoor, @Utumno and @Sharlikran, Wrye Bash got
SSE support added: 76b8abd5437042dd5b1f1e4505a651211d5523e8.
See the 'ESLs' section below for some of the following challenges with
SSE support. Of course, patcher support was spotty at first too - we
ported all Skyrim patchers to it in
c076e9fe1f5a27746a11b91ba9e0e1b43b80a915.
One more merge worth mentioning here is
064d5021e068260cc99225be29d8d651b0761c61, which sped up startup in all
games, but most notably in SSE. Our reference setup we used for testing
(with ~200 saves) went from 10s down to 2s.
-----------------------------------------------------------------------
### Boot (#373 and #390)
The boot phase was nothing short of a mess. There was no clear guiding
principle of what is initialized when, leading to hard to debug
problems. Unexpected errors could take down Wrye Bash for good without
any way to tell what the problem even was. Restoring settings wasn't
working at all. This is still somewhat in flux just due to how complex
the boot phase is, but it's definitely gotten better.
0e3ef608e2906afb3405b009c622c632caa21dab began the process by
centralizing the wxPython import. In
6060a157be13372ff2a8e09c8de9878c16190f2a, @D4id4los rewrote core parts
of the boot procedure to gracefully handle and show errors, even when
not in debug mode - making fixing startup errors encountered by users
much easier.
Restoring settings was addressed in
17f2266525e5095f8c068804fe3cc3471c9bac1a. In short, when restoring
settings, we would override the settings we just tried to restore due
to our atexit hook firing immediately afterwards. Instead of hacking
away at it with monkey patches, @Utumno carefully reworked the boot
sequence to clearly lay out what gets initialized when, breaking barb's
dependency on the rest of Wrye Bash (balt, bosh, bush, etc.) in the
process and adding a new top-level module, initialization.py, to better
encapsulate init procedures.
The locale mess that bolt did on import was addressed during the
wxPython 3 upgrade in 050391ca22d7c8451390cbff6fd150ab0b9bcabd. This
also resulted in a much more well-documented early boot process,
including setting up the BashBugDump and bolt.depring much earlier,
allowing us to use it to consistently log during the entire boot phase.
-----------------------------------------------------------------------
### Cosaves (#437)
xSE (i.e. the script extenders - OBSE, SKSE, etc.) create cosaves for
each save you make. Wrye Bash originally only needed these for its
master remapping feature to not break things, but over the course of
307 we've come to use them to display save masters with ESLs in them
(since those saves store two separate lists, an accurate master list is
only possible by looking at the PLGN chunk in the cosave). They present
a unique challenge in that each cosave is attached to a regular save,
and all operations on that save need to respect the cosave. That means
renaming, deleting, backing up, etc. need to not just apply to the
main save, but also to its cosave, if it has one.
0a300af01a85bb3535d3194203fd34cc0d3e9b29 introduced the initial API for
this, bosh/cosaves.py. It was mostly just a collection of code from
various parts of bosh (mostly _saves.py), and as such was difficult to
understand, maintain and extend.
822c0bd16e5788d1811449810f34edfee16d77a1 then refactored it (the commit
looks like a rewrite, but was actually a gigantic refactoring
comprising 100+ commits that had to get squashed down to a single one
in order to not break dev) for maintainability and to bring Pluggy
(an ancient cosave format in Oblivion) support into the cosave
hierarchy. It also added support for saves with ESL masters, as
mentioned above.
Finally, e4dc76995703f16ee97fb07d495b6db635a2c6a8 reworked our handling
of cosaves to be much more robust
-----------------------------------------------------------------------
### ESLs (#382 and #429)
ESLs are a new type of plugin file introduced with SSE and FO4. They
present a unique challenge in that they can bypass the usual 256
plugins limit, and as such stress-test many central assumption in any
tool that tries to support them. Our APIs stood well to the test
however, with d507111773d459e41468ac835955fe88915e622e only having to
make minimal changes to add initial support.
Due to the limited understanding of ESLs at the time, we were very
conservative in what we allowed users to do with them. There was no way
to verify ESL flags, add and remove them, and the Bashed Patch excluded
ESLs completely. That was fixed in
547565a32f8d1d4a2944984c8c29092834f4fff6, a join merge by @Sharlikran,
@Utumno and @Infernio. With it, Wrye Bash gained the ability to add the
ESL flag to ESL-capable mods, importing from ESLs into the Bashed Patch
was reenabled and load order operations for ESL-flagged ESPs were
fixed.
Once again, we were quite conservative in implementing the ESL flagging
in Wrye Bash. For all record types we had not decoded yet, we simply
failed the verification and told people to use xEdit to check instead.
c137405418360063b6d767e7179c31fa94936cbc changed that to use a generic
method that does not rely on our record definitions, since the only
thing we actually have to care about are the headers of all records in
the file. The contents of those records do not matter. This made
ESL-flagging both faster and completely accurate for all record types
in both SSE and FO4.
-----------------------------------------------------------------------
### Game Handling (#358)
Along the way, especially after adding initial ESL suport (see 'ESLs'
section above), it became clear that Wrye Bash's game handling would
become a big issue that needed addressing. Adding support for a new
game involved dozens of edits all over the codebase due to fsName
checks and copy-pasting and editing a big constants.template file.
This cripped the IDE's static analysis, since it couldn't check that
any given game constant existed, let alone had the right type. We were
also importing way too much for each game (e.g. all the constants, the
default tweaks, the vanilla files, etc.), when we should really just
import them for the one game we're actually managing. Thrown away
performance and memory, plus just plain inelegant.
In a joint merge by @Utumno and @GandaG,
11fa0f6a71ca8420071e76357b89f5d3e221c904, the game constants were moved
from module-level into classes, allowing them to be inherited. This got
rid of tons of duplicate code (1229 insertions(+), 1824 deletions(-))
and made adding both a new game and a new game constant easier and less
error-prone.
Some more work to move game-specific constants out of random files and
into the game/*/constants.py files they belong into happened in
fa74a1b7a61d9b3150f0d2b171145e171f2d27e5, along with some fixes in
06d6a6bdaa925f379ffc1f05d0fa5977057ac739 and
57d3a621a08f4852dc5d5cc36878db9286351579.
4050e60bd372494c46860c85fa15c1701abcc5ae devised a way for us to avoid
importing the constants, default tweaks, etc. for every game, while
ddda9393d9b08a132c4a346fbdd8cc84454bccbd and
0fecde47b73d3204735c28b37260b7af8a01f700 finally finished off the last
few constants outside game/*/constants.py, closing this issue for the
time being.
-----------------------------------------------------------------------
### Fallout 3 & New Vegas (#150 and #468)
Not originally planned to be part of 307, but after it was accidentally
included in Beta 3, we had to merge it:
0f06e4fd306684aafccd2764b47a66d2205d10fc
@valda originally ported Wrye Bash to FO3/FNV as Wrye Flash. Efforts to
backport the changes to WB had been dragging along since forever, so
the main thing we learned from this merge was that leaving games to rot
around in branches is a *terrible* idea. Better to have the WIP code on
dev without explicitly providing support, as leaving it on a branch
makes it accumulate subtle bugs from refactoring extremely quickly.
Some work on synchronizing the FO3 and FNV versions happened in the
form of 54b8e614acd0841460f550d33d23340824f06e31 - since FNV is a
vastly more popular game when it comes to modding, many of the
improvements that valda made to the NV version of Wrye Flash did not
make it back to the FO3 version. With us being based on a single
codebase, doing that is much easier - eventually culminating in the FNV
constants being entirely deduplicated in
3f96076501d5c260af856dac1f6475c21aa53a6e,
e68b7af1eb9a3eed96b72cda067be82d86e067c6 and
dccd28c70ffb857ccc70265a5ca22ca718d8e442 so that they are based on the
FO3 ones, meaning that adding e.g. a new patcher or bash tag to FO3
will automatically add it to FNV as well.
We're almost at feature parity with valda's version now - only the race
patcher is missing from our version. This will be addressed in 308/309.
On the other hand, we support several patchers and tags that valda's
version doesn't, on top of tons of other features and bugfixes (see,
for example, the rest of this commit message ;)).
-----------------------------------------------------------------------
### Readmes (#432 and #464)
307 includes a significant reworking of the readmes, courtesy of
@FelesNoctis in 493c76b38c760d31757657a5b9bcf70f196d1dcd. It was later
followed up with more edits for maintainability and to update the
screenshots: see 18969116f7547148bf7b66196f91accd1225b465 and TODO
-----------------------------------------------------------------------
### Wizards (#446, #445, #444, #436 and #189)
Several issues related to wizards were fixed (see e.g.
169d8347c1e4f3a3f6d696d4a660f89987f6bffc,
1f71bf335b85276566c12db43b53097842e05981 and
30f698520be938cd3cb6aa950cf979bc5468edb6).
We also refactored the code quite a bit and deprecated the old 'Espm'
versions of keywords and functions in favor of new 'Plugin' versions
(more intuitive, easier to spell and remember, and more accurate with
the advent of ESLs) in 3426384083bd5d7c61d42730ed7fa9c5629bc2db.
Finally, 9244f8536ff0e4b780e3190cc40a032771310f4c added a new wizard
function that had been requested a long time ago, enabling wizards to
alter their behavior based on a plugin's load order.
There is still a lot to do on wizards. For a start, the format is not
formally defined - and the parser that acts as a reference is quite
buggy (e.g. `Note thisIsAString` will print out 'thisIsAString',
because the parser gets confused about its token states and
accidentally treats 'thisIsAString' as a string). See
https://github.com/Infernio/wizparse for my POC attempt at defining a
formal grammar based on ANTLR that other mod managers will also be able
to use. This is low priority, but will be continued in 309.
Additionally, the wizard GUI presents a significant challenge in
de-wx'ing (see 'wxPython' section above) and has a lot of duplication
with the new FOMOD GUI (see 'FOMODs' section below).
-----------------------------------------------------------------------
### Python 3 (#460)
We officially started the process of porting Wrye Bash over to Python 3
in September 2019, seeing as Python 2 has reached its end of life. This
has turned out to be nothing short of a giant can of worms. Wrye Bash
makes heavy use of bytestrings, so simply letting 2to3 run over the
codebase would be disastrous - we'd be fixing unicode/bytes tracebacks
for the next few months and getting no actual work done. In addition,
our policy of having no breaking commits on dev means that an eventual
Python 3 port will have to be a single commit, which makes bisecting
useless. So the result is that we need that py3 commit to be as small
as possible. With that goal in mind, a lot of prerequisite work that
brings us closer to a py3 port without breaking py2 has landed:
- 660ecbb81f49d478bdc8e4a8905e328d1daf9dca: py3 has no 'ur' prefix for
strings since the one in py2 wasn't actually a 'raw' prefix:
>>> print(ur'\u03B3')
γ
So dropping this one from the codebase was necessary. This commit
just dropped all usages in strings that didn't actually have
backslashes or were autogenerated paths (i.e. vanilla_files).
- d43ad244170e2110a6daca7d5febed4020550247: This commit by @syntaxaire
finished off the 'ur' removal mentioned above.
- 5a98eb4c025651f4e9366db2a7d488ec2068f1fc: cmp and __cmp__ do not
exist in py3. For the most part, we just had to implement rich
comparisons.
- fa74a1b7a61d9b3150f0d2b171145e171f2d27e5,
cae844b9dde8af014b09a1cb24af2348d5620058 and
6db5b8b59e28bc46a9d42e966d31007e113c59e6: Changing old-style classes
to new-style ones work fine, except when the class is used as a
mixin with a new-style one that uses __slots__. That can lead to
nasty layout conflicts, as seen in the first of these three commits.
- 8e201c49bbc809da89b1bda1d269f4cb7619dfc0: Our codebase included an
ancient version of chardet (1.0.1 from 2008) due to a single manual
edit that was needed to make it avoid returning the EUC-TW encodings
that Python doesn't support. We dropped it in favor of the PyPI
version, and addressed the EUC-TW problem in
60d0c29dbae91c12c1f7825df9f4e8e243ca09d2.
- d8d03ca39e1e9f85250fd014cabcc2a65945e5e7,
197b6a2de78acd723f9d747fd6751fd2c68cf944 and
659e5b696be5083b9bef0d39356acc30ab46b5a4: Long integers don't exist
in py3 due to its int type having no max size. So we needed to drop
all 'L' postfixes and usages of sys.maxint.
- 664f1722a53c91794f192e343936dfd34b8e86a8: Fixes for various issues
encountered during an experimental run of 2to3 by @lojack5.
- 33eac7624971ecd22e1f65ff5e47bc71ca175dbc: Merge by @GandaG
addressing various py3 issues like print, moved stdlib modules, old
exception syntax and usage of local absolute imports.
- 050391ca22d7c8451390cbff6fd150ab0b9bcabd and
22de7ff9e804b2bcaa8a819922f4b5100b86d80f: We were stuck on wxPython
2.8 for a long time, but the first version of wxPython that actually
has py3 support is wxPython 4. These two commits (as well as tons of
prerequisite refactoring, see the 'wxPython' section above) cleared
that blocker for good.
The Python 3 port is one of the primary 308 goals, along with patcher
refactoring (#312) and records refactoring (#480), on which it is
blocked (due to the aforementioned heavy bytestrings usage, which
those refactorings will help us isolate and encapsulate).
-----------------------------------------------------------------------
### Build Scripts (#415)
An enormous productivity gain for developers came in the form of
@GandaG's reworking of build scripts in
a1b5bfaa40fdbe04549ba3775106ffdff471e62e and
5f31a2adf39a607db282f49bd43e66c992a1baad. The ancient
package_for_release.py has been replaced with a sleek new build.py
script that does everything you need to do to build Wrye Bash in a
single invocation. On top of that, Ganda also dropped tons of weird
legacy things the build scripts did, like using ResHacker.exe to set
the Wrye Bash icon - more binaries gone <3
-----------------------------------------------------------------------
### Enderal (#433)
Support for the Steam release of Enderal: Forgotten Stories, a total
conversion mod for Skyrim was added in
c2d73965fba4d0a82bb95f7cbe13b7f5dbcc0155. This was a fairly simple game
to suport since it is pretty much just a pre-modded version of Skyrim
LE. Of course, the work on refactoring game handling is the reason why
we had so few problems with this merge. Once again, we left this game
too long on a branch, meaning it began to accumulate bugs. That
necessitated a fixup commit almost immediately in
3afa217d987c8c4ab86341ed8a8ec906b099767a. For all future game merges,
we resolved to merge more quickly, as long as adding support for the
game doesn't break any other code.
-----------------------------------------------------------------------
### Records (#480)
After all the above, there were a few spots left in the codebase that
needed *heavy* refactoring: records, patchers, saves (*not* save
headers, the Oblivion-specific save editing code) and BAIN. Since
the records and patchers code are very closely intertwined, they need
to be attacked in tandem (refactoring the patchers is sort of a
'top-down' approach, while refactoring the records is a 'bottom-up'
approach to the same problem). See the 'Patchers' section above for
more information on that refactoring.
The whole shebang began in ecac15d01dc5c89463ad47aab74260abfbea4167 and
3a3e9c935f5c1a798211eb0eaed0a0dd76a9af24, which were mostly just random
commits improving some record definitions.
Heavyweight refactoring began in
134433fde71534fc09e357ad64c696194f51a8eb, which moved an awful lot of
records code into brec by creating new tools for defining record
definitions. The result is a massive reduction in code size:
6787 insertions(+), 9581 deletions(-)
..and a very nice situation where almost the entire *implementation* of
PBash sits in brec, while the (almost) purely declarative definitions
sit in game/*/records.py. Unfortunately, this bloated brec to 3000+
lines and made it much harder to tell which classes belonged together.
This was addressed in 28c11cb934056790e2a07703a9a09b4a5de8aa48, a huge
merge that split brec into a package, added OBME support to PBash, sped
up plugin loading by using AOT construction of struct.Struct instances
instead of struct.pack/unpack and implemented merging of all record
types in Oblivion. That's right, PBash can now merge everything CBash
can. See the 'CBash Deprecation' section below for more information.
After reading both the 'Python 3' and 'Patchers' sections, you should
already know what's coming here: much more in 308. There's already a
large refactoring ('part 2.5' of #480) that just needs some testing
before it can be merged, and @Utumno is working on a 'part 3' of #480
that will seriously turn some parts of the records code on its head.
-----------------------------------------------------------------------
### Usability and Accessibility
Wrye Bash has a (not entirely undeserved ;)) reputation for being
difficult for newcomers to get started with (in UX terms, we'd say its
out-of-box experience is bad). While this is obviously a big goal that
we're nowhere close to solving today (really, someone with actual UX
experience would be needed on the team), 307 does include some work
towards both this goal and the goal of making Wrye Bash accessible to
everyone, regardless of disabilities.
- 1bf488a10a4c9fedb2737e4b2eee86c484f7b93d: The ability to jump to a
plugin's matching installer from the Mods tab has been added (#53).
- 3d7b9816ec0e2b7d666a7bead0cd45db5347aed7 by @fireundubh added the
ability to jump to the matching plugin when a master is
double-clicked in a masterlist (#311).
- 261a1029a78e2cae773386c4e41f496a05f018c0 and
f4987d1d0db38e4379f6470f67c37b982192817f by @BeermotorWB and
@MacSplody trimmed the jungle that was the package context menu on
the Installers tab by moving the more rarely used commands into
submenus.
- f65112bea111157558f78f056b550a7f689752a3 added the ability to jump
to a plugin from the Plugin Filter on the Installers tab.
- 92691409567ce020c6958325ee8c4bd8c9e820cc made the 'Sort By',
'Columns' and 'File' submenus on each tab consistent by putting them
in the same places (they were in seemingly random positions on the
context menus before).
- 206ffbd9b09a0b8107ac5dacd9b0b3f9c2d1bfc2 by @warmfrost85 allowed
users to get a preview of what Clean Data and Sync From Data are
going to do, as well as the ability to use that preview to change
which files the commands will affect.
- e5685f3e87abd0bce199fe619509a9648ce2db89, also by @warmfrost85,
allowed Sync From Data to work with archives. Previously if you
wanted to, say, clean a plugin and sync it back into its archive,
you would have to either do it manually in 7zip or use the
workaround of unpacking the archive to a project, then using Sync
From Data and finally packing the project back into an archive. Now
you can just do it all in one go by using Sync From Data directly
on the archive.
- b427bbd383688a2f0fd55266b040cdc6ddf7c590 and
a2297331b6571b6989bf1ea3b1c253a85c834448 increased the contrast on
all our checkbox images to pass WCAG AAA guidelines, to make it much
easier for people with weak eyesight to use Wrye Bash. In the
future, we want to allow people to customize the colors of the
checkboxes so that colorblind people can make use of them too
(see #511).
- edbe5e51d294ed0706478a9f8896d1e170d76058 added a menubar to improve
discoverability of Wrye Bash's column context menus, as well as
making it possible to access them purely using a keyboard.
Previously you would have had to right click one of the columns to
access these commands and options, which is a bit arcane and doesn't
work for people who can't use a mouse at all. The same merge also
reworked our half-baked settings menu that was implemented as a list
of popup options when the tiny gear icon is clicked to instead be a
proper settings dialog. Not only is this much easier to navigate, it
will scale far better for our future needs (e.g. extensions).
-----------------------------------------------------------------------
### Morrowind (#479)
Morrowind is not officially supported in 307. We've added some (very)
WIP code in 45ed499de6b5564756867ce4b586ed8f4acb6c1f, enough to install
mods and manage load order, but nowhere close to the featureset that
Wrye Mash sports. The main purpose of this code is to act as a sort of
regression test, making sure that we won't introduce anything breaking
Morrowind support in the future (e.g. during a refactoring).
However, adding full Morrowind support on par with Wrye Mash won't be a
goal for quite a while (309+ at least). There's a lot of research and
refactoring to get into before we can approach that. Thankfully,
Elminster has began adding Morrowind support to xEdit as well, so we
may soon have all the tools and documentation we need to write a modern
version of Wrye Mash's features.
-----------------------------------------------------------------------
### Skyrim VR & Fallout 4 VR (#401 and #454)
The situation here is the same as with Morrowind, but for different
reasons. None of us developers own VR hardware, so we have no way to
actually test if Wrye Bash isn't breaking everything on these games.
Which is why there is no official support yet. Still, @nallar and
@nephatrine contributed some initial code in
8273f37e431dfebcfbf114d2fee8cf8365dec889 and
b2418eb47007a110fb3e255190d79304837a85f1 that seems to be working fine,
judging by the fact that we're apparently included on at least one
Skyrim VR guide already. And again, including this semi-supported game
in our codebase means we are much less likely to break it in the
future (not to mention that each game we add generally leads us down a
rabbit hole of refactoring to make Wrye Bash more flexible and
game-agnostic, which is definitely a good thing - one of our long-term
goals is adding support for games outside the 'Bethesda sphere').
-----------------------------------------------------------------------
### FOMODs (#380)
One of the most commonly requested features in Wrye Bash's entire
history. FOMODs are a fairly terrible mod format due to their
incredibly loose nature. You can place files pretty much anywhere in
the package, and there is no specification. That means mod managers are
free to implement whatever they want - as long as it vaguely works like
NMM's installer did, it's an FOMOD installer. Still, this format has
become very common in Skyrim and Fallout 4, so we should support it.
Huge thanks to @GandaG for contributing the initial version in
fa48b1d26bcaf3e9a49b96376df56e2c5b425146, including a full FOMOD
parser, which was a showstopper before. While all other parts of
FOMOD support (including the GUI, the command itself and BAIN
integration) have been rewritten since then (see below), the parser is
still pretty much just as Ganda left it.
Due to BAIN being built around mods being, well, *structured*, it
wasn't a great fit for FOMOD support at first. We eventually came up
with a way to hide all the ugliness of FOMODs from BAIN and just
present it with the clearly structured resulting list of files to
install in a big merge (54844fbe1ea4ddfd48128e61f4232439431aac45) that
also contained a bunch of GUI improvements.
It turns out that BAIN wasn't *actually* that bad a fit for FOMODs
after all: it already had machinery in place to map one path in an
package to a different path in the Data folder, since that's necessary
for its 'root heuristic', remapping of docs into the `Docs` folder and
the rarely used plugin remapping feature of the Plugin Filter. By
hooking into the right part of refreshDataSizeCrc, we were able to use
this feature to map the ugly mess that an FOMOD can be to a final list
of neat paths. BAIN can now work with this regular list of paths, and
when it does need to deal with the real package it will look them back
up in the mapping. Of course all the BAIN refactoring that went into
307 (see 'BAIN' section above) is responsible for making this
possible :)
We still don't recognize all FOMODs, but to get any better would
require extensive BAIN refactoring - which just so happens to be a 309
goal.
-----------------------------------------------------------------------
### CI & Tests (#474, #508)
After two of our betas needed point releases immediately afterwards to
correct blatantly obvious errors, we decided it was time to seriously
push for a CI service that will build and test each Wrye Bash commit.
40285cbab414e3b7fcbcffc33c0be816e68d13f1 added the first version of
this using GitHub Actions.
The test suite is still very limited (only cosaves and a few parts of
bolt are extensively tested) and there are tons of open questions - for
example, we're currently using a very hacky way to change bush.game to
fake having restarted Wrye Bash with a different game selected. While
this works fine for the few tests we have right now, it *will not*
scale, especially not once we get to testing bosh and patcher (which
are the ones we really *want* to test). Expanding on this will be
another important goal in 308/309.
-----------------------------------------------------------------------
### Nehrim (#514)
Support for the Steam release of Nehrim: At Fate's Edge was added in
1bdcd9df2cd9cee164645b74beec1f513aa24129. Unlike Enderal, Nehrim is
not installed as a separate game. Instead, the launcher backs up your
Oblivion installation and allows you to switch between Nehrim and
Oblivion by simply swapping the two folders around. That means we
instead check which game is currently installed in your Oblivion folder
and launch Wrye Bash for that.
That meant a lot of refactoring, including some ugly warts that will
need profiles (#250) to fully resolve, but it also means we were able
to drop our hacky half-baked support for the manual Nehrim version. One
particular thing about this merge that is worth highlighting is that we
merged it *instantly*. As soon as the branch was finished and somewhat
tested, it landed on dev. The reasoning was twofold:
1. We wanted to merge a huge records refactoring branch afterwards,
and did not want to put others through resolving conflicts for it
all the time.
2. FO3, FNV and Enderal have taught us that letting games sit around
on branches is a terrible idea, since they quickly accumulate
subtle bugs.
-----------------------------------------------------------------------
### CBash Deprecation (#520)
With all the refactoring that has happened in 307, CBash has been
completely left behind in the dust. PBash can now merge all record
types (#516), has functional OBME support (#515) and supports several
tags and patchers that CBash does not (#461). Additionally, CBash is
starting to become both a maintenance burden and a roadblock on the way
to refactoring and the patchers and therefore the upgrade to Python 3.
209d884469581248d8ca97954bcb4d05c8ef0d61 officially deprecates CBash.
In fact, the whole reason we are putting out the 307 release *right
now* is so we can get on with removing CBash for good in 308 ;)
-----------------------------------------------------------------------
Massive thanks to everyone who contributed to this release, including:
@Utumno, @Infernio, @Sharlikran, @GandaG, @lojack5, @nycz,
@BeermotorWB, @leandor, @syntaxaire, @fireundubh, @Ortham,
@warmfrost85, @Arthmoor, @D4id4los, @MacSplody, @saebel, @nephatrine,
@nallar, @llde, @FelesNoctis, @DianaNites, @valda and many more that
GitHub's contribution tracker doesn't list.
Utumno
added a commit
that referenced
this issue
Jan 19, 2021
The ghost of #163 - reduce self.window.data_store/selected uses - Link readability and infos encapsulation (few remain actually and those are mostly fine). Mopy/bash/balt.py: _askOpenMulti was unused Mopy/bash/archives.py Mopy/bash/bosh/converters.py Mopy/bash/bosh/__init__.py: random nit - note local function name prefix - otherwise I catch myself looking for a module function :P Mopy/bash/bosh/bain.py: we are inside refreshDataSizeCrc and fileExt comes from os.path.splitext Move isel** methods to Link - ghost of #174
Utumno
removed
the
M-backburner
Utumno
added a commit
that referenced
this issue
Jun 20, 2023
Further encapsulate self.window - under #163. Random nit.
Utumno
added a commit
that referenced
this issue
Sep 30, 2023
StatusBar_Button(Link), by having as a parent the DnDStatusBar, binds Link to balt (in a typing comment) and it's the next step in de-wx'ing - StatusBar_Button is-not a Link for one. But this needs some more clean up started in ee0b8c7 and 45f1261 (see comments in 6659eb3 merge). Move java to class scope, initialized once - appArgs was used once - this removes the init override. Erode SetBitmapButton: - Turns out image was never passed but internally, replace with private method (wip, it's images after all) - onRClick was to override DoPopupMenu - getting there Simplify UpdateIconSizes: One less call of _addButton get_icon: Unused since 7615cf38ebd3b83ed7633316aa9ed210dab2e27em (yey!) Under #190, #163
Utumno
added a commit
that referenced
this issue
Oct 21, 2023
Utumno
added
A-wx
Utumno
added a commit
that referenced
this issue
Nov 3, 2023
I was thinking of this for a while now - there are a few other places where we could use Lazy: - gui patcher panels (an opportunity to investigate why they are created once) - Link - yep wrapping wx.Menu() is in the last 10% of de-wx'ing and might make sense to cache those (and clearly differentiate between stateless and stateful ones) - Imagewrapper ;) Currently it's over-fitted to its single use -> StatusBar_Button, but there is fit quite nicely (note a few methods/class variables gone and the rest I believe reads more self explanatory than before). For now I disallow creating the widget through a _native_widget property access but that might not fit the other usecases (or indeed all usecases that allow_create is True - note that _append in links is an allow_create check) Make _is_created private and instead override the tooltip to not blow if the _native_widget was not created. See dicussion in RRR _present -> allow_create and remove useless SetState: Refactor OBSE handling - remove obseButtons and have the instances check their _obseTip and the state of the OBSE button to set the correct tooltip on creation/flip state of obseButton. This will simply reset the tooltips of all *created* items, probably just resetting some tooltips to the same str, but the simplification is more than worth it. Execute -> sb_click Use the new ring to kill a dragon :P Use the link.uid in Hide/Unhide that gets us rid of some more dragons Under #190, #163 #174 We still must make sure the _tip is not empty for the SB settings dialog or rather that the sb_button_tip returns an unique value
Utumno
added a commit
that referenced
this issue
Nov 4, 2023
This decouples belt from the rest of Bash, by moving the non Bash specific parts into PreParser and belt itself into basher - hereby closing #163 after 9+ years of work - gui imports are confined in balt/bash/basher, bolt is thinned nd decoupled from belt and belt code is ready to be deduplicated (gui_fomod shares some gui logic). Some (!) work on #219, but mainly under #600. Signed-off-by: MrD <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Basher must be split to a package. Subgoals:
baltgui package -> see De-wx'ing #190306:
still needs some review: 3fcbe3f- well, reviewed here: e7f9493__init__()plus deprecate TankData paramsAppendToMenu- not only huge amount of boilerplate that mixes UI in but also a huge source of wx usages.Tank.gList and List.list (list !?) must be unifieddone in favor of gList, List.list dropped)Consider registering some not widely used callbacks like OnChar, OnDClick via hasattr calls- hasattr is evil but not totally (or is it ?) --> it is.307:
_gListprivate (__gList)Link.Frameuses in bosh:307+:
UIList make self.data_store private (around 40 uses of self.window.data_store in Links)would be nice to have a property thoughThe text was updated successfully, but these errors were encountered: