De-wx'ing #190

Utumno · 2015-02-02T20:38:47Z

All wx code must eventually end up in ~~balt~~ gui. This does not simply mean that one must create wrappers for wx classes/constants and drop them into gui - it means that gui must export an API - so gui should not return wx items on which wx methods will be called - must wrap those methods into class methods it exports - in other words we must not only eliminate wx imports but also wx calls on wx objects returned by gui functions (such as Bind). This (apart its obvious, I hope, value vis a vis code quality) will also make it easier to move to later versions of wxPython / offer an entirely different backend. There is already a great deal of it done (edited out of this issue). But there is also an awful lot left to be done. So:

Some newer TODOs:

where did the tracebacks go? -> fixed in 6f121cb
the issue with warnings on booting still allowing the user to interact with main frame -> fixed on wx4 yey! -> https://trac.wxwidgets.org/ticket/17816
the progress dialog - ugh. Our whole CancelError stuff is not working anymore (and I never liked that exception control flow anyway). ~~More importantly, there is the strange issues of losing focus and I think it's the progress window again.~~ -> wx bug, fixed by switching to GenericProgressDialog
related to the above - make sure we don't refresh twice - we must follow a bit the events (with good old prints) and make sure they are processed once - I am thinking also of cases we display dialogs and such, is RefreshData triggered? More @balt.conversation ? We could maybe add an _is_shown attribute and use the evt_activate argument to only refresh when the _is_shown becomes True from False -> seems working ok but keep an eye open!
not sure about properties yet - sometimes it's ok but sometimes methods are preferred - more explicit - WIP but stabilizing - current code is fine
events - it seems wx did not doubly register a handler - see notes on 4766e10. We do and we skip by default which means we have to go through the previous uses and carefully evaluate if skip is needed. Seems it's not the default because it is not meant to be used often. So there are system handlers that might need to be skipped - could not really understand their docs: https://wxpython.org/Phoenix/docs/html/events_overview.html#how-events-are-processed - I guess system handlers run after user ones
- revisit event handling during boot - especially size events need be handled after setting sizes before maximizing apparently etc - see 4d70274 (Attempt to fix installers tab positions not being set)
balt will no longer serve a purpose once we're done - all balt code should either go into basher (highly specific Wrye Bash code) or into gui (pure abstractions over wx). Useful rule of thumb: if you think the class / method could be easily copied into another project without dragging core Bash code along while still being useful, it's gui
revisit locale handling - see> wxWidgets/Phoenix#1702 from wxPython 3 Upgrade #15 (comment)

Under #163

The text was updated successfully, but these errors were encountered:

Under #190. mod_links.py: wx.DEFAULT_DIALOG_STYLE was not needed and resize param added

Under #190. Apart from couple uses in bash.py where wx is used directly. default parameter should be used in more (all) dialogs.

Under #190. Eliminate FindWindowById() boilerplate wx code. I also renamed local vars in ListBoxes.init for readability. Mopy/bash/basher/__init__.py replaced a `with ListBoxes` with `ListBoxes.Display` - not sure why I hadn't # This is the 2nd commit message: belt: FindWindowById(wx.ID_FORWARD) calls encapsulated

Tank and List must be unified - first step is this class but the hard part will be the TankData part. UIList calls wx.Panel and wx.ListCtrl `__init__()` Notice MastersList did not register a callback for OnItemSelected, made it event.Skip() # fixup: IniList did not register a callback either - it is iniList.Bind(wx.EVT_LIST_ITEM_SELECTED, self.OnSelectTweak) Ideally this should be in IniList id parameter removed from Tank and List `__init__` and also from balt.ListCtrl `__init__` _sizeHints class parameter - notice it was set in some cases via SetSizeHints - TODO: hunt this down dndColumns=() - made immutable, should be OK

More work is needed (bosh.settings['bash.statusbar.iconSize'], tip (see #174)) Mopy/bash/basher/__init__.py: I do not like those additions to the image dictionnary - I would like to see those refactored (into the button classes ?) Under #190 for pruning ids

The Button one remains but hey only in balt. Still I suspect that the parts of code that use it are bad - they do not seem to register their buttons - EDIT: addressed in next couple commits Under #190 for pruning ids

Under #190 - de-wx'ing. Last id in balt API dropped: @@ -396,3 +396,3 @@ class Button(wx.Button): def __init__(self, parent, label=u'', pos=defPos, size=defSize, style=0, - val=defVal, name='button', wxId=None, onClick=None, tip=None, + val=defVal, name='button', onClick=None, tip=None, default=False):

@mjpieters

This release features an almost complete refactoring of the codebase, fixing many bugs and opening the way to extend Bash functionality. Bash, being a community work, has over the years become an assortment of hacks, patches and cruft and the program was just about to become non-operable. Example issues - in no particular order (although probably the worst was the horrible performance): - deleting an item in the lists displayed by Bash ran different code depending on whether the user hit delete or right clicked and chose delete - most of those pieces of code were buggy - start up took more than 15 seconds on large Oblivion Load Orders and alt tabbing out and back into Bash would take 6-7 seconds - last one should take no time in the usual case - as seen from the (user visible) debug prints the game detection algorithm run 3-5 times - many, many broken things - including performance in general (most simple operations would hang Bash for some seconds), display glitches and general UI inconsistencies and internal data structures corruption (see #176) Those bugs reflected the state of the code - again in no particular order: - bosh.py (the data model) in 305 was 24635 lines (reduced from 30550 lines in 304.4 - see bosh section below) - basher.py (the UI lists/menus etc) was 18936 lines - the seven tabs that Bash featured were backed up by Tank (2) and List (5) instances - so implementing a feature (say delete) needed edits to at least two classes (usually 7) - the wx library was intermingled with the basher code - the menu items (Links) defined 215 AppendToMenu methods with nearly identical wx code - readability suffered - most attributes/variables were named `data` or `items`, globals hacks crippled the IDE static analysis, copy paste (aka ravioli) code was everywhere. When I started in it I had no idea if refactoring the code was even possible, core Bash is 80k lines of code. It was clear that bosh needed splitting, that candidate number one was the patchers code (well defined and functional) and that basher should follow - but was not at all clear if those were possible, let alone how exactly should the code be structured and how much time would this take. Turns out that it was possible: Bash was indeed coded with modularity in mind (by Wrye ?) but later additions bloated the initial modules beyond recognition and broke the (non enforced) encapsulation - however it wasn't too late. Also turns out that refactoring needed 2 full work years of a single engineer (learning python). The huge performance boost that this release features is the measurable effect of the increase in the code quality. In the items below I try to describe in a quantitative manner what "code quality" means and how it was increased. That's meant to be read from an app that links to the commits - both github and gitk do but additionally github links to the issues and files. ### bosh This release features splitting the patchers' code out of bosh to a dedicated package (whose final structure is still at stake). This reduced bosh to 10516 lines while naturally exposing class dependencies issues, hidden by the "one file program" nature of Bash. The attempt to split bosh had already started in 305, involving: - an effort to eliminate code duplication in C and P patchers using multiple inheritance (chopped off around 1500 lines of pure copy paste code) - see commits in 3c40989 - splitting out what was to become `record_groups.py` - see c3e3543, but don't imitate it - lots has changed since then, including import conventions and commit style - for instance commits on dev that do not launch are strictly forbidden. - splitting out what was to become `parsers.py` - see commits in 08ab6e2 Final (`record_groups.py`, `parsers.py`): 6ae30fc With those out of the way it was finally possible to rip the patchers out of bosh - starting here: 983f259 That was not a mere copy paste rip - warnings were squashed, long lines were wrapped, and most importantly a lot of tricky refactoring was applied to eliminate code duplication - see for ex. commits in: 72fc8a0 for ripping and commits in (note the negative line count): 5d21377, fdab163, 78a85e0 for refactoring See #3 for commit links (finally closed in b6b743b) and #124 for the (still open) package layout issue. Be aware that a lot of things changed since, including import style and the overall package layout - see: e6b2e0e The remaining bosh.py code had some preliminary refactoring applied to it (archives handling, BAIN error handling and refresh etc - see 6ddd4a8) but core bosh refactoring will take place in 307. Major goals are splitting bosh into a number of modules, getting rid of old style classes (DataDict hierarchy, that is the data store singletons) and refactoring settings to efficiently support profiles. ### .data There are 544 occurrences of `.data` in 306 (1081 of `data`) as opposed to 1071 in 305 (2221 `data`) - baring string literals and comments (Pycharm only recently made these scopes available in IDE search, they were much needed). Practically one had to run the debugger to see the type of the object one had at hand, since the variable/attribute name was always 'data': - data is the wrapped dictionary of `DataDict`. Most accesses to this dict were not via the wrappers the DataDict provides. This took many commits to fix - ex 56198be. Before even _thinking_ "performance" read the performance section below (no, none of those modInfos.data.keys() calls was in the middle of a tight loop). - data also was an attribute of the Link hierarchy which had _different meaning based on an isinstance check_: 92e50cd. It could either mean a DataDict subclass (so in Links code there were `self.data.data` accesses) _or_ the list of selected items in the UI, leading to a complete chaos - see the commit above up till the final removal in ffae59e. - UIList subclasses have a `data` attribute that points to the DataDict singletons in bosh. That was dealt with throughout coding 306 as it is closely related to a deeper issue namely the intermingling of UI and model/business logic code. This stems from the 'one file program' nature of Bash so solving this properly means refactoring is at 100%. - to somehow improve readability I introduced idata and pdata attributes for the Installers and People data links - eba53f9 - Those are transitory and meant to also help in eliminating Tank. The links of List subclasses were using self.window.data, whose uses I strived to encapsulate - random example: 0934f1f - all kind of different classes attributes were named `data`. Those required dedicated commits (hey, I couldn't just do a search and replace for "data") - ex. 9839c47. An IDE really helps here. - finally data local variables were renamed - ex. 8e77283 The war with data was going on in parallel with smaller scale wars with `items` (e87dca7 and its parents), `refresh` (4d872ab), etc. ### basher (#163) This is the part of refactoring that I consider almost done - and it was a big pleasure coding it. Since I first tried coding for Bash the dichotomy between balt.Tank and basher.List clearly stood out as a major issue, as did the omnipresence of the wx library and of course the sheer size of basher.py (the UI module). Basher became a package and then the UI API went through a complete rewrite. Basher globals served as a guide in the splitting process, and a metric of class dependencies. Eliminating those globals was an ongoing effort throughout 306 coding - see for instance: 892a19c (screenlist, statusBar), 5852328 (gInstallers), faea771 (modList) till final elimination in bab7c00 Guest star: bosh.links (353be43) Globals served as a link also between Bash's modules (so they evince class and module coupling issues) - eventually encapsulated as static fields of the balt.Link.Frame singleton - see 0690cf3 Link.Frame is an interesting binding of the UI (basher) and the rest of bash that will serve as guide to a later refactoring phase. Especially Link.Frame.modList use in bosh must be eliminated but this is related to the RefreshUI/delete API finalization. The items below detail the UI refactoring. #### wx wx usages went down by an impressive 1148: 305: 2260 usages (balt: 381, basher: 1586) 306: 1112 usages (balt: 418, basher/: 494) Note balt barely going up. This is because, as was stressed in the relevant issue (#190), decoupling the wx library from the code does not simply mean "create wrappers for wx classes/constants and drop them into balt - it means that balt must export an API", so: - balt should not return wx items on which wx methods will be called - see: 9856c69 for example fix. - the code outside of balt should be agnostic of wx ids. Bash code was far form agnostic of wx ids - on the contrary it used them extensively: - balt was exporting it as a function parameter - this mostly resulted in client code calling it with `wx.ID_ANY` (adding wx usages) - see for ex. a555160, 18cf2b1 up till finally dropping the last one of them in 370bef3. - event handlers were using `event.GetId()` - for example fix see 081bfa6. - Link.Execute() has an event parameter which was mostly used with the IdList hack to define lists of menus - this was taken care by the ChoiceLink subclass, introduced in 6fbec32 till finally the last uses of IdList were binned: 000f320 and IdList was removed (967b921) - note that this permitted dropping the `_id` parameter from `Link.__init__`. ItemLink.Execute() `event` parameter is in fact _never used_ in the 306 Link API iteration and will be removed. That's "decoupling wx from Bash code". For commits that combine both fixes see: 46599bb, 9a77bfc The bulk of what remains is to hash up a clever interface for sizers - needs thought as it must accommodate all usages while being powerful enough to cater for new ones. That being said, another thought is incorporating some basher wx dependent stuff back to balt which anyway should be split to a package - at least its bosh depended classes should be clearly separated from its "abstract" wx wrapping API. #### Link API In a gigantic merge (054970e - avoid) the Link subclasses were ripped from basher to per-tab modules - this is not final but it was a necessary step to start splitting basher. As with the rip of patchers this was not a copy paste rip - a new Link class API was introduced. Unlike the patchers rip however (where package structure and the class hierarchy is still at stake) the Link API has few rough edges left in. The guide for designing the Link hierarchy was `AppendToMenu()` calls that featured boilerplate wx code - for wip boilerplate elimination commits see: 6cf40c1 where `EnabledLink` is introduced and 658fcac where my favourite `TransLink` is introduced (AppendToMenu could become quite _complex_ - see also `ChoiceLink`: 6fbec32). The merge chopped off 785 wx uses and AppendToMenu was confined in balt, and reduced to 8 different implementations vs 215 (!) in 305. The Link API significantly evolved since - compare the 306 (semi final) form with the 305 implementation: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/balt.py#L2098-L2114 306: https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/balt.py#L2392-L2446 Previous (old style) Link class explicitly defined _different_ attributes based on an isinstance check, checking if the underlying window was a List or Tank instance. This made the code plain unreadable (given also the names used, see .data section above) while enforcing the Tank/List dichotomy. The isinstance check was finally removed here: 6d260f0. Here is `INI_ListErrors` Link before and after the refactoring: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/basher.py#L11168-L11191 306: https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/basher/ini_links.py#L71-L90 Note AppendToMenu code is encapsulated in EnabledLink, wx wrapper introduced for the clipboard, text and help are now class variables, "data" is now "selected" and the lines are wrapped for more readable, declarative and concise code. For a more complicated example see: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/basher.py#L11029-L11122 306 :https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/basher/mods_links.py#L84-L161 Previous implementation directly manipulated wx IDs (by global ID_*** complex hacks) while the new one declares classes and lets ChoiceLink superclass take care of creating and appending the menu items. This permitted focusing on the Execute code which as seen got improved while refactored to remove duplication. This class still has rough edges (\_self...) which will be corrected in 307 along with rough edges in the API - for one dropping ItemLink.Execute() `event` parameter. Of note that the Link subclasses featured a lot of duplicate code apart from the wx related stuff - see for instance: 1a9a29e for duplicate code in Mod_Export Links. #### UIList After the Links were ripped from basher it became obvious that refactoring could not progress while balt.Tank and basher.List were both still around. That lead, one fine morning, to a new class - UIList: 1462227. Both Tank and List had an attribute (glist and list (!) respectively) pointing to a ListControl instance (the items displayed in the UI). The first UIList merge: abd5f24 (see commits there for how UIList was engineered from common Tank/List code) has List.list removed in favour of gList: fc7bde8, to be finally encapsulated as _gList in c48ce7d (note there is a completely unrelated `gList` patcher attribute). ##### UIList.SortItems() The first real snug though was unifying the sorting API of Tank and List. ListControl does not support sorting, one has to introduce a hacky dictionary mapping indexes to displayed items' ids (not wx ids) - that's what Tank was doing (undocumented !) while List was using no less than PopulateItems - resulting in bad performance most apparent in the ini tab, where sorting would stat the file system (so clicking on a column header Bash would hang couple seconds). I proceeded unifying the List.SortItems() overrides (6f0adc9), then moving the Tank mechanism to the ListControl class where it belongs (52a4a22) and finally moving SortItems to base: 42d213a - note OnColumnClick callback to base and ignore the isinstance(self, Tank) - corrected later. This made also various other fixups around the Sort API possible, namely adding sort indicators in all Bash lists (only mod tabs had them), fixing master lists sorting (and display) and the beginning of TankData param API deprecation: 7a3b872. ##### UIList.RefreshUI() Second snug and a blocker for fixing performance was centralizing (and cleaning up) the RefreshUI family and underlying PopulateItem(s) (UpdateItem(s) in Tank). Some commits: - Moving RefreshUI to List: 39e1e60 - List.PopulateItem(): f390ab7, ecf30e9 - UIList.PopulateItems(): 350075a - Tank and List `__init__()` eliminated: a510b23 Unifying the RefreshUI API made finally possible to remove List itself: d94a09c. More work is done - for instance see some work on refreshing the details displayed in: 3ff6cf2. ##### UIList.DeleteItems() Once List was no more and Tank a relic of itself it was finally possible to tackle the delete API. First relevant merge is: 02a5891 but the API is yet in alpha due to calling UI code from bosh - see: 6452bcb and its parent merge containing 3da60e6 (the ugly DataDict.delete_Refresh()). This API will hit beta once the data refresh methods return deleted, modified and added - see how this precious info is currently discarded: return bool(_added) or bool(_updated) or bool(_deleted) https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/bosh.py#L3887 ### Performance (#123) As seen by profiling, the performance bottleneck was the libloadorder32.dll we bundled with bash. Given the central place load order operations have in a mod manager application, the introduction of a laggy dll had taken Bash's performance down on its knees since 51f99f2. But this was far from being the only problem. Due to the spaghetti nature of the code (throw some event programming in) Bash was very generous with its refreshes: - it would refresh its data three (3) times on boot - RefreshData callback was triggered whenever Bash took focus - _including_ after Bash itself displayed a dialog to the user - on Oblivion, while Bash had refreshed its mod infos it was _always_ requesting also a refresh from libloadorder, to be sure - so, for instance, when just tabbing out and back in to Bash (with absolutely no changes in the Data dir) libloadorder was triggered and went through its refresh cycle - see ad05f44 for the fix - refresh in BAIN, say, install operations was called twice, along with a double call of modList RefreshUI - refreshing the mods panel would result in also refreshing the saves panel - even when the change that triggered the refresh did not affect saves - see commits in: 63d8dec Now, given that most of those refreshes ended up triggering a refresh from the libloadorder32.dll, Bash would take of the order of 10 seconds for all but the most trivial operations. But even if libloadorder was lightning fast the double and triple refreshes were as much a bug as anything - so the dll lag was even a blessing in disguise as it made apparent the underlying chaos. To solve the dll issue one alternative would be to reimplement it in python. But given that the same dll is used by other mod managers and related apps I anyway had to know what's in there. So I ended up forking libloadorder and going through the C++ ~~pain~~ code: https://github.com/Utumno/libloadorder/ I realized that some of the operations performed by the dll are inherently costly and would actually need a complete rewrite using caches - for instance the mod validity checks (see Utumno/libloadorder#3), which by the way involves yet another library and is not clear what is checked (as is not clear what Bash checks and if those checks are basically performed twice - Utumno/libloadorder#6). As the situation was critical in Oblivion mainly (due to stating the filesystem for mod times) I explicitly bypassed the dll because I anyway have all the data needed to determine the load order - 7cd6533 - while I still use the dll for saving it. Liblo 7.6.0 commit (f5de6c8) apart from performance fixups fixes some other issues with liblo 4 and 6 like writing loadorder.txt when not there, return active plugins in order, adding files to libloadorder.txt that are added to Data, etc - see: Utumno/libloadorder@7bb5cfb But rewriting the dll was not enough - on the Python side of things the load order internal API was a giant hack, complete with event handlers and absolutely no encapsulation, let alone proper caches. I introduced a new module `load_order.py` (for first iteration and design decisions see 98ed549) to centralize load_order handling - will be finalized in 307 but already Plugins is again encapsulated in ModInfos which serves as the basic internal load order API for the rest of Bash. Load order API will be final when implementing features such as "undo load order change" is a breeze, but already makes implementing minor load order related features much easier: 14ecafc. Using the caches I introduced I was able to fix performance of getOrdered() - was O(n^3\*logn) instead of O(n\*logn): 5d7ed0b The double and triple refreshes were a deeper issue - the "one file program" thing - so rewriting refreshes (RefreshUI, the DataDicts refresh, RefreshData etc) - was an ongoing effort throughout 306 development, going hand to hand with the UI refactoring (see #163 for RefreshUI and #123 for links to some of the most relevant commits). In short: - the refreshes on bash dialogs were taken care using a decorator to unbind RefreshData(): a71f3f2 - the BAIN issues were addressed finally here: 8107f7b - BAIN however needs to be split in a dedicated package to start properly fixing it. Goals for the next few releases: - cache ini reads - finalize then profile the load order handling API - profile the refresh of data on boot and optimize it - centralize refresh of data and UI so we can eventually - switch to an event based model of runtime refresh (watchdog ?) ### Warnings Pycharm code inspection emits 2228 warnings in 306 as opposed to 4229 in 305. Metrics are from Pycharm 4.5.4 - using this inspection: https://github.com/Utumno/wrye-bash-pycharm/blob/168253bca81313a3cc7dc85ee53747b116e984fb/.idea/inspectionProfiles/default_copy_no_typos.xml on this scope: https://github.com/Utumno/wrye-bash-pycharm/blob/168253bca81313a3cc7dc85ee53747b116e984fb/.idea/scopes/wrye_bash_all_no_cint.xml Warnings come in various sizes and shapes but in general show low code quality. Some I batch fixed (like 'Comparison to None performed with equality operator') but in general I was fixing warnings when visiting nearby code so the remaining ones mostly mark code areas that need attention. On the other end of the spectrum there are warnings that deserve issues of their own (like 'Too broad exception clause' - nicknamed the most diabolical python anti-pattern - see #200). Other interesting cases were: - "Function 'close' doesn't return anything" led to a rewrite of the archives handling code: c29d7b8 - copy paste unused variables were kind of a guide to otherwise heavily undocumented code - but also often pointed to bugs - "method may be static" (264 in 305, went down to 72 in 306) is a classic example of a warning one fixes while editing the code - the remaining ones should be revisited for 307, along with their classes. - unused imports are on their way to extinction - the remaining cases involve `import *` patterns and normalizing the import style, which shall be done when package layout is frozen - and the ugly `__all__` directives one has to update on adding a menu item: 6a2e4a9 - UILists and their links belong together. ### Cruft (#173) Cruft removal got a dedicated issue so returning developers could readily track when and why code they were familiar with got removed. Cruft is not just unused classes, commented out code etc - it is also compatibility with ancient Bash settings (will be removed in 307), non working tabs (like the PM tab also to be removed in 307) and compatibility code with pre 2.7 python (see 6d0a944). One major piece of cruft removed in 306 was BALO: b520591. As you can see removing BALO was needed for tackling refresh - as it was a complex, obsolete piece of code that greatly complicated both refresh and load order handling - and even the Link API (see notoriously complex Mod_BaloGroups(ChoiceLink) subclass here: aa5b89a). ### Formatting & wrapping After some discussion it was clear that PEP8'ing the code would be a pointless effort at the state it was in - however word wrapping is a _functional_ requirement and 79 chars is well thought of, even if you use a wall projector for programming. 305: Total number of non-blank lines: 82463 in 29 files 96.22 lines less than 80 characters (3120 lines exceed) 98.63 lines less than 100 characters (1130 lines exceed) 99.44 lines less than 120 characters (462 lines exceed) 306: Total number of non-blank lines: 85366 in 63 files 97.89 lines less than 80 characters (1803 lines exceed) 99.26 lines less than 100 characters (631 lines exceed) 99.71 lines less than 120 characters (248 lines exceed) Note: - I do not include `cint.py` (and the `chardet` package which should become a submodule) - a big part of the line wrapping was made by manually correcting Pycharm's formatter - so have a close look - the increase of line count is mainly due to 21de440 which adds 5130 lines most (4974) of it being static data in Mopy/bash/game/skyrim.py. Considering there were 34 files added for 34 * 23 = 782 lines of licence text and that wrapping should have produced another 1k lines at least _core Bash code was actually reduced by ~4000 lines_. That is more than welcome and should be imitated - _less code more features_ is a great motto. ### Boot Booting process got a facelift to be finalized in 307. This involves game loading (see 97fc9cf), fixes to the game detection algorithm (e19237c) which used to run 3-5 times on boot, a fix for our custom importer (43e359a) compliments of @mjpieters, and finally a reworking of boot error messages (8dbdd49). ### WINE As reported in #203 "WryeBash v304.2 works beautifully in Wine" - the windows only shell UI stuff broke it though. One of the last things I did for 306 was fixing this - now Bash runs again on WINE: 124f314. Still making it run correctly and in a second phase making Bash run _on bare Linux_ is a code quality goal that got dedicated issues (#240, #241, #243) ------------------------------------------------------------------------ This release is dedicated to Wrye. Special thanks to: - @Sharlikran, @Supierce, @lojack5, @Arthmoor, @alt3rn1ty, @psilocide (aka lefenger), @leandor for testing, support and sharing their knowledge, - @jfpelland, @wduda, @Isenr and @valda for patches, - @mjpieters for saving my sanity at SO a couple times as well as @zed for helping me rewrite the archives handling code. ------------------------------------------------------------------------ # Conflicts: # Mopy/bash/bass.py @@@ -21,9 -21,42 +21,46 @@@ # https://github.com/wrye-bash # # ============================================================================= """This module just stores some data that all modules have to be able to access - without worrying about circular imports.""" + without worrying about circular imports. Currently used to expose layout + and environment issues - do not modify or imitate (ut).""" + import os as _os + import ConfigParser as _cp language = None <<<<<<< HEAD +AppVersion = u"304.4" ======= + AppVersion = u"306" + bashIni = None + + #--Null strings (for default empty byte arrays) + null1 = '\x00' + null2 = null1*2 + null3 = null1*3 + null4 = null1*4 + + def GetBashIni(iniPath=None, reload_=False): ##: needs work + iniPath = iniPath or u'bash.ini' + global bashIni + if reload_ or bashIni is None: + if _os.path.exists(iniPath): + bashIni = _cp.ConfigParser() + bashIni.read(iniPath) + return bashIni + + class Resources: # this belongs to basher but leads to cyclic imports, so... + fonts = None + #--Icon Bundles + bashRed = None + bashBlue = None + bashDocBrowser = None + bashMonkey = None + + # move with its uses to a cross platform 'env.py' module - maybe add bashIni + try: + import _winreg as winreg + except ImportError: + winreg = None >>>>>>> rel-306 ------------------------------------------------------------------------ Build with `2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)]` Closes #187.

This release features an almost complete refactoring of the codebase, fixing many bugs and opening the way to extend Bash functionality. Bash, being a community work, has over the years become an assortment of hacks, patches and cruft and the program was just about to become non-operable. Example issues - in no particular order (although probably the worst was the horrible performance): - deleting an item in the lists displayed by Bash ran different code depending on whether the user hit delete or right clicked and chose delete - most of those pieces of code were buggy - start up took more than 15 seconds on large Oblivion Load Orders and alt tabbing out and back into Bash would take 6-7 seconds - last one should take no time in the usual case - as seen from the (user visible) debug prints the game detection algorithm run 3-5 times - many, many broken things - including performance in general (most simple operations would hang Bash for some seconds), display glitches and general UI inconsistencies and internal data structures corruption (see #176) Those bugs reflected the state of the code - again in no particular order: - bosh.py (the data model) in 305 was 24635 lines (reduced from 30550 lines in 304.4 - see bosh section below) - basher.py (the UI lists/menus etc) was 18936 lines - the seven tabs that Bash featured were backed up by Tank (2) and List (5) instances - so implementing a feature (say delete) needed edits to at least two classes (usually 7) - the wx library was intermingled with the basher code - the menu items (Links) defined 215 AppendToMenu methods with nearly identical wx code - readability suffered - most attributes/variables were named `data` or `items`, globals hacks crippled the IDE static analysis, copy paste (aka ravioli) code was everywhere. When I started in it I had no idea if refactoring the code was even possible, core Bash is 80k lines of code. It was clear that bosh needed splitting, that candidate number one was the patchers code (well defined and functional) and that basher should follow - but was not at all clear if those were possible, let alone how exactly should the code be structured and how much time would this take. Turns out that it was possible: Bash was indeed coded with modularity in mind (by Wrye ?) but later additions bloated the initial modules beyond recognition and broke the (non enforced) encapsulation - however it wasn't too late. Also turns out that refactoring needed 2 full work years of a single engineer (learning python). The huge performance boost that this release features is the measurable effect of the increase in the code quality. In the items below I try to describe in a quantitative manner what "code quality" means and how it was increased. That's meant to be read from an app that links to the commits - both github and gitk do but additionally github links to the issues and files. This release features splitting the patchers' code out of bosh to a dedicated package (whose final structure is still at stake). This reduced bosh to 10516 lines while naturally exposing class dependencies issues, hidden by the "one file program" nature of Bash. The attempt to split bosh had already started in 305, involving: - an effort to eliminate code duplication in C and P patchers using multiple inheritance (chopped off around 1500 lines of pure copy paste code) - see commits in 3c40989 - splitting out what was to become `record_groups.py` - see c3e3543, but don't imitate it - lots has changed since then, including import conventions and commit style - for instance commits on dev that do not launch are strictly forbidden. - splitting out what was to become `parsers.py` - see commits in 08ab6e2 Final (`record_groups.py`, `parsers.py`): 6ae30fc With those out of the way it was finally possible to rip the patchers out of bosh - starting here: 983f259 That was not a mere copy paste rip - warnings were squashed, long lines were wrapped, and most importantly a lot of tricky refactoring was applied to eliminate code duplication - see for ex. commits in: 72fc8a0 for ripping and commits in (note the negative line count): 5d21377, fdab163, 78a85e0 for refactoring See #3 for commit links (finally closed in b6b743b) and #124 for the (still open) package layout issue. Be aware that a lot of things changed since, including import style and the overall package layout - see: e6b2e0e The remaining bosh.py code had some preliminary refactoring applied to it (archives handling, BAIN error handling and refresh etc - see 6ddd4a8) but core bosh refactoring will take place in 307. Major goals are splitting bosh into a number of modules, getting rid of old style classes (DataDict hierarchy, that is the data store singletons) and refactoring settings to efficiently support profiles. There are 544 occurrences of `.data` in 306 (1081 of `data`) as opposed to 1071 in 305 (2221 `data`) - baring string literals and comments (Pycharm only recently made these scopes available in IDE search, they were much needed). Practically one had to run the debugger to see the type of the object one had at hand, since the variable/attribute name was always 'data': - data is the wrapped dictionary of `DataDict`. Most accesses to this dict were not via the wrappers the DataDict provides. This took many commits to fix - ex 56198be. Before even _thinking_ "performance" read the performance section below (no, none of those modInfos.data.keys() calls was in the middle of a tight loop). - data also was an attribute of the Link hierarchy which had _different meaning based on an isinstance check_: 92e50cd. It could either mean a DataDict subclass (so in Links code there were `self.data.data` accesses) _or_ the list of selected items in the UI, leading to a complete chaos - see the commit above up till the final removal in ffae59e. - UIList subclasses have a `data` attribute that points to the DataDict singletons in bosh. That was dealt with throughout coding 306 as it is closely related to a deeper issue namely the intermingling of UI and model/business logic code. This stems from the 'one file program' nature of Bash so solving this properly means refactoring is at 100%. - to somehow improve readability I introduced idata and pdata attributes for the Installers and People data links - eba53f9 - Those are transitory and meant to also help in eliminating Tank. The links of List subclasses were using self.window.data, whose uses I strived to encapsulate - random example: 0934f1f - all kind of different classes attributes were named `data`. Those required dedicated commits (hey, I couldn't just do a search and replace for "data") - ex. 9839c47. An IDE really helps here. - finally data local variables were renamed - ex. 8e77283 The war with data was going on in parallel with smaller scale wars with `items` (e87dca7 and its parents), `refresh` (4d872ab), etc. This is the part of refactoring that I consider almost done - and it was a big pleasure coding it. Since I first tried coding for Bash the dichotomy between balt.Tank and basher.List clearly stood out as a major issue, as did the omnipresence of the wx library and of course the sheer size of basher.py (the UI module). Basher became a package and then the UI API went through a complete rewrite. Basher globals served as a guide in the splitting process, and a metric of class dependencies. Eliminating those globals was an ongoing effort throughout 306 coding - see for instance: 892a19c (screenlist, statusBar), 5852328 (gInstallers), faea771 (modList) till final elimination in bab7c00 Guest star: bosh.links (353be43) Globals served as a link also between Bash's modules (so they evince class and module coupling issues) - eventually encapsulated as static fields of the balt.Link.Frame singleton - see 0690cf3 Link.Frame is an interesting binding of the UI (basher) and the rest of bash that will serve as guide to a later refactoring phase. Especially Link.Frame.modList use in bosh must be eliminated but this is related to the RefreshUI/delete API finalization. The items below detail the UI refactoring. wx usages went down by an impressive 1148: 305: 2260 usages (balt: 381, basher: 1586) 306: 1112 usages (balt: 418, basher/: 494) Note balt barely going up. This is because, as was stressed in the relevant issue (#190), decoupling the wx library from the code does not simply mean "create wrappers for wx classes/constants and drop them into balt - it means that balt must export an API", so: - balt should not return wx items on which wx methods will be called - see: 9856c69 for example fix. - the code outside of balt should be agnostic of wx ids. Bash code was far form agnostic of wx ids - on the contrary it used them extensively: - balt was exporting it as a function parameter - this mostly resulted in client code calling it with `wx.ID_ANY` (adding wx usages) - see for ex. a555160, 18cf2b1 up till finally dropping the last one of them in 370bef3. - event handlers were using `event.GetId()` - for example fix see 081bfa6. - Link.Execute() has an event parameter which was mostly used with the IdList hack to define lists of menus - this was taken care by the ChoiceLink subclass, introduced in 6fbec32 till finally the last uses of IdList were binned: 000f320 and IdList was removed (967b921) - note that this permitted dropping the `_id` parameter from `Link.__init__`. ItemLink.Execute() `event` parameter is in fact _never used_ in the 306 Link API iteration and will be removed. That's "decoupling wx from Bash code". For commits that combine both fixes see: 46599bb, 9a77bfc The bulk of what remains is to hash up a clever interface for sizers - needs thought as it must accommodate all usages while being powerful enough to cater for new ones. That being said, another thought is incorporating some basher wx dependent stuff back to balt which anyway should be split to a package - at least its bosh depended classes should be clearly separated from its "abstract" wx wrapping API. In a gigantic merge (054970e - avoid) the Link subclasses were ripped from basher to per-tab modules - this is not final but it was a necessary step to start splitting basher. As with the rip of patchers this was not a copy paste rip - a new Link class API was introduced. Unlike the patchers rip however (where package structure and the class hierarchy is still at stake) the Link API has few rough edges left in. The guide for designing the Link hierarchy was `AppendToMenu()` calls that featured boilerplate wx code - for wip boilerplate elimination commits see: 6cf40c1 where `EnabledLink` is introduced and 658fcac where my favourite `TransLink` is introduced (AppendToMenu could become quite _complex_ - see also `ChoiceLink`: 6fbec32). The merge chopped off 785 wx uses and AppendToMenu was confined in balt, and reduced to 8 different implementations vs 215 (!) in 305. The Link API significantly evolved since - compare the 306 (semi final) form with the 305 implementation: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/balt.py#L2098-L2114 306: https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/balt.py#L2392-L2446 Previous (old style) Link class explicitly defined _different_ attributes based on an isinstance check, checking if the underlying window was a List or Tank instance. This made the code plain unreadable (given also the names used, see .data section above) while enforcing the Tank/List dichotomy. The isinstance check was finally removed here: 6d260f0. Here is `INI_ListErrors` Link before and after the refactoring: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/basher.py#L11168-L11191 306: https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/basher/ini_links.py#L71-L90 Note AppendToMenu code is encapsulated in EnabledLink, wx wrapper introduced for the clipboard, text and help are now class variables, "data" is now "selected" and the lines are wrapped for more readable, declarative and concise code. For a more complicated example see: 305: https://github.com/wrye-bash/wrye-bash/blob/d8f05202e6485a3bef0d92bb55a731b8040eb94e/Mopy/bash/basher.py#L11029-L11122 306 :https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/basher/mods_links.py#L84-L161 Previous implementation directly manipulated wx IDs (by global ID_*** complex hacks) while the new one declares classes and lets ChoiceLink superclass take care of creating and appending the menu items. This permitted focusing on the Execute code which as seen got improved while refactored to remove duplication. This class still has rough edges (\_self...) which will be corrected in 307 along with rough edges in the API - for one dropping ItemLink.Execute() `event` parameter. Of note that the Link subclasses featured a lot of duplicate code apart from the wx related stuff - see for instance: 1a9a29e for duplicate code in Mod_Export Links. After the Links were ripped from basher it became obvious that refactoring could not progress while balt.Tank and basher.List were both still around. That lead, one fine morning, to a new class - UIList: 1462227. Both Tank and List had an attribute (glist and list (!) respectively) pointing to a ListControl instance (the items displayed in the UI). The first UIList merge: abd5f24 (see commits there for how UIList was engineered from common Tank/List code) has List.list removed in favour of gList: fc7bde8, to be finally encapsulated as _gList in c48ce7d (note there is a completely unrelated `gList` patcher attribute). The first real snug though was unifying the sorting API of Tank and List. ListControl does not support sorting, one has to introduce a hacky dictionary mapping indexes to displayed items' ids (not wx ids) - that's what Tank was doing (undocumented !) while List was using no less than PopulateItems - resulting in bad performance most apparent in the ini tab, where sorting would stat the file system (so clicking on a column header Bash would hang couple seconds). I proceeded unifying the List.SortItems() overrides (6f0adc9), then moving the Tank mechanism to the ListControl class where it belongs (52a4a22) and finally moving SortItems to base: 42d213a - note OnColumnClick callback to base and ignore the isinstance(self, Tank) - corrected later. This made also various other fixups around the Sort API possible, namely adding sort indicators in all Bash lists (only mod tabs had them), fixing master lists sorting (and display) and the beginning of TankData param API deprecation: 7a3b872. Second snug and a blocker for fixing performance was centralizing (and cleaning up) the RefreshUI family and underlying PopulateItem(s) (UpdateItem(s) in Tank). Some commits: - Moving RefreshUI to List: 39e1e60 - List.PopulateItem(): f390ab7, ecf30e9 - UIList.PopulateItems(): 350075a - Tank and List `__init__()` eliminated: a510b23 Unifying the RefreshUI API made finally possible to remove List itself: d94a09c. More work is done - for instance see some work on refreshing the details displayed in: 3ff6cf2. Once List was no more and Tank a relic of itself it was finally possible to tackle the delete API. First relevant merge is: 02a5891 but the API is yet in alpha due to calling UI code from bosh - see: 6452bcb and its parent merge containing 3da60e6 (the ugly DataDict.delete_Refresh()). This API will hit beta once the data refresh methods return deleted, modified and added - see how this precious info is currently discarded: return bool(_added) or bool(_updated) or bool(_deleted) https://github.com/wrye-bash/wrye-bash/blob/b40b1a10ff414dd41dd412c8484ca253e42ca92c/Mopy/bash/bosh.py#L3887 As seen by profiling, the performance bottleneck was the libloadorder32.dll we bundled with bash. Given the central place load order operations have in a mod manager application, the introduction of a laggy dll had taken Bash's performance down on its knees since 51f99f2. But this was far from being the only problem. Due to the spaghetti nature of the code (throw some event programming in) Bash was very generous with its refreshes: - it would refresh its data three (3) times on boot - RefreshData callback was triggered whenever Bash took focus - _including_ after Bash itself displayed a dialog to the user - on Oblivion, while Bash had refreshed its mod infos it was _always_ requesting also a refresh from libloadorder, to be sure - so, for instance, when just tabbing out and back in to Bash (with absolutely no changes in the Data dir) libloadorder was triggered and went through its refresh cycle - see ad05f44 for the fix - refresh in BAIN, say, install operations was called twice, along with a double call of modList RefreshUI - refreshing the mods panel would result in also refreshing the saves panel - even when the change that triggered the refresh did not affect saves - see commits in: 63d8dec Now, given that most of those refreshes ended up triggering a refresh from the libloadorder32.dll, Bash would take of the order of 10 seconds for all but the most trivial operations. But even if libloadorder was lightning fast the double and triple refreshes were as much a bug as anything - so the dll lag was even a blessing in disguise as it made apparent the underlying chaos. To solve the dll issue one alternative would be to reimplement it in python. But given that the same dll is used by other mod managers and related apps I anyway had to know what's in there. So I ended up forking libloadorder and going through the C++ ~~pain~~ code: https://github.com/Utumno/libloadorder/ I realized that some of the operations performed by the dll are inherently costly and would actually need a complete rewrite using caches - for instance the mod validity checks (see Utumno/libloadorder#3), which by the way involves yet another library and is not clear what is checked (as is not clear what Bash checks and if those checks are basically performed twice - Utumno/libloadorder#6). As the situation was critical in Oblivion mainly (due to stating the filesystem for mod times) I explicitly bypassed the dll because I anyway have all the data needed to determine the load order - 7cd6533 - while I still use the dll for saving it. Liblo 7.6.0 commit (f5de6c8) apart from performance fixups fixes some other issues with liblo 4 and 6 like writing loadorder.txt when not there, return active plugins in order, adding files to libloadorder.txt that are added to Data, etc - see: Utumno/libloadorder@7bb5cfb But rewriting the dll was not enough - on the Python side of things the load order internal API was a giant hack, complete with event handlers and absolutely no encapsulation, let alone proper caches. I introduced a new module `load_order.py` (for first iteration and design decisions see 98ed549) to centralize load_order handling - will be finalized in 307 but already Plugins is again encapsulated in ModInfos which serves as the basic internal load order API for the rest of Bash. Load order API will be final when implementing features such as "undo load order change" is a breeze, but already makes implementing minor load order related features much easier: 14ecafc. Using the caches I introduced I was able to fix performance of getOrdered() - was O(n^3\*logn) instead of O(n\*logn): 5d7ed0b The double and triple refreshes were a deeper issue - the "one file program" thing - so rewriting refreshes (RefreshUI, the DataDicts refresh, RefreshData etc) - was an ongoing effort throughout 306 development, going hand to hand with the UI refactoring (see #163 for RefreshUI and #123 for links to some of the most relevant commits). In short: - the refreshes on bash dialogs were taken care using a decorator to unbind RefreshData(): a71f3f2 - the BAIN issues were addressed finally here: 8107f7b - BAIN however needs to be split in a dedicated package to start properly fixing it. Goals for the next few releases: - cache ini reads - finalize then profile the load order handling API - profile the refresh of data on boot and optimize it - centralize refresh of data and UI so we can eventually - switch to an event based model of runtime refresh (watchdog ?) Pycharm code inspection emits 2228 warnings in 306 as opposed to 4229 in 305. Metrics are from Pycharm 4.5.4 - using this inspection: https://github.com/Utumno/wrye-bash-pycharm/blob/168253bca81313a3cc7dc85ee53747b116e984fb/.idea/inspectionProfiles/default_copy_no_typos.xml on this scope: https://github.com/Utumno/wrye-bash-pycharm/blob/168253bca81313a3cc7dc85ee53747b116e984fb/.idea/scopes/wrye_bash_all_no_cint.xml Warnings come in various sizes and shapes but in general show low code quality. Some I batch fixed (like 'Comparison to None performed with equality operator') but in general I was fixing warnings when visiting nearby code so the remaining ones mostly mark code areas that need attention. On the other end of the spectrum there are warnings that deserve issues of their own (like 'Too broad exception clause' - nicknamed the most diabolical python anti-pattern - see #200). Other interesting cases were: - "Function 'close' doesn't return anything" led to a rewrite of the archives handling code: c29d7b8 - copy paste unused variables were kind of a guide to otherwise heavily undocumented code - but also often pointed to bugs - "method may be static" (264 in 305, went down to 72 in 306) is a classic example of a warning one fixes while editing the code - the remaining ones should be revisited for 307, along with their classes. - unused imports are on their way to extinction - the remaining cases involve `import *` patterns and normalizing the import style, which shall be done when package layout is frozen - and the ugly `__all__` directives one has to update on adding a menu item: 6a2e4a9 - UILists and their links belong together. Cruft removal got a dedicated issue so returning developers could readily track when and why code they were familiar with got removed. Cruft is not just unused classes, commented out code etc - it is also compatibility with ancient Bash settings (will be removed in 307), non working tabs (like the PM tab also to be removed in 307) and compatibility code with pre 2.7 python (see 6d0a944). One major piece of cruft removed in 306 was BALO: b520591. As you can see removing BALO was needed for tackling refresh - as it was a complex, obsolete piece of code that greatly complicated both refresh and load order handling - and even the Link API (see notoriously complex Mod_BaloGroups(ChoiceLink) subclass here: aa5b89a). After some discussion it was clear that PEP8'ing the code would be a pointless effort at the state it was in - however word wrapping is a _functional_ requirement and 79 chars is well thought of, even if you use a wall projector for programming. 305: Total number of non-blank lines: 82463 in 29 files 96.22 lines less than 80 characters (3120 lines exceed) 98.63 lines less than 100 characters (1130 lines exceed) 99.44 lines less than 120 characters (462 lines exceed) 306: Total number of non-blank lines: 85366 in 63 files 97.89 lines less than 80 characters (1803 lines exceed) 99.26 lines less than 100 characters (631 lines exceed) 99.71 lines less than 120 characters (248 lines exceed) Note: - I do not include `cint.py` (and the `chardet` package which should become a submodule) - a big part of the line wrapping was made by manually correcting Pycharm's formatter - so have a close look - the increase of line count is mainly due to 21de440 which adds 5130 lines most (4974) of it being static data in Mopy/bash/game/skyrim.py. Considering there were 34 files added for 34 * 23 = 782 lines of licence text and that wrapping should have produced another 1k lines at least _core Bash code was actually reduced by ~4000 lines_. That is more than welcome and should be imitated - _less code more features_ is a great motto. Booting process got a facelift to be finalized in 307. This involves game loading (see 97fc9cf), fixes to the game detection algorithm (e19237c) which used to run 3-5 times on boot, a fix for our custom importer (43e359a) compliments of mjpieters, and finally a reworking of boot error messages (8dbdd49). As reported in #203 "WryeBash v304.2 works beautifully in Wine" - the windows only shell UI stuff broke it though. One of the last things I did for 306 was fixing this - now Bash runs again on WINE: 124f314. Still making it run correctly and in a second phase making Bash run _on bare Linux_ is a code quality goal that got dedicated issues (#240, #241, #243) ------------------------------------------------------------------------ This release is dedicated to Wrye. Special thanks to: - Sharlikran, Supierce, lojack5, Arthmoor, alt3rn1ty, psilocide (aka lefenger), leandor for testing, support and sharing their knowledge, - jfpelland, wduda, Isenr and valda for patches, - mjpieters for saving my sanity at SO a couple times as well as zed for helping me rewrite the archives handling code. ------------------------------------------------------------------------ Build with `2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)]` Closes #187.

When I say I want to decouple wx from Bash I mean it - actually this was perfectly possible in 306, just realized it while writing 1cd839f Note using super in: Mopy/bash/basher/app_buttons.py Mopy/bash/basher/ini_links.py Mopy/bash/basher/misc_links.py Mopy/bash/basher/mods_links.py Dunno why was not using it before (relic from when Link was old style class ?) Note also ugly `event_` shadowing gone This drops some unused parameters in mod_links.py (_doRefresh...) and in mods_links.py Double underscore for discarded parameters/internal parameters Mopy/bash/basher/app_buttons.py: Took the opportunity to eliminate the extra arguments in Execute() overrides (took some warnings for the ride). Cleaner, still needs work - note win32gui for instance... kwdargs.pop('onClick', None) would always return None - dropped Also dropped """Handle menu selection.""" Note also that in new files the formatter now makes minimal changes - missing space in (self,event) gone Under #174, #163, #190.

Clearly differentiate between buttons that need to process the event - so we can refactor this later. onClick is a bad name - used everywhere - this will change soon patcher_dialog.PatchDialog.Execute() -> PatchExecute(), leave Execute for links. Under #190 - see also next commits

Leftovers from 306 in the Link API (and in general #163, #174, #190). Most notably, Execute's event parameter was _usnused_ - eliminating it not only took a lot of unused parameters for the ride but from an engineering point of view greatly contributes in decoupling wx from Bash - as events now are much more easily isolated - and events belong to wx. `event` occurrences went down to 325 from 608 (!) in the code 59 warnings were corrected Signed-off-by: MrD <[email protected]>

Takes an import wx along - from ___bosh___. Under #190, basically this whole merge was triggered by this frigging import. TODO: better error handling - this be better called on boot We have to add `self.vdata['boltPaths'] = True` by hand cause otherwise settings created by 307 will blow on 306 - cause it won't find boltPaths attribute and run the updatePaths() which will blow: Wrye Bash starting Using Wrye Bash Version 307 Python version: 2.7.10 wxPython version: 2.8.12.1 (msw-unicode) input encoding: cp437; output encoding: None; locale: ('en_US', 'cp1252') filesystem encoding: mbcs Searching for game to manage: bush.py 87 _supportedGames: Detected the following supported games via Windows Registry: bush.py 147 _detectGames: Detecting games via the -o argument, bash.ini and relative path: bush.py 153 _detectGames: Set game mode to oblivion specified via -o argument: \mnt\win\GAMES\TESIV\Oblivion bush.py 187 setGame: No preferred game specified. bush.py 168 __setGame: Using oblivion game: \mnt\win\GAMES\TESIV\Oblivion testing UAC bosh.py 5224 __init__: Using libbsa API version: 2.0.0 bosh.py 5230 __init__: Using LOOT API version: 0.6.0 load_order.py 321 <module>: Using libloadorder API version: 7.6.0 # Generating comtypes.gen.SHDocVw Traceback (most recent call last): File "/mnt/win/Dropbox/eclipse_workspaces/python/wrye-bash/Mopy/Wrye Bash Launcher.pyw", line 88, in <module> bash.main() File "Z:\mnt\win\Dropbox\eclipse_workspaces\python\wrye-bash\Mopy\bash\bash.py", line 466, in main frame = app.Init() # Link.Frame is set here ! File "bash\basher\__init__.py", line 4347, in Init self.InitData(progress) File "bash\basher\__init__.py", line 4379, in InitData bosh.modInfos = bosh.ModInfos() File "bash\bosh.py", line 4038, in __init__ FileInfos.__init__(self,dirs['mods'],ModInfo) File "bash\bosh.py", line 3818, in __init__ self._initDB(dir_) File "bash\bosh.py", line 3807, in _initDB self.bashDir.join(u'Table.pkl'))) File "Z:\mnt\win\Dropbox\eclipse_workspaces\python\wrye-bash\Mopy\bash\bolt.py", line 1748, in __init__ dictFile.load() File "bash\bosh.py", line 200, in load self.updatePaths() File "bash\bosh.py", line 251, in updatePaths update(self.data) File "bash\bosh.py", line 232, in update xnew = dict((update(key),update(value)) for key,value in x.iteritems()) File "bash\bosh.py", line 232, in <genexpr> xnew = dict((update(key),update(value)) for key,value in x.iteritems()) File "bash\bosh.py", line 245, in update changed.add(x._path) AttributeError: 'Path' object has no attribute '_path' So: @@ -1507,2 +1507,3 @@ def save(self): #--Pickle it + self.vdata['boltPaths'] = True with self.path.temp.open('wb') as out:

Under #190. Note that in InstallersList.OnDClick event was not skipped `if not item` Now it is skipped, for better or worse (seems to be for no difference). On a closer look the event was sometimes skipped sometimes not - left the mess intact (apart from InstallersList) but one of these days this must be cleaned up. ModList.OnLeftDown should be left alone though - the event should only be skipped as it is - I wonder what's the purpose of the SetDnD call.

Under #190. 347 -> 338 occurences of `event` - all dropped from basher

Rename methods to more descriptive names and built the Panel around an OrderedDict, got me rid of self.sortKeys. ini panel will now remember the lastDir over a restart (I added 'bash.ini.lastDir' setting). The IDE rearranged the imports left them that way. Under #190 too, WIP - I should move OnSelect there too.

Renamed methods to more descriptive names and built the Panel around an OrderedDict, got me rid of self.sortKeys. ini panel will now remember the lastDir over a restart (I added 'bash.ini.lastDir' setting). The IDE rearranged the imports left them that way. Under #190 too for common wx code of tweaks controls in balt, WIP - I should move OnSelect there too. Did some silly refactoring on INIListCtrl but I should just DeleteAll when the ini changes - has a ridiculous lag when changing from a game ini to a (much) smaller one And a fixup for #290: @@ -1639,2 +1625,3 @@ def SetBaseIni(self,path=None): self.button.Enable(not isGameIni) + self.editButton.Enable(not isGameIni or target.path.isfile()) selected = None

I loose of course but every single commit in this branch is a victory. 1075 wx usages in df2fcc5 of which 476 in basher/, 104 in belt, 58 in bash and the rest 436 in balt. 923 usages in this merge, of which 380 in basher/, 67 in belt, 56 in Bash and the rest 425 in balt. So usages dropped below 1000, and actually below 500 outside balt. This almost closes the long standing #190 - I leave it open for now to try and move some more stuff to balt (the BashStatusBar DnD) and investigate more the sizers uses to try and extend the API to absorb more wx uses. Signed-off-by: MrD <[email protected]>

Under #190.

Phoenix compatibility; ..also, I don't see anything special that is being used anywhere that SplitterWindow doesn't have already. Under #190, #15.

@Metallicow

De-wx'ing: this merge counts 902 uses of wx in the code, 454 of which (more than half) in balt. Puts #190 in the backburner for now - see there for what remains to be done. There is some work towards master UI issue, #163, namely, at long last, UIList._gList control is encapsulated inside the UIList - a major goal of 306, finally attended - compare this to the Tank/List mess on 305. The merge culminates in a commit by @Metallicow removing the wx.gizmos dependency - a blocker for #15. Signed-off-by: MrD <[email protected]>

General maintenance - features: - a UIList namespace cleanup and wrapping UIListCtrl._native_widget - some good old de-wx'ing - creating gui wrappers for dialogs - 115 -> 88 uses of wx in balt - moving part of balt that is coupled with Link/UIList to gui (ask/show functions) - note askContinue is tightly coupled with balt constants. - some cleanup of the status bar - this one is entangled with Link like no tomorrow - its buttons should probably become ClickableImages, not Links. The size change crashes on me big time - may be related to events left unhandled, but whatever the cause we need a more transparent model to see what's going on - first attempts here by using dicts instead of lists to simplify the code - GetLink gone. That being said we are now in the last 10% - which has the last 10% kind of complexity. In particular, it's now clear that balt is responsible for binding with the {settings, sizes} globals - and although one can easily hack those out of there, a more elegant solution is wanted. Another issue that is almost done is the coupling of bosh/env with balt/gui - needs also a careful solution. Under #190. Signed-off-by: MrD <[email protected]>

Part 2 of the #190 merge getting rid of one of the ugliest APIs in Wrye Bash's codebase, balt.ListBoxes. See the merge commit of the first part for details, I really don't want to write that novel again: d1290a5 Under #190

@Utumno

I wanted this release to be smaller for once. 500 commits and nearly a year later, mission failed :( Hopefully we'll manage to keep the next one smaller. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525, #650) The first major change in 311 was the beginning of the road towards proper Fallout 4 records support and hence a fully function Bashed Patch in FO4. - 00c9ee8 implemented a key piece of the records infrastructure, NVNM. It also dropped some hacks to further pave the way. The first two FO4 record types were implemented. - dc6311a refactored CTDA and implemented the next seven record types. - 641bc73 tackled the single biggest challenge of the entire undertaking: VMAD. As a bonus, if you enjoy rants about stupid design decisions, see c5b5d8a. The next ten record types were also implemented. - 28578a6 moved some record types that are identical between Skyrim and FO4 to brec and implemented the next eighteen records. - This was immediately followed by bb7e5ae, which slightly refactored GameInfo.init() but mostly just implemented the next twelve record types. - 3d8b23e implemented nine more record types. - 9bfb6de implemented two more. Also, @BeermotorWB began the porting process by bringing Tweak Settings to FO4 in 57b2443. There are about sixty FO4 record types left to implement. This will be concluded in 312. I chose this part for the title of the release ("Fallout 4 preparation") since it's what occupied my brain during most of 311's development. ----------------------------------------------------------------------- ### Python 3 (#618, #619, #644) Since moving to Python 3 in 310 (see #460), we've been taking full advantage of everything py3 offers us. Some example commits: - 8de5f40 cleaned up one of the most annoying files to edit, common_subrecords.py. - cc198c9 shaved hundreds of lines off by replacing verbose nested OrderedDict instantiations with regular, now-ordered dicts. - fd1d4d9 took advantage of ordered dicts to drop duplication between the records list we used to keep track of records in a Mob* class and the id_records dict we used for fast random access by FormID. Now we only need the dict. - In bbb8fa2, @lojack5 used Python 3's type annotations to refactor our flags handling. We also upgraded to Python 3.10 in 7b3573c and to Python 3.11 in 69e7e3a. We also optimized a bunch of central code for e.g. the BP, FName and Path to take advantage of changes like 3.11's adaptive interpreter, which gave pretty good speedups. We've also been doing what we call 'fdusting' - replacing any random pieces of code that look like a + '.esp' with f-strings, i.e. f'{a}.esp'. Another (made-up) example: 'Loaded %d images' % len(images) becomes f'Loaded {len(images)} images'. This has been happening as a sort of "if you're editing in the vicinity, fix any you touch" thing, so spread out over tons of commits. F-strings are one of the best things about py3 :) ----------------------------------------------------------------------- ### Records (#480, #647) The eternal #480 rears its head again, as it does every release. First off, every single merge mentioned in the FO4 section above also falls under this section. In addition to that, we are approaching setDefault. Uprooting that will give us major speed boosts, simplify code, fix lots of random "the BP is editing this value for some reason" bugs (all of those are due to defaults), etc. One major blocker was FormID handling (see section "FormIDs (#637)" below), now cleaned up. Some other commits related to records refactoring in 311: - 28e0734 reworked defaults handling, making defaults explicit where they are needed and eliminating them from most structures in the records code. - fd1d4d9, already mentioned in the "Python 3 (#618, #619, #644)" section above, reduced duplication in the record_groups code by getting rid of the 'self.records' list in favor of just the (thanks to py3, now ordered) 'self.id_records' dict. - 71520f7 massively reduced code duplication in the various games' init() methods and the __slots__ definitions for all the record types for a total of nearly 1000 LOC removed! - bbb8fa2 by @lojack5, already mentioned in the "Python 3 (#618, #619, #644)" section above, took advantage of Python 3's type hints to refactor our flags handling. The biggest merge of this development cycle also falls in this category. In a truly herculean merge (d2152aa), @Utumno tackled record_groups, turning it from a TODO-riddled mess that didn't really know if it wanted to be Oblivion-only or game-agnostic into a modern API that will serve us well for the future (e.g. for FO4, which makes QUST a new top-level group). Seriously, this refactoring was so complex, its merge has two sub-merges. ----------------------------------------------------------------------- ### FormIDs (#637) One of the major refactoring goals of 311 was tackling FormIDs. We used to repesent them as either integers (called "short FormIDs") or tuples of strings and integers (called "long FormIDs"). Note that the latter became tuples of FNames and integers in 310 (see section "FName (#543)" in the 310 release commit. This was achieved in 4964710, which is once again an entire branch squashed down to a commit so as to not break dev. They are now stored as a class instead (brec.FormId). This required careful engineering and testing to make sure we don't kill BP performance. For the full story, see the linked commit. ----------------------------------------------------------------------- ### Localization (#55, #500) We're trying to replace all '%s' specifiers with more useful '%(something)s' specifiers. This is still nowhere close to being finished, but 2442285 began the process. Once this is done, we'll want to rethink how we allow people to contribute localizations - see #500 for more info. ----------------------------------------------------------------------- ### Epic Games Store & GOG (#646, #648) No release is complete without adding support for new games, and 311 is no exception. Bethesda released most of their games on GOG and some on the Epic Games Store, so we added support. See the linked issues and the referencing commits for more information. The biggest difficulty was adding support for yet *another* method of game detection (for the Epic Games Store). Still to come is refactoring the game detection on the Steam side, since some of these rereleases will actually overwrite the registry key used by the Steam versions, making it impossible for WB to reliably detect the installation of both versions. This will also pave the way for Proton-based game detection on Linux (see "Linux (#243)" section below). ----------------------------------------------------------------------- ### Nexus integration (#459) 5e688fd prepared us for proper Nexus integration, which will be one of the big feature goals of 312, by merging a nexus.py file that wraps the Nexus API into the source tree so that keeping it up to date is easier. ----------------------------------------------------------------------- ### FileInfos (#336) Some refactoring work on FileInfos and related APIs occurred as well, mostly on the INIs. See 22c8c5e, 2495972 and b3e4116 for details. ----------------------------------------------------------------------- ### wx (#190) That's right, more of this. - 6f2bd1d finally de-wx's the mighty UIList (which backs all of WB's tabs) by making it inherit from our PanelWin instead of wx.Panel. Turns out it was that easy (I'm sure it had nothing to do with the metric tons of refactoring that had been done towards this in 310, 309, 308 and 307). - f7dd4dc decouples us from wx with regards to the default, builtin art from wx we used to use. We instead now use SVGs (mostly from Font Awesome - don't worry, I was careful with the licensing), which also gives us some consistency across platforms. - d1290a5 begins the fight against balt.ListBoxes, an absolutely horrible class that not only had a really weird API but also featured terrible usability. I wrote a mini-novel on this refactoring, see the linked commit. The second part arrived later, in 15444ca. - 6659eb3 features some good old de-wx'ing of dialogs for opening/saving/choosing files. Also included is some refactoring of status bar-related classes. ----------------------------------------------------------------------- ### Usability (#625, #643, #645, #652, #656) Back on my bullshit, aka trying to make Wrye Bash more usable. For 311, we have: - bfdee3e, which added a Ctrl+S shortcut (and matching link for discoverability) to make frequent saving easier. - 47ef378, which aimed to improve the INI Edits tab's usability by making it more consistent with the other tabs (e.g. adding Alt+Click support, opening tweaks via Enter or double click, etc.). - effcf83, which added links for LO undo/redo to improve their discoverability. - 486a640, which restricted the red background color to timestamp conflicts since it kept confusing people. See the relevant issue (#656) for some more background info on this change. - 941fb25, which was born out of me going "hey, I've never used the Fit Contents and Fit Header options before, let's try that", really liking how much easier it made adjusting columns for Master lists, then noticing that it applied *globally* and so nuked my carefully adjusted columns on the Mods tab. Instead, each setting now only applies to the tab/list you enable it on, allowing you to use it for just the lists where it helps you. - d1290a5, which was already mentioned in "wx (#190)". ListBoxes was just no good for usability. They didn't remember sizes, always started out being sized wrong and don't even wrap text correctly when you do resize them. Replacing them with custom classes allowed us to massively improve the usability of all kinds of popups and dialogs throughout WB. The second part of this refactoring arrived later, in 15444ca. ----------------------------------------------------------------------- ### Image handling (#366, #557) As an offshoot of #190, we have the venerable #366. Image handling is still very much a work in progress, but f7dd4dc brings us a lot closer. Fleshing out the image API and making it ready for SVGs, it also replaces many of our somewhat crusty images with SVGs that can scale to any size we need. It also further decouples us from wx (see the second entry in the "wx (#190)" section up above). ----------------------------------------------------------------------- ### File operations (#241) In b5bb441, @lojack5 changed our file operations backend on Windows to use the newer IFileOperation API instead of the old SHFileOperation API. This was no easy task, as pywin32's support for IFileOperation is broken for some reason and so @lojack5 had to write a wrapper from scratch. ----------------------------------------------------------------------- ### Update checking (#663) Come 312, we want to replace our nine(!) Nexus page with a single one in the Modding Tools section. The LOOT team recently took the same step. The main reason is simply because updating nine pages for every release *sucks*. Even moreso if the release gets detected as a virus. However, the problem with moving to a single page is discoverability. The Nexus team have indicated that they're interested in adding the ability for pages in the Modding Tools section to indicate that they support certain games, so that they'll show up in the search results for searches on those games' pages, but that's not implemented yet. To that end, we implemented an update check. Once 312 is released on the new Nexus page, we can stop updating the other pages and direct users to the new one via the update popup. Of course, the update check can be disabled in the settings and a manual version of the check is also accessible from there. You can also adjust how frequently the update check happens (the default is once every hour). ----------------------------------------------------------------------- ### Linux (#243) Near the end of 311's development, I bought an NVME SSD and took the opportunity to redo my dual boot environment. After backing up and restoring the Windows part of my dual boot, Windows promptly decided to ruin my day by first breaking Firefox and Thunderbird, then breaking the entire Start menu. Since I can't really get work done when I have to use stupid workarounds like "Windows+R > explorer.exe or wt.exe > launch the app I actually want to launch", I said "screw it" and moved entirely to Linux. Born out of that is the last part of 311: - 1d6db12 fixed a smorgasbord of random problems when using WB on Linux. - 1fe6d2d made BAIN wizard and FOMOD images work on Linux. - 1581953 fixed the GTK webview refusing to load when we have a file with '&' in its filename. - 1d07fc3 implemented sending files to the recycle bin on Linux. Most of these apply to our (even more WIP) macOS support as well. ----------------------------------------------------------------------- There were many, many, *many* other changes - again, 500 commits. One person who hasn't been mentioned so far is @sibir-ine, who contributed lots of minor improvements to WB's out-of-the-box usability (e.g. default BAIN directories) and tirelessly helped with monitoring the Discord server and recording the reported bugs on our project board. Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @sibir-ine, @lojack5, @BeermotorWB and many more that GitHub's contribution tracker doesn't list.

@Utumno

I wanted this release to be smaller for once. 500 commits and nearly a year later, mission failed :( Hopefully we'll manage to keep the next one smaller. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- The first major change in 311 was the beginning of the road towards proper Fallout 4 records support and hence a fully function Bashed Patch in FO4. - 00c9ee8 implemented a key piece of the records infrastructure, NVNM. It also dropped some hacks to further pave the way. The first two FO4 record types were implemented. - dc6311a refactored CTDA and implemented the next seven record types. - 641bc73 tackled the single biggest challenge of the entire undertaking: VMAD. As a bonus, if you enjoy rants about stupid design decisions, see c5b5d8a. The next ten record types were also implemented. - 28578a6 moved some record types that are identical between Skyrim and FO4 to brec and implemented the next eighteen records. - This was immediately followed by bb7e5ae, which slightly refactored GameInfo.init() but mostly just implemented the next twelve record types. - 3d8b23e implemented nine more record types. - 9bfb6de implemented two more. Also, @BeermotorWB began the porting process by bringing Tweak Settings to FO4 in 57b2443. There are about sixty FO4 record types left to implement. This will be concluded in 312. I chose this part for the title of the release ("Fallout 4 preparation") since it's what occupied my brain during most of 311's development. ----------------------------------------------------------------------- Since moving to Python 3 in 310 (see #460), we've been taking full advantage of everything py3 offers us. Some example commits: - 8de5f40 cleaned up one of the most annoying files to edit, common_subrecords.py. - cc198c9 shaved hundreds of lines off by replacing verbose nested OrderedDict instantiations with regular, now-ordered dicts. - fd1d4d9 took advantage of ordered dicts to drop duplication between the records list we used to keep track of records in a Mob* class and the id_records dict we used for fast random access by FormID. Now we only need the dict. - In bbb8fa2, @lojack5 used Python 3's type annotations to refactor our flags handling. We also upgraded to Python 3.10 in 7b3573c and to Python 3.11 in 69e7e3a. We also optimized a bunch of central code for e.g. the BP, FName and Path to take advantage of changes like 3.11's adaptive interpreter, which gave pretty good speedups. We've also been doing what we call 'fdusting' - replacing any random pieces of code that look like a + '.esp' with f-strings, i.e. f'{a}.esp'. Another (made-up) example: 'Loaded %d images' % len(images) becomes f'Loaded {len(images)} images'. This has been happening as a sort of "if you're editing in the vicinity, fix any you touch" thing, so spread out over tons of commits. F-strings are one of the best things about py3 :) ----------------------------------------------------------------------- The eternal #480 rears its head again, as it does every release. First off, every single merge mentioned in the FO4 section above also falls under this section. In addition to that, we are approaching setDefault. Uprooting that will give us major speed boosts, simplify code, fix lots of random "the BP is editing this value for some reason" bugs (all of those are due to defaults), etc. One major blocker was FormID handling (see section "FormIDs (#637)" below), now cleaned up. Some other commits related to records refactoring in 311: - 28e0734 reworked defaults handling, making defaults explicit where they are needed and eliminating them from most structures in the records code. - fd1d4d9, already mentioned in the "Python 3 (#618, #619, #644)" section above, reduced duplication in the record_groups code by getting rid of the 'self.records' list in favor of just the (thanks to py3, now ordered) 'self.id_records' dict. - 71520f7 massively reduced code duplication in the various games' init() methods and the __slots__ definitions for all the record types for a total of nearly 1000 LOC removed! - bbb8fa2 by @lojack5, already mentioned in the "Python 3 (#618, #619, #644)" section above, took advantage of Python 3's type hints to refactor our flags handling. The biggest merge of this development cycle also falls in this category. In a truly herculean merge (d2152aa), @Utumno tackled record_groups, turning it from a TODO-riddled mess that didn't really know if it wanted to be Oblivion-only or game-agnostic into a modern API that will serve us well for the future (e.g. for FO4, which makes QUST a new top-level group). Seriously, this refactoring was so complex, its merge has two sub-merges. ----------------------------------------------------------------------- One of the major refactoring goals of 311 was tackling FormIDs. We used to repesent them as either integers (called "short FormIDs") or tuples of strings and integers (called "long FormIDs"). Note that the latter became tuples of FNames and integers in 310 (see section "FName (#543)" in the 310 release commit. This was achieved in 4964710, which is once again an entire branch squashed down to a commit so as to not break dev. They are now stored as a class instead (brec.FormId). This required careful engineering and testing to make sure we don't kill BP performance. For the full story, see the linked commit. ----------------------------------------------------------------------- We're trying to replace all '%s' specifiers with more useful '%(something)s' specifiers. This is still nowhere close to being finished, but 2442285 began the process. Once this is done, we'll want to rethink how we allow people to contribute localizations - see #500 for more info. ----------------------------------------------------------------------- No release is complete without adding support for new games, and 311 is no exception. Bethesda released most of their games on GOG and some on the Epic Games Store, so we added support. See the linked issues and the referencing commits for more information. The biggest difficulty was adding support for yet *another* method of game detection (for the Epic Games Store). Still to come is refactoring the game detection on the Steam side, since some of these rereleases will actually overwrite the registry key used by the Steam versions, making it impossible for WB to reliably detect the installation of both versions. This will also pave the way for Proton-based game detection on Linux (see "Linux (#243)" section below). ----------------------------------------------------------------------- 5e688fd prepared us for proper Nexus integration, which will be one of the big feature goals of 312, by merging a nexus.py file that wraps the Nexus API into the source tree so that keeping it up to date is easier. ----------------------------------------------------------------------- Some refactoring work on FileInfos and related APIs occurred as well, mostly on the INIs. See 22c8c5e, 2495972 and b3e4116 for details. ----------------------------------------------------------------------- That's right, more of this. - 6f2bd1d finally de-wx's the mighty UIList (which backs all of WB's tabs) by making it inherit from our PanelWin instead of wx.Panel. Turns out it was that easy (I'm sure it had nothing to do with the metric tons of refactoring that had been done towards this in 310, 309, 308 and 307). - f7dd4dc decouples us from wx with regards to the default, builtin art from wx we used to use. We instead now use SVGs (mostly from Font Awesome - don't worry, I was careful with the licensing), which also gives us some consistency across platforms. - d1290a5 begins the fight against balt.ListBoxes, an absolutely horrible class that not only had a really weird API but also featured terrible usability. I wrote a mini-novel on this refactoring, see the linked commit. The second part arrived later, in 15444ca. - 6659eb3 features some good old de-wx'ing of dialogs for opening/saving/choosing files. Also included is some refactoring of status bar-related classes. ----------------------------------------------------------------------- Back on my bullshit, aka trying to make Wrye Bash more usable. For 311, we have: - bfdee3e, which added a Ctrl+S shortcut (and matching link for discoverability) to make frequent saving easier. - 47ef378, which aimed to improve the INI Edits tab's usability by making it more consistent with the other tabs (e.g. adding Alt+Click support, opening tweaks via Enter or double click, etc.). - effcf83, which added links for LO undo/redo to improve their discoverability. - 486a640, which restricted the red background color to timestamp conflicts since it kept confusing people. See the relevant issue (#656) for some more background info on this change. - 941fb25, which was born out of me going "hey, I've never used the Fit Contents and Fit Header options before, let's try that", really liking how much easier it made adjusting columns for Master lists, then noticing that it applied *globally* and so nuked my carefully adjusted columns on the Mods tab. Instead, each setting now only applies to the tab/list you enable it on, allowing you to use it for just the lists where it helps you. - d1290a5, which was already mentioned in "wx (#190)". ListBoxes was just no good for usability. They didn't remember sizes, always started out being sized wrong and don't even wrap text correctly when you do resize them. Replacing them with custom classes allowed us to massively improve the usability of all kinds of popups and dialogs throughout WB. The second part of this refactoring arrived later, in 15444ca. ----------------------------------------------------------------------- As an offshoot of #190, we have the venerable #366. Image handling is still very much a work in progress, but f7dd4dc brings us a lot closer. Fleshing out the image API and making it ready for SVGs, it also replaces many of our somewhat crusty images with SVGs that can scale to any size we need. It also further decouples us from wx (see the second entry in the "wx (#190)" section up above). ----------------------------------------------------------------------- In b5bb441, @lojack5 changed our file operations backend on Windows to use the newer IFileOperation API instead of the old SHFileOperation API. This was no easy task, as pywin32's support for IFileOperation is broken for some reason and so @lojack5 had to write a wrapper from scratch. ----------------------------------------------------------------------- Come 312, we want to replace our nine(!) Nexus page with a single one in the Modding Tools section. The LOOT team recently took the same step. The main reason is simply because updating nine pages for every release *sucks*. Even moreso if the release gets detected as a virus. However, the problem with moving to a single page is discoverability. The Nexus team have indicated that they're interested in adding the ability for pages in the Modding Tools section to indicate that they support certain games, so that they'll show up in the search results for searches on those games' pages, but that's not implemented yet. To that end, we implemented an update check. Once 312 is released on the new Nexus page, we can stop updating the other pages and direct users to the new one via the update popup. Of course, the update check can be disabled in the settings and a manual version of the check is also accessible from there. You can also adjust how frequently the update check happens (the default is once every hour). ----------------------------------------------------------------------- Near the end of 311's development, I bought an NVME SSD and took the opportunity to redo my dual boot environment. After backing up and restoring the Windows part of my dual boot, Windows promptly decided to ruin my day by first breaking Firefox and Thunderbird, then breaking the entire Start menu. Since I can't really get work done when I have to use stupid workarounds like "Windows+R > explorer.exe or wt.exe > launch the app I actually want to launch", I said "screw it" and moved entirely to Linux. Born out of that is the last part of 311: - 1d6db12 fixed a smorgasbord of random problems when using WB on Linux. - 1fe6d2d made BAIN wizard and FOMOD images work on Linux. - 1581953 fixed the GTK webview refusing to load when we have a file with '&' in its filename. - 1d07fc3 implemented sending files to the recycle bin on Linux. Most of these apply to our (even more WIP) macOS support as well. ----------------------------------------------------------------------- There were many, many, *many* other changes - again, 500 commits. One person who hasn't been mentioned so far is @sibir-ine, who contributed lots of minor improvements to WB's out-of-the-box usability (e.g. default BAIN directories) and tirelessly helped with monitoring the Discord server and recording the reported bugs on our project board. Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @sibir-ine, @lojack5, @BeermotorWB and many more that GitHub's contribution tracker doesn't list.

Let's go nuclear on this. I *think* the problem might actually be some kind of overflow in native Windows/wxWidgets code with a large enough Data folder and enough large files in it that need to be updated. This may actually speed up a Full Refresh. For a large file (e.g. Fallout4 - Textures1.ba2 at 2.5G) we would issue thousands of progress calls (1290 for that BA2, to be exact), which definitely isn't ideal in the middle of a CRC calculation.

I also renamed the params to bain methods - we could use ask_confirm for all those but still maybe best to differentiate between env and bosh uses? Under #190

StatusBar_Button(Link), by having as a parent the DnDStatusBar, binds Link to balt (in a typing comment) and it's the next step in de-wx'ing - StatusBar_Button is-not a Link for one. But this needs some more clean up started in ee0b8c7 and 45f1261 (see comments in 6659eb3 merge). Move java to class scope, initialized once - appArgs was used once - this removes the init override. Erode SetBitmapButton: - Turns out image was never passed but internally, replace with private method (wip, it's images after all) - onRClick was to override DoPopupMenu - getting there Simplify UpdateIconSizes: One less call of _addButton get_icon: Unused since 7615cf38ebd3b83ed7633316aa9ed210dab2e27em (yey!) Under #190, #163

This does it finally (at the expense of dropping the progress from the refresh call - note this would pop up on tab in for instance). Reduced duplication around rescanMergeable. Drop balt from converters: refreshConverters was binding us - single use in irefresh. I pass a progress in there. A very few progresses may disappear - oh well, plus they would just flash by as converters are usually few to none, plus once this is frozen we can revisit progress handling. Edit:the only call sites of irefresh this could make a difference IIUC: - refresh_n/refresh_ns are - but then should we run refreshConverters at all? This is the FIXME in irefresh itself, we should revisit the `if changed or ...` and be more specific. - the calls of irefresh(what='I') in bain - this should be handled someplace central and eventually be passed a Progress from the UI. Drop balt from mods_metadata: This change is a bit more tricky as checkMods does a few things before the progress was created and so a CancelError could be thrown outside the try catch. Leaving separate to review And with this and that ___bosh is decoupled from gui___ Under #190.

This merge finalizes 6659eb3 by finally decoupling bosh from balt/gui. This was a major goal of the #190 refactoring - was achieved at the expense of some callback parameters (keep them at a min and ideally move them to basher somehow) and a few changes in progress, that we will revisit at some point. This also lead to: - aligning the default for ask_confirm between windows and shutil implementations - this will result in some file operation dialogs previously shown in windows to not be shown anymore. We can always fine tune. - saving sizes for real (most of them were saved for the session - the ones that were not will reset their sizes for the users that had saved them, but just once, no backwards compat needed) The branch finishes with some more work around the status bar and app_buttons, see the merge cited above for the whys. Under #600 - no headless mode (aka Bash as a library) possible while the data model was coupled to wx.

Utumno · 2023-10-07T11:22:22Z

I am working on this and will hopefully close it for 312, as it's been around for a while :P and keeps blocking some nice refactoring - some of it was ideas that I had for a while - encapsulate the JIT behavior of some of the balt/gui classes is the Lazy class - wrap Objects above Window another one. Now this makes controlling access to the _native_widget more imperative, think also of Bash ctd'ing very very hard when for instance wx.App is not initialized (or...?). Think also of switching gui backend, even if we won't ever I hope, a gui wrapper would profit from being designed with this in mind. I was thinking of something like:

+def _native(*, native_get: str = None, native_set: str = None):
+    """Decorator to delegate method calls to the native widget. Used to
+    control native widgets access (and potentially swap gui backend)."""
+    if not (bool(native_set) ^ bool(native_get)): # xor them!
+        raise ValueError(f'Pass either {native_get=} or {native_set=}')
+    def call_native(meth):
+        if native_get:
+            @functools.wraps(meth)
+            def _call_native(self, *args, **kwargs):
+                return getattr(self._native_widget, native_get)(*args, **kwargs)
+        if native_set:
+            @functools.wraps(meth)
+            def _call_native(self, *args, **kwargs):
+                getattr(self._native_widget, native_set)(*args, **kwargs)
+        return _call_native
+    return call_native
+
 class _AComponent:
@@ -207,2 +224,3 @@ def set_component_name(self, new_ctrl_name: str):
     @property
+    @_native(native_get='IsShown')
     def visible(self) -> bool:
@@ -212,5 +230,5 @@ def visible(self) -> bool:
         :return: True if this component is currently visible."""
-        return self._native_widget.IsShown()
 
     @visible.setter
+    @_native(native_set='Show')
     def visible(self, is_visible: bool):
@@ -219,3 +237,2 @@ def visible(self, is_visible: bool):
         :param is_visible: Whether or not to show this component."""
-        self._native_widget.Show(is_visible)

Has the advantage of pruning and singling out most boilerplate uses of _native_widget, can be used to control access in a central place and can be conditionally passed different functions for different backends (I am not familiar with any so this may not apply but I guess we should have a _native_widgtet too and access similar methods - if not pass different functions). One default is harder typing - in particular we cant control click on the gui method and jump to the stub but this is still a couple clicks away (plus we should not be revisiting much the code in gui once frozen).

lojack5 · 2023-10-07T22:02:08Z

I've been slowly working on a similar (more for a generic solution) attack on this same idea. I like the approach. My pet project is taking a lot of work because I'm really trying to get the IDE/static type-checkers to still be able to do some helpful work for me. I want the typechecker to be able to point out when the method I'm forwarding to actually does have the same signature of the one I'm wrapping (just passing in the name of the method basically removes all ability of IDEs/etc to help you out there). It's a hard problem getting that working properly, especially with all the different types of functions that could be accessed on an instance (bound method, unbound method, class method, static method, attribute that's a function but not an instance method, properties). Anyway, it's a fun little problem to work on fleshing out all those details, who know's if I'll ever get it to a good working state though.

That aside, this is good enough for it's purposes :)

Utumno · 2023-10-09T08:20:02Z

An interesting typing exercise (passed when minimal to no code is added :P) would be to use typing for the events handling in our gui library - useful judging from the couple wrong params I passed to the on_drag callbacks recently.

lojack5 · 2023-10-09T17:11:44Z

Not too hard to do, but would require a least one type definition (per event handler signature) when first defining the event handler. It can be done in a way to even work with lambdas properly (which are notoriously hard to get to work with type information). I'll mock something up, but they do end up using TypeVar and TypeVarTuple, which means we should hold off till 3.12 to make it much cleaner PEP 695 is a godsend for cleaning up most of the boilerplate cruft involved.

lojack5 · 2023-10-09T22:51:31Z

Ok, lojack-typed-eventhandler is an example of what can be done (requires python 3.12, hence the CI failure). There's one thing blocking this really being useful, and one edge case that slightly hinders usability:

Blocker - wxPython type stubs don't have return type information. Since most of our argument processors are calling wx methods on event objects, the typechecker/IDE can't properly infer the types for the event handler.
Minor niggle - lambda functions defined inline (inside the call to _AComponent._evt_handler) work well, because they can get type information from _evt_handler itself. Lambda functions defined elsewhere have almost no useable type information, because the event parameter has no type information in that context.

I've already wanted to revisit the wxPython typestubs (they're autogenerated with a custom python script), this might make me go back and actually work on it.

Here's an example of what VSCode can infer about the return types - it can't figure out what GetLabel is returning, due to those type-stubs not having the information.

Utumno · 2023-10-11T08:04:02Z

Thanks Lo! Seems like typing is going beta :P Will have to have a closer look (need to set up 3.12 and I fell into a rabbit hole of regexes - hopefully done soon)

lojack5 · 2023-10-11T20:38:50Z

If you just want to play around on 3.12, everything seems to work fine out of the box, we just can't package (pygit2 not available) and PDFs aren't shown (pymupdf not available).

Some last-minute de-wx'ing to *actually* close off this extremely versatile branch. _widget_to_panel[self.GetPage(self.GetSelection()).GetId()] == \ tabInfo[self.GetPage(self.GetSelection()).Label][2] # True but then there are translations, duh. So we still need an inverse dict. The main benefit is of course getting rid of the wx_id_, *one* use remains! The other benefit is not having two data structures to update. However, the Notebook API (still) escapes me so [!] currentPage property: It's as often as ShowPanel/RefreshData from a quick look - encapsulate nasty logic there - could get a setter too . GetPageText idea from https://stackoverflow.com/a/9023919/281545 Under #190

See: https://discord.com/channels/537706885965676554/537710082755133460/1158908657472372786 Introduced in 5459ca2 in windows but not in linux - probably cause wx.GenericProgressDialog is implemented differently? I have been bitten before, see 223f5e5 - revisit when we get to Progress. Nit in env from future (from_path is already used as a factory and some fdust/typing - initStatusBar is a hell) Under #190

Careful reader will recognise #174, #366 and #570 radiation and also fixups from recent ea96e99, under #163. Exact measurements show we just passed the #190 event horizon.

I was thinking of this for a while now - there are a few other places where we could use Lazy: - gui patcher panels (an opportunity to investigate why they are created once) - Link - yep wrapping wx.Menu() is in the last 10% of de-wx'ing and might make sense to cache those (and clearly differentiate between stateless and stateful ones) - Imagewrapper ;) Currently it's over-fitted to its single use -> StatusBar_Button, but there is fit quite nicely (note a few methods/class variables gone and the rest I believe reads more self explanatory than before). For now I disallow creating the widget through a _native_widget property access but that might not fit the other usecases (or indeed all usecases that allow_create is True - note that _append in links is an allow_create check) Make _is_created private and instead override the tooltip to not blow if the _native_widget was not created. See dicussion in RRR _present -> allow_create and remove useless SetState: Refactor OBSE handling - remove obseButtons and have the instances check their _obseTip and the state of the OBSE button to set the correct tooltip on creation/flip state of obseButton. This will simply reset the tooltips of all *created* items, probably just resetting some tooltips to the same str, but the simplification is more than worth it. Execute -> sb_click Use the new ring to kill a dragon :P Use the link.uid in Hide/Unhide that gets us rid of some more dragons Under #190, #163 #174 We still must make sure the _tip is not empty for the SB settings dialog or rather that the sb_button_tip returns an unique value

Part one of using lazy wrappers for the rest of the wx classes that were not wx.Window instances. Turns out the sought after images API is a Lazy wx.Object that puts Lazy's _native widget property to the test. By wrapping images we were able to address the API part of #570, which evolved into refactoring large parts of initialisation (#600). Under #190. Signed-off-by: MrD <[email protected]>

@Utumno

On the commit side, we managed to keep this one smaller (~250 commits rather than the ~500 commits that made up 311). On the complexity side, we probably failed, introducing a couple huge refactorings. A lot of Fallout 4 refactoring and patcher porting, Starfield support, native Linux support and refactoring WB's refresh handling are the biggest contributors to line count and commit number. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### FileInfo(s) (#336) #336 came back with a vengeance this time around. It started with fbb1925, which reworked the AFile API to cover installers and netted us some very nice performance gains and simplifications of refresh APIs - we'll some more work on refreshes later on. ----------------------------------------------------------------------- ### Native Linux Support (#243) Since Windows crapped out on me recently, I decided to ditch it and use Linux full-time. As a result, native Linux support for WB suddenly became a much more pressing issue. 312 improves it to the point where we now mark it as supported, with only some QOL issues remaining (and launchers, but those are a complete WIP anyways, see the later Launchers section in this commit message). There were a ton of random commits that made up this improved support. Basically, whenever I noticed something broken or in need of improvement, I would end up committing the fix, which means the commits are scattered like crazy. Nevertheless, here are some highlights: - e86e939 reworked temporary files handling, but this deserves its own section, which it will get below. - 1993f9b reworked parent handling in wx menus. We were always using Link.Frame.show_popup_menu, but this broke the Bashed Patch menus on Linux entirely. This turned into a mini-refactoring under #190 to track down and do this properly. - b762cc6 made the date and time dialog work on Linux (at the cost of no longer using locale-specific date/time formatting for it). - 20dd955 rewrote Steam game detection entirely to avoid the Windows registry. This was prompted by Starfield not creating a registry key at all, but I was planning to do this anyways, because Linux obviously does not have a registry and because it means users no longer need to launch the game once to generate a registry key. - 61d4d87 is what prompted me to actually mark Linux as supported. This merge added: - Proton detection, meaning the out of the box experience on Linux is now comparable to the one on Windows (in terms of it simply detecting your installed games with no manual intervention needed). - A warning when the Data folder is mounted case-sensitively. - Various fixes for race conditions and more, exposed by Linux's filesystems (perhaps specifically by btrfs?). - Functional Ctrl+Tab handling for cycling through WB's tab and some other misc improvements. - Also worth mentioning here is the File Operations (#241) section, see below for more on that. Linux was then finally marked as supported in c855882. Shortly before 312 was released, @BGazotti also contributed a whole host of small fixes and improvements for WB's Linux support in 00381da. Many thanks! ----------------------------------------------------------------------- ### Temporary Files (#665) This was born out of three needs: - On Linux, the global temporary directory (/tmp) is generally mounted in RAM. This means one can easily run out of space here when e.g. extracting a giant couple BSAs. And even worse, if the system has no swap configured, it can completely lock up when this happens. Wrye Bash should, in fact, *not* lock up the entire system. - We can get a decent speed boost by making sure the temporary directory we're using sits on the same device/filesystem as the Data folder. That way, the OS only has to rename some file paths rather than copying data around. - And lastly, our temporary file APIs were all over the place. There were three distinct types of temp file handling, and various pieces of code seemingly randomly chose which one to use: - bass.getTempDir/newTempDir/rmTempDir - Path.temp and Path.untemp - Just use Path.baseTempDir/tempDir or even tempfile directly and do it completely manually See the commit that introduced this refactoring (e86e939) for a full breakdown of the problems these APIs had. So we designed a new API that can cover all use cases and makes it hard to get wrong. Because, as it turns out, correct temporary file handling is *hard*. And another huge advantage of this design is that it will work with multiple instances of WB running in parrallel, which is an eventual goal. See the aforementioned e86e939 for the full lowdown. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525) The refactoring towards full FO4 Bashed Patch support is still ongoing. The goal was to get it done for 312, but then Starfield released and took the title of this version, plus we didn't want to drag out the release of 312 even further. Still, 312 brings the BP for FO4 very far. Work began in 2327ef4, which cleaned up header flags and implemented the next ten record types. Next up, 548bce5 tackled two difficult record types (NPC_ and OMOD, see the referenced commit for details on their problems) and implemented the next seven record types. With so many record types done, it was time to test them properly. To that end, d941cae ported the first batch of patchers over. In total, we now have 20/33 patchers available for FO4, though not all of them support all the record types they could support yet, simply because those record types aren't implemented yet. 28fb000 continued the records refactoring, though with the added complication that now, each time we implement a record type that an already-ported patcher can target, we also add support for that record type to the patcher in question. In that vein, this merge implements the next thirteen record types and adds them to patchers where appropriate. ----------------------------------------------------------------------- ### Starfield (#667) The star of the show (no pun intended). Note that this is early support, meaning that we don't actually support reading or writing plugins for Starfield yet. The main reason for that is Starfield's... *poor* design, when it comes to the plugin format. You can see the merge commit for more details (ec30f02), but basically, Bethesda clearly did not take moddability into account when designing the plugin format for Starfield. Unless they *drastically* rework the engine before releasing the Creation Kit, the Bashed Patch might not happen, period. ----------------------------------------------------------------------- ### wx (#190) It never ends (okay, maybe soon, but no guarantees). ea96e99 at least took some good steps towards closing off #190 for good by decoupling bosh (the data model) from balt/gui (the GUI backend). An important step towards #600 (headless mode) as well. Some more work landed in 170ad99, where @Utumno introduced gui.Lazy, a wrapper class for components that need to lazily create their native widgets. This led to heavy refactoring of the status bar and its buttons, taking down lots of dodgy code in the process. Also contained in that merge is a sub-merge, bd0a897, which utilized the new lazy classes to really tackle WB's image handling. Especially useful in preparation for the high DPI improvements that will be elaborated on later in this commit message and the launchers (see Launchers section below). ----------------------------------------------------------------------- ### Overlay Plugins (#668) The actual reason to be excited for Starfield's potential is shown in 9d21b40. Overlay plugins are a new type of plugin that does not take up a plugin slot at all, meaning you can have an unlimited number of them(!), with the downsides that such plugins can't contain new records (since they don't have a plugin slot to place the new records into) and must have at least one master. Unfortunately, due to the aforemnentioned massive problems with Starfield's plugin format, overlay plugins and ESLs aren't usable right now. Let's hold out hope that Bethesda can fix this, if only so the ~1 day of engineering I put into supporting them won't go to waste :P But jokes aside, the refactoring in this merge is very much worthwhile on its own. It makes supporting multiple mergeability checks for one game possible (since Starfield supports both ESLs and Overlay plugins - well, in theory it does) and thus opens the door for #596. ----------------------------------------------------------------------- ### Refreshes (#265, #353) We already mentioned fbb1925, which tackled refreshes from the perspective of BAIN and AFile. But midway through 312's development, I wanted to try my hand at implementing a new tab, namely the xSE Plugins tab (#456). That made me run headfirst into a wall, namely the fact that BAIN only knows about plugins and INI tweaks when it comes to trackin and updating other data stores. BSAs were kind of in there too, but not fully. My new tab needed BAIN to keep proper track of it as well, which meant refactoring. That bloomed into a very nice merge in c6ec399, which took down a bunch of old and rusty APIs (e.g. RefreshUIMods, saveListRefresh, _is_ini_tweak, etc.). The refactoring eventually culminated in us centralizing cross-tab communication via bass.Store and taking down a few hundred lines of really annoying, duplicate code. Some followup refactorings grew out of this as well. 70fe061 took down lots of complicated code related to ghost handling, paving the way for much more refresh handling work to come in 313+. Similarly, b8d9e0a refactored ModInfos.refresh, slowly unraveling the complicated stack of refreshes we've got going on. 387c9df tackled refreshes from the perspective of load order handling, attempting to simplify the latter significantly. One big goal here was to preserve as much information about the changes in load order throughout any given LO operation as possible, since that is crucial in order to properly refresh only the changed files - think elegance *and* performance. ----------------------------------------------------------------------- ### File Operations (#241) d897347 reworked our file operations backend significantly, prompted by two things: - Linux support, since we can't delegate to ifileoperation like we do on Windows (duh). - The last couple bits of FOMOD support (fingers crossed). Specifically, it turned out that StarUI, a Starfield mod, would fail to install correctly in WB when run via FOMOD. The reason turned out to be that it wanted to install a single file to two destinations. You can see the linked commit for all the details. BAIN did not have much trouble handling this, since it generally stores everything as destination -> source dicts, but our file operations backends did not support this at all. As part of this commit, we also introduced support for reflinks/file cloning to Wrye Bash. This currently has to be done via a less-than-ideal third-party depedency. An upstream request to the CPython project has stalled. As a result, we only support reflinks/file cloning on Linux and macOS filesystems right now (not a major problem, since ReFS has shown no signs of making it to mainstream Windows deployment yet). But what are reflinks? They behave like regular file copies, except that the backing data is shared between both copies. This is generally enabled by copy-on-write (COW) filesystems, since this feature is pretty much implemented for free on such filesystems. This lets all of WB's copy operations become much faster on Btrfs, XFS, ZFS, APFS, etc. ----------------------------------------------------------------------- ### Auto-Splitting the BP (#533) A long-standing feature request (open for more than three years at this point) has been addressed in 312: automatically splitting the Bashed Patch into multiple files once it ends up with too many masters for a single file. The advantage is obvious, especially compared to the previous behavior (throwing all work away at the end). 6ef2198 introduced the feature, though it turned out to be simultaneously much less work than I expected *and* much more complicated than I expected. Which is weird. ----------------------------------------------------------------------- ### Scattered Packages (#670) I have had this idea for more than three years, at least since I completed work on #380 back in 2020. The last big problem with FOMOD support in Wrye Bash was that a whole bunch of weirdly packaged FOMODs could not be installed in BAIN without a silly workaround (creating an empty plugin and putting it somewhere in the package to trick BAIN into thinking it's a simple package, then using its FOMOD support to remap files and thereby skip the empty plugin, which won't be referenced by the FOMOD's ModuleConfig). The plan was to model the properly, then see what happens. That meant creating an entirely new type of BAIN package. Previously, we had three types: - Simple packages, which contain some files to be placed in the Data folder. - Complex packages, which contain multiple sub-packages. Each sub-package behaves like a simple package. - Invalid packages, whose layout BAIN can't make sense of. Granted, from an end-user perspective, there are more types of 'packages': - Simple/complex packages, which behave like simple packages but have some folders in between the package root and the actual files that need to be placed in the Data folder. In WB's code, these are treated as simple packages. - Markers, which don't exist in the filesystem and are only an aid for users to better organize their packages. In WB's code, these aren't technically packages at all. 759055c introduces a new type of package: - Scattered packages, which have an unrecognized layout, but also come with instructions that tell BAIN how to make sense of the *scattered* mess of files that it otherwise can't parse - hence the name. Combined with some more refactoring to make BAIN recognize scattered packages by the presence of an 'fomod' folder and the aforementioned multi-destination file installation support, this finally tackles the last remaining FOMOD issues (again, fingers crossed). ----------------------------------------------------------------------- ### High DPI (#511) One big goal for 313 is #511, i.e. the ability to customize checkbox colors. But this is much more than just that - it's also about simplifying the resources for checkboxes by using a single SVG instead of a ton of different PNGs, making it possible to have more overlays over the icons (the current implementation of the wizard overlay is just terrible, it's literally a matter of copy-pasting all the checkbox icons, adding a wand over them and then loading all of them manually. This won't scale if we want to, e.g., add an FOMOD overlay). And, of course, this is also about using an SVG to make these scale properly on screens with >100% scaling. To that end, ff276b1 addresses high DPI handling of images. Yes, we already had support for high DPI images via SVGs since #557 in 313, but it turns out that implementation had lots of holes - most notably at 200% scaling, where wxPython decided to scale all our icons *again*, making them absolutely massive and really ugly. With the aforementioned commit, nothing should stop us anymore from tackling #511 in 313. ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @BGazotti and many more that GitHub's contribution tracker doesn't list.

@Utumno

On the commit side, we managed to keep this one smaller (~250 commits rather than the ~500 commits that made up 311). On the complexity side, we probably failed, introducing a couple huge refactorings. A lot of Fallout 4 refactoring and patcher porting, Starfield support, native Linux support and refactoring WB's refresh handling are the biggest contributors to line count and commit number. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### FileInfo(s) (#336) #336 came back with a vengeance this time around. It started with fbb1925, which reworked the AFile API to cover installers and netted us some very nice performance gains and simplifications of refresh APIs - we'll some more work on refreshes later on. ----------------------------------------------------------------------- ### Native Linux Support (#243) Since Windows crapped out on me recently, I decided to ditch it and use Linux full-time. As a result, native Linux support for WB suddenly became a much more pressing issue. 312 improves it to the point where we now mark it as supported, with only some QOL issues remaining (and launchers, but those are a complete WIP anyways, see the later Launchers section in this commit message). There were a ton of random commits that made up this improved support. Basically, whenever I noticed something broken or in need of improvement, I would end up committing the fix, which means the commits are scattered like crazy. Nevertheless, here are some highlights: - e86e939 reworked temporary files handling, but this deserves its own section, which it will get below. - 1993f9b reworked parent handling in wx menus. We were always using Link.Frame.show_popup_menu, but this broke the Bashed Patch menus on Linux entirely. This turned into a mini-refactoring under #190 to track down and do this properly. - b762cc6 made the date and time dialog work on Linux (at the cost of no longer using locale-specific date/time formatting for it). - 20dd955 rewrote Steam game detection entirely to avoid the Windows registry. This was prompted by Starfield not creating a registry key at all, but I was planning to do this anyways, because Linux obviously does not have a registry and because it means users no longer need to launch the game once to generate a registry key. - 61d4d87 is what prompted me to actually mark Linux as supported. This merge added: - Proton detection, meaning the out of the box experience on Linux is now comparable to the one on Windows (in terms of it simply detecting your installed games with no manual intervention needed). - A warning when the Data folder is mounted case-sensitively. - Various fixes for race conditions and more, exposed by Linux's filesystems (perhaps specifically by btrfs?). - Functional Ctrl+Tab handling for cycling through WB's tab and some other misc improvements. - Also worth mentioning here is the File Operations (#241) section, see below for more on that. Linux was then finally marked as supported in c855882. Shortly before 312 was released, @BGazotti also contributed a whole host of small fixes and improvements for WB's Linux support in 00381da. Many thanks! ----------------------------------------------------------------------- ### Temporary Files (#665) This was born out of three needs: - On Linux, the global temporary directory (/tmp) is generally mounted in RAM. This means one can easily run out of space here when e.g. extracting a giant couple BSAs. And even worse, if the system has no swap configured, it can completely lock up when this happens. Wrye Bash should, in fact, *not* lock up the entire system. - We can get a decent speed boost by making sure the temporary directory we're using sits on the same device/filesystem as the Data folder. That way, the OS only has to rename some file paths rather than copying data around. - And lastly, our temporary file APIs were all over the place. There were three distinct types of temp file handling, and various pieces of code seemingly randomly chose which one to use: - bass.getTempDir/newTempDir/rmTempDir - Path.temp and Path.untemp - Just use Path.baseTempDir/tempDir or even tempfile directly and do it completely manually See the commit that introduced this refactoring (e86e939) for a full breakdown of the problems these APIs had. So we designed a new API that can cover all use cases and makes it hard to get wrong. Because, as it turns out, correct temporary file handling is *hard*. And another huge advantage of this design is that it will work with multiple instances of WB running in parrallel, which is an eventual goal. See the aforementioned e86e939 for the full lowdown. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525) The refactoring towards full FO4 Bashed Patch support is still ongoing. The goal was to get it done for 312, but then Starfield released and took the title of this version, plus we didn't want to drag out the release of 312 even further. Still, 312 brings the BP for FO4 very far. Work began in 2327ef4, which cleaned up header flags and implemented the next ten record types. Next up, 548bce5 tackled two difficult record types (NPC_ and OMOD, see the referenced commit for details on their problems) and implemented the next seven record types. With so many record types done, it was time to test them properly. To that end, d941cae ported the first batch of patchers over. In total, we now have 20/33 patchers available for FO4, though not all of them support all the record types they could support yet, simply because those record types aren't implemented yet. 28fb000 continued the records refactoring, though with the added complication that now, each time we implement a record type that an already-ported patcher can target, we also add support for that record type to the patcher in question. In that vein, this merge implements the next thirteen record types and adds them to patchers where appropriate. ----------------------------------------------------------------------- ### Starfield (#667) The star of the show (no pun intended). Note that this is early support, meaning that we don't actually support reading or writing plugins for Starfield yet. The main reason for that is Starfield's... *poor* design, when it comes to the plugin format. You can see the merge commit for more details (ec30f02), but basically, Bethesda clearly did not take moddability into account when designing the plugin format for Starfield. Unless they *drastically* rework the engine before releasing the Creation Kit, the Bashed Patch might not happen, period. ----------------------------------------------------------------------- ### wx (#190) It never ends (okay, maybe soon, but no guarantees). ea96e99 at least took some good steps towards closing off #190 for good by decoupling bosh (the data model) from balt/gui (the GUI backend). An important step towards #600 (headless mode) as well. Some more work landed in 170ad99, where @Utumno introduced gui.Lazy, a wrapper class for components that need to lazily create their native widgets. This led to heavy refactoring of the status bar and its buttons, taking down lots of dodgy code in the process. Also contained in that merge is a sub-merge, bd0a897, which utilized the new lazy classes to really tackle WB's image handling. Especially useful in preparation for the high DPI improvements that will be elaborated on later in this commit message and the launchers (see Launchers section below). ----------------------------------------------------------------------- ### Overlay Plugins (#668) The actual reason to be excited for Starfield's potential is shown in 9d21b40. Overlay plugins are a new type of plugin that does not take up a plugin slot at all, meaning you can have an unlimited number of them(!), with the downsides that such plugins can't contain new records (since they don't have a plugin slot to place the new records into) and must have at least one master. Unfortunately, due to the aforemnentioned massive problems with Starfield's plugin format, overlay plugins and ESLs aren't usable right now. Let's hold out hope that Bethesda can fix this, if only so the ~1 day of engineering I put into supporting them won't go to waste :P But jokes aside, the refactoring in this merge is very much worthwhile on its own. It makes supporting multiple mergeability checks for one game possible (since Starfield supports both ESLs and Overlay plugins - well, in theory it does) and thus opens the door for #596. ----------------------------------------------------------------------- ### Refreshes (#265, #353) We already mentioned fbb1925, which tackled refreshes from the perspective of BAIN and AFile. But midway through 312's development, I wanted to try my hand at implementing a new tab, namely the xSE Plugins tab (#456). That made me run headfirst into a wall, namely the fact that BAIN only knows about plugins and INI tweaks when it comes to trackin and updating other data stores. BSAs were kind of in there too, but not fully. My new tab needed BAIN to keep proper track of it as well, which meant refactoring. That bloomed into a very nice merge in c6ec399, which took down a bunch of old and rusty APIs (e.g. RefreshUIMods, saveListRefresh, _is_ini_tweak, etc.). The refactoring eventually culminated in us centralizing cross-tab communication via bass.Store and taking down a few hundred lines of really annoying, duplicate code. Some followup refactorings grew out of this as well. 70fe061 took down lots of complicated code related to ghost handling, paving the way for much more refresh handling work to come in 313+. Similarly, b8d9e0a refactored ModInfos.refresh, slowly unraveling the complicated stack of refreshes we've got going on. 387c9df tackled refreshes from the perspective of load order handling, attempting to simplify the latter significantly. One big goal here was to preserve as much information about the changes in load order throughout any given LO operation as possible, since that is crucial in order to properly refresh only the changed files - think elegance *and* performance. ----------------------------------------------------------------------- ### File Operations (#241) d897347 reworked our file operations backend significantly, prompted by two things: - Linux support, since we can't delegate to ifileoperation like we do on Windows (duh). - The last couple bits of FOMOD support (fingers crossed). Specifically, it turned out that StarUI, a Starfield mod, would fail to install correctly in WB when run via FOMOD. The reason turned out to be that it wanted to install a single file to two destinations. You can see the linked commit for all the details. BAIN did not have much trouble handling this, since it generally stores everything as destination -> source dicts, but our file operations backends did not support this at all. As part of this commit, we also introduced support for reflinks/file cloning to Wrye Bash. This currently has to be done via a less-than-ideal third-party depedency. An upstream request to the CPython project has stalled. As a result, we only support reflinks/file cloning on Linux and macOS filesystems right now (not a major problem, since ReFS has shown no signs of making it to mainstream Windows deployment yet). But what are reflinks? They behave like regular file copies, except that the backing data is shared between both copies. This is generally enabled by copy-on-write (COW) filesystems, since this feature is pretty much implemented for free on such filesystems. This lets all of WB's copy operations become much faster on Btrfs, XFS, ZFS, APFS, etc. ----------------------------------------------------------------------- ### Auto-Splitting the BP (#533) A long-standing feature request (open for more than three years at this point) has been addressed in 312: automatically splitting the Bashed Patch into multiple files once it ends up with too many masters for a single file. The advantage is obvious, especially compared to the previous behavior (throwing all work away at the end). 6ef2198 introduced the feature, though it turned out to be simultaneously much less work than I expected *and* much more complicated than I expected. Which is weird. ----------------------------------------------------------------------- ### Scattered Packages (#670) I have had this idea for more than three years, at least since I completed work on #380 back in 2020. The last big problem with FOMOD support in Wrye Bash was that a whole bunch of weirdly packaged FOMODs could not be installed in BAIN without a silly workaround (creating an empty plugin and putting it somewhere in the package to trick BAIN into thinking it's a simple package, then using its FOMOD support to remap files and thereby skip the empty plugin, which won't be referenced by the FOMOD's ModuleConfig). The plan was to model the properly, then see what happens. That meant creating an entirely new type of BAIN package. Previously, we had three types: - Simple packages, which contain some files to be placed in the Data folder. - Complex packages, which contain multiple sub-packages. Each sub-package behaves like a simple package. - Invalid packages, whose layout BAIN can't make sense of. Granted, from an end-user perspective, there are more types of 'packages': - Simple/complex packages, which behave like simple packages but have some folders in between the package root and the actual files that need to be placed in the Data folder. In WB's code, these are treated as simple packages. - Markers, which don't exist in the filesystem and are only an aid for users to better organize their packages. In WB's code, these aren't technically packages at all. 759055c introduces a new type of package: - Scattered packages, which have an unrecognized layout, but also come with instructions that tell BAIN how to make sense of the *scattered* mess of files that it otherwise can't parse - hence the name. Combined with some more refactoring to make BAIN recognize scattered packages by the presence of an 'fomod' folder and the aforementioned multi-destination file installation support, this finally tackles the last remaining FOMOD issues (again, fingers crossed). ----------------------------------------------------------------------- ### High DPI (#511) One big goal for 313 is #511, i.e. the ability to customize checkbox colors. But this is much more than just that - it's also about simplifying the resources for checkboxes by using a single SVG instead of a ton of different PNGs, making it possible to have more overlays over the icons (the current implementation of the wizard overlay is just terrible, it's literally a matter of copy-pasting all the checkbox icons, adding a wand over them and then loading all of them manually. This won't scale if we want to, e.g., add an FOMOD overlay). And, of course, this is also about using an SVG to make these scale properly on screens with >100% scaling. To that end, ff276b1 addresses high DPI handling of images. Yes, we already had support for high DPI images via SVGs since #557 in 313, but it turns out that implementation had lots of holes - most notably at 200% scaling, where wxPython decided to scale all our icons *again*, making them absolutely massive and really ugly. With the aforementioned commit, nothing should stop us anymore from tackling #511 in 313. ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @BGazotti and many more that GitHub's contribution tracker doesn't list.

Utumno added the C-todo Category: TODO, specific item that needs to be accomplished in working towards a goal label Feb 2, 2015

Utumno mentioned this issue Feb 2, 2015

Basher package #163

Closed

29 tasks

Utumno added a commit that referenced this issue Apr 19, 2015

balt.button: class from function - mod_links.py de-wx'd:

3344c55

Under #190. mod_links.py: wx.DEFAULT_DIALOG_STYLE was not needed and resize param added

Utumno added a commit that referenced this issue Apr 19, 2015

SetDefault() calls replaced by 'default' button.init parameter:

9a77bfc

Under #190. Apart from couple uses in bash.py where wx is used directly. default parameter should be used in more (all) dialogs.

Utumno modified the milestone: Code Quality Jun 11, 2015

Utumno added the A-wx Area: wxPython interactions (the gui package, balt.py and the basher package) label Sep 15, 2015

Utumno added a commit that referenced this issue Dec 15, 2015

More unused event parameters pruning:

f8cfc82

Under #190. 347 -> 338 occurences of `event` - all dropped from basher

Utumno self-assigned this Dec 29, 2015

Utumno mentioned this issue Jun 14, 2016

wxPython 3 Upgrade #15

Closed

Utumno mentioned this issue Jul 3, 2016

Refactoring Backburner #300

Closed

Utumno added a commit that referenced this issue Sep 11, 2016

Move StatusBar wx code to balt:

b9a1089

Under #190.

Utumno added a commit that referenced this issue Sep 11, 2016

WryeBashSplashScreen to balt:

08ffa08

Under #190.

Utumno pushed a commit that referenced this issue Sep 11, 2016

kill gizmos; ThinSplitterWindow to SplitterWindow:

597d64c

Phoenix compatibility; ..also, I don't see anything special that is being used anywhere that SplitterWindow doesn't have already. Under #190, #15.

Utumno modified the milestones: Code Quality, 312 May 31, 2023

Utumno referenced this issue Jun 21, 2023

XXX Proof of concept: Nexus button

0473a29

Utumno added a commit that referenced this issue Sep 30, 2023

Decouple bosh from gui:

727415d

I also renamed the params to bain methods - we could use ask_confirm for all those but still maybe best to differentiate between env and bosh uses? Under #190

Utumno added a commit that referenced this issue Oct 21, 2023

Merge branch 'ut-fixups-nit-links' into dev:

5f16a7f

Careful reader will recognise #174, #366 and #570 radiation and also fixups from recent ea96e99, under #163. Exact measurements show we just passed the #190 event horizon.

Infernio modified the milestones: 312, 313 Dec 16, 2023

Utumno modified the milestones: 313, 314 Jul 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

De-wx'ing #190

De-wx'ing #190

Utumno commented Feb 2, 2015 •

edited

Loading

Utumno commented Oct 7, 2023

lojack5 commented Oct 7, 2023

Utumno commented Oct 9, 2023

lojack5 commented Oct 9, 2023 •

edited

Loading

lojack5 commented Oct 9, 2023 •

edited

Loading

Utumno commented Oct 11, 2023

lojack5 commented Oct 11, 2023 •

edited

Loading

De-wx'ing #190

De-wx'ing #190

Comments

Utumno commented Feb 2, 2015 • edited Loading

Utumno commented Oct 7, 2023

lojack5 commented Oct 7, 2023

Utumno commented Oct 9, 2023

lojack5 commented Oct 9, 2023 • edited Loading

lojack5 commented Oct 9, 2023 • edited Loading

Utumno commented Oct 11, 2023

lojack5 commented Oct 11, 2023 • edited Loading

Utumno commented Feb 2, 2015 •

edited

Loading

lojack5 commented Oct 9, 2023 •

edited

Loading

lojack5 commented Oct 9, 2023 •

edited

Loading

lojack5 commented Oct 11, 2023 •

edited

Loading