QUADS 1.1.4

About 1.1.4

QUADS 1.1.4 is a major enhancement and bugfix release for the 1.1 gaúcho series and is highly recommended. This comes with around 7 months of development since 1.1.3 and building on the landmark, next-generation 1.1.0 series.

With almost 100 peer-reviewed commits to the codebase, 1.1.4 is comprised of around 25 x enhancements and 30+ bug fixes.

🦃 1.1.4 arrives just in time for Thanksgiving 🦃

Major Feature Highlights

• Hardware Metadata Model: QUADS now populates useful characteristics about your hosts and keeps them in the MongoDB database.

  • You will need to populate data about your systems to take advantage of this.
  • This is baked into the serveral CLI options as well via the --filter command
    • quads-cli --ls-hosts --filter "interfaces.mac_address==ac:1f:62:2f:19:42"
    • quads-cli --ls-available --filter "model==FC640,interfaces__size==5"

• QUADS Available UI (Tech Preview): Flask-based web UI for querying systems availability provided by new service quads-web

  • Runs on TCP/5001
  • Runs from the quads-web service if using RPM / systemd
  • Relies on the MongoDB metadata model for filtering based on model

• New options for managing cloud extensions

  • The following commands can be used for easier management of extensions
    • quads-cli --extend-cloud cloud02 --weeks 2 --check
    • quads-cli --extend-cloud cloud02 --weeks 2

• New commands for managing top-level cloud attributes

  • The --mod-cloud command is now available to manage individual settings at the cloud assignment level

• Configurable time-lock for defining new environments is present

  • To prevent against human race conditions and the potential for more than one operator to choose a cloud from quads-cli --find-free-cloud we've implemented configurable time locks that prevent future re-definition of a cloud environment for a period set in cloud_reservation_lock

• Management of broken hosts moved to MongoDB

  • We no longer rely on Foreman host parameters to manage broken/faulty systems, this is now done in MongoDB and managed via QUADS CLI
    • quads-cli --ls-broken
    • quads-cli --mark-faulty --host host01
    • quads-cli --mark-repaired --host host01

• validate_env.py and our netcat health checks for validation are now fully using asyncio

• move_and_rebuild_hosts.py will now also unmount virtual media ISO (SuperMicro)

• Network automation is now idempotent and will not run twice if not needed.

• quads-cli --summary --detail gives additional full-readout display of active assignments.

• Additional validation check added quads/tools/netcat.py which checks that a system is reachable via TCP/22, this also adds intelligence to validate_env.py to try and better determine where a set of systems is in the baseline/provision process to enact smarter automation.

Major Bug Fixes

• validate_env.py reboot on marked for build commit

• Format error message for --extend-cloud commit

• simple_table_web.py fixes for heat map visuals commit

• Fixes for summary listings commit

• VLAN fixes for last interface for optional public VLANS commit

• Notification fixes commit

• Fix for old cloud objects not being cleared commit

• Fixes for verify_switchconf.py and not processing the last interfaces commit

• Fixes for SuperMicro one-time boot commit

• Fixes for netcat.py health checks socket prematurely closing commit

• Fixes for --ls-available filtering commit

• Fix for applying PXE persistence flags on SuperMicro commit

• Q-in-Q argument fixes and extensibility commit

• CloudHistory fixes with prep_data commit

• VLAN ID fixes for CloudHistory commit

• Please consult the full changelog of differences between the last release and current upstream codebase for full details of all fixes and enhancements.

New Configuration Options (Read before Migrating from 1.1.3)

The following are changes in /opt/quads/conf/quads.yml you should adjust or add, it's a good idea to diff the updated quads.yml with your existing one to make note of any breaking changes via diff -u /opt/quads/conf/quads.{yml,yml.rpmnew}

• New entries for optional ticket system association with QUADS

  • ticket_url:
  • ticket_queue:

• Optional value for inserting a preferred day-of-week deadline for tenants to submit extension requests.

  • quads_request_deadline_day:

• Set default wipe policy (defaults to True)

  • default_wipe:

• Reservation lock (defaults to 48hrs)

  • cloud_reservation_lock:

• Models Information Added (comma separated list of supported models that match metadata model strings)

  • models: R620,R630,R640,R930,R730XD

• Expiring cloud notification setting:

  • quads_notify_until_extended: True

Removed Configuration Options

The following are key/value pair configuration options that have been removed from QUADS

• gather_dell_configs:
• foreman_check_host_health:
• rt_url: (replaced by ticket_url:)
• report_dir:

Other Notable Changes and Upgrade Notes

Migrating Broken Hosts Management to Mongo

If you used Foreman host parameters to manage your broken systems before you can migrate them to Mongo with this command set

Migrating from 1.1.3 to 1.1.4 Reservation Locks

• If you see an issue with reservation locks not going away you may need a one-time adjustment of timestamp data outlined in the above issue.

Foreman RBAC Changes

We have moved to a more manageable method of juggling role-based access permissions for tenants using Foreman which is based on host ownership instead of a series of filters and roles individually added to each system.

• foreman_heal.py has been refactored to use this new approach so running it is enough but you can find the updated setup documentation here for more info.

QUADS 1.1.3

About 1.1.3

QUADS 1.1.3 is a major bugfix and minor enhancement release for the 1.1 gaúcho series and is highly recommended. This comes around 3 months after 1.1.2 and building on the the landmark, next-generation 1.1.0 series.

1.1.3 is comprised of around 5 x enhancements and 28 x bug fixes, it also adds support to badfish for the Dell fc640 blade hardware and auto-generation of OpenShift inventory metadata to help automate OpenShift deployment on bare-metal.

🐰 1.1.3 arrives just in time for the Easter Bunny 🐰

Feature Highlights

• Add Support for OpenStack Inventory for assisting in automated OpenShift on bare-metal deployments.

Similar to ironic-based OpenStack deployments you can now download the equivalent OCP inventory file to feed into your installer for automated deployments.

• Support for Dell fc640 bladecenters is now present.
• More flexible Redfish / iDRAC behavior via querying and using supported reset types.
• Better readability for notifications.
• Page revisions are limited in the Wordpress wiki container images for better performance.

Major Bug Fixes

• Badfish Fixes for Dell fc640 blade servers.
• Fix for heat map visualizations calculating and displaying data for the next four months into the future.
• Fix for ordering interfaces for verify_switchconf.py
• Fix for sorting order for --ls-interfaces
• VLAN fix for clearing --vlan with --define-cloud
• Fix for Juniper automation dealing with changeset commits
• Fixes for managing public VLANS and verify_switchconf.py for the Juniper platform.
• Fix for premature notifications when cloud assignments are being validated
• Fix for badfish graceful restart functionality on supported systems
• Backport fixes/updates from upstream Badfish into QUADS badfish.py library.
• Fix for no wipe assignments and new hosts being added and not validated as they should.
• Please see changelog below for all changes

New Configuration Options (Read before Migrating from 1.1.x)

• You'll need to define openshift_management: parameter in quads.yml. By default this is set to openstack_management: false.

• You'll need to set a password prefix, this is to tackle some vendors like SuperMicro imposing an IPMI password length policy. New user IPMI and Foreman passwords are now infra_location@ticket

  • Setting to add: 4-character string for infra_location in quads.yml
  • Typically most people would just use the airport code here, or some other short identifier
  • Prior to 1.1.3 if your user IPMI/Foreman password was 12345 (--cloud-ticket number) then it will now be rdu2@12345.

infra_location: rdu2

• If you have a request URL or form for users to submit assignment requests for QUADS-managed environments you can enter that in quads.yml (optional):

quads_request_url:

• If you were previously using rt_url (and the RT ticketing system like us) this has changed as we've moved to JIRA. The following parameters are now changed/added:

  • rt_url is now ticket_url
  • ticket_queue is now required for rt_url to support the base URL path corresponding to your ticket queue.

• In general it's always a good idea to diff your running quads.yml with the new version to compare any new options that might have been added:

diff -u /opt/quads/conf/quads.{yml,yml.rpmnew}

All Changes

You can view the full changelog here

QUADS 1.1.2

About 1.1.2

QUADS 1.1.2 is a major enhancement and bug fix release for the 1.1 gaúcho series. This comes almost three months after 1.1.1 and building on the the landmark, next-generation 1.1.0 series.

1.1.2 is comprised of around 11 x feature enhancements and 20 x fixes.

🎆 A new year, a new QUADS! 1.1.2 arrives as we move into 2020! 🍾

Feature Highlights

• Systems and assignment reporting functionality has been added in 1.1.2

  • quads-cli --report-available
  • quads-cli --report-scheduled --months 6

• Validation now occurs for hosts added to an already-existing environment

• Flexible overcloud parameters

• Configuration value added to ignore environments with foreman_heal.py

• Date arguments added for --move-hosts --dry-run #299

• validate_env.py now has a --debug option

• Manual validation documentation is now available

• Easier to use VLAN q-in-q output for quads-cli --ls-qinq to see your cloud VLAN settings/mode

• quads-cli --date is now re-implemented for the 1.1 series #235

• Support for Dell r640 systems for Badfish

Bug Fixes

• Fix for Q-in-Q VLAN usage
• Fix for --rm-schedule
• Fix for premature cloud validations when adding future systems to join an already-running environment at a later date.
• Input sanitization fixes for non-existing environments referenced
• Wiki visual fixes for column sorting when not generating instackenv.json
• Added reboot directive for Badfish failures
• Important fix for cloud history rendering in visuals and in some situations they are not being rendered, or rendered in duplicate areas.
• Fix for resetting the host object on --rm-schedule
• Other fixes for --rm-schedule post-automation when systems have their current, active schedule removed.
• Fix for q-in-q VLAN remaining true after the environment is redefined
• Further q-in-q VLAN fixes and network automation fixes.
• Important foreman_heal.py fixes

All Changes

You can view the full changelog here

QUADS 1.1.1

About 1.1.1

QUADS 1.1.1 is a minor enhancement and bug fix release for the 1.1 gaúcho series. This comes a month after the landmark 1.1.0 stable release and is comprised of around 17 x enhancement and bug fix patches.

🎂 QUADS 1.1.1 also happens to land on the 3rd birthday of QUADS (2016-10-31) 🎃

Feature Highlights

• Monthly allocation percentage available on heatmap visual
• Percentage of environment provisioning/validation now visible on the auto-generated assignments wiki 287

• Better error messages when quads-server daemon is down 286
• Setting a cloud to nowipe now skips validations 283
• IPMI credentials validation are implemented 145 and 280
• VLAN ranges that QUADS uses to automate network changes is now user-configurable 295

Bug Fixes

• Several important Badfish library fixes
• Fixes around cloud data fields
• Improved support for Dell r740xd systems with Badfish
• Around 7 x bug fixes with the full changelog here

QUADS 1.1.0

QUADS 1.1.0 gaúcho brings about a near-rewrite of QUADS with massive improvements, architectural and design enhancements and next-generation framework updates. Special thanks to our existing and new contributors!

Highlights

• Complete move to Python3.6+
• Move to asyncio for provisioning concurrency
• MongoDB database backend replaces flat YAML files
• MongoEngine drives object document mapping
• CherryPy Python Web Framework for API
• Massive re-factoring and code structure improvements
• All shell tools rewritten in Python or made proper libraries
• Expect network automation ported to PyExpect
• Foreman provisioning is now done concurrently across all systems
• Systems/Network provisioning time improvement per system of 34%
• Lots and lots of bug fixes and other improvements

Changes

• 323 commits since 1.0.2
• 68 total issues closed and many other countless fixes
• Over 35 bugs fixed
• More than 25 feature enhancements

Platform Changes

• RHEL7/CentOS7 is no longer supported via RPM due to Python3 dependencies
  • Fedora or RHEL8/CentOS8 are the recommended platform for RPM package deployment
• A fully container-based deployment is available in addition to RPM or source installations

Links

• QUADS Blog
• QUADS COPR RPM Repository

QUADS 1.0.2

This will be the last active release of QUADS for the 1.0 series, though we may backport fixes here for any of our environments still running 1.0 codebase.

Soon, we'll be shifting to the 1.1 version which is a near-rewrite using Python3, CherryPy, MongoDB and lots and lots of improvements.

Release Highlights

• RPM packages found here
• Bug fixes in wiki generation and VLAN stub creation
• Removal of bin/quads.py in lieu of quads-cli
• Lots of documentation updates and additions
• Fixes to Juniper automation to rollback uncommitted changes
• All changes can be found here

QUADS 1.0.1

• Bump version 1.0.1
• PDU control feature added - issue #100
• Public VLAN management added into cloud definitions - issue #192
• We can now check against broken hosts in Foreman if broken_state
  host parameter is set before allowing those machines to be scheduled - issue #190
• Lots of other bug fixes, minor features and fixes to warrant bumping the version.

QUADS 1.0.0

Same as v1.0 but including updated RPM .spec bump, small documentation fixes and some missing Ansible fixes.

QUADS 1.0

Adding a release/tag on our last commit. We are going to start migrating to the CherryPy framework, MongoEngine and other functionality as outlined here so this is a good time to tag the repository:

#140