Skip to content

Open Hour Agendas and Notes: 2020 05

Jump to bottom
Cassandra edited this page Jun 3, 2020 · 1 revision

2020-05-28

Agenda

  • Salt news and updates
  • SEP #28: Move unsupported releases to a separate URL
  • SEP #18: Release label improvements
  • Feedback Item of the Week: POP build and Salt Bin discussion
  • Open discussion and questions

Salt News and Updates

Cassandra Faris

  • New event schedule for Meetups and other events coming to community.saltstack.com
  • First German language meetup on May 27 went well. It included a formal talk and open conversation. Additional Meetups in other languages are coming.
  • Watch for the weekly "What's Happening at SaltStack This Week?" post on the Community Wiki and in Slack. It details the event schedule, SEPs, release updates, and the week's Core Team focus in one place.
  • Weekly communication schedule
    • Monday: "What's Happening?" post
    • Tuesday: Publish Open Hour Agenda
    • Thursday: Community Open Hour

SEP #28: Move unsupported releases to a separate URL

Cassandra Faris

SEP #18: Release Label Improvements

Wayne Werner

  • https://github.com/saltstack/salt-enhancement-proposals/pull/24
  • This SEP is a process improvement. Currently, labels are sometimes confusing because they have both a priority and a visibility that don't always align. This simplifies labels to make it more clear to tell what is a priority.
  • Proposed priorities in order of highest to lowest:
    • Critical: Issues that are severe enough to block a release. They'll impact most or all users in a specific area and cause problems like data loss and crashes.
    • High: Serious issues that aren't serious enough to block a release. They'll likely affect most people but will also have a workaround.
    • Medium: Issues that affect about half the people in a particular area. These produce incorrect or bad functionality or confusing user experience
    • Low: Cosmetic issues or those with simple/reasonable workarounds
  • Priority labels will help contributors know how quickly various issues will be fixed

Feedback Item of the Week: POP Build and Salt Bin Discussion

Tyler Johnson

  • POP Build, being renamed Tiamat to remove confusion with POP
  • Makes building Salt easier to develop and deploy by packaging it into a single binary
  • POP build and Salt Bin have been used the last few months for Arista native minions at IBM, Juniper native minion, and Cisco being built. Being used at limited sites.
  • POP build will make it so that the version of Python doesn't matter since it will be able to be packaged as a single binary
  • Additional POP Build information: Watch Tom Hatch's Maryland Python Virtual Meetup presentation at https://www.youtube.com/watch?v=wt3lrLYXvFg

Open Discussion and Questions

Community and Team Core

  • Question: Can you please clarify Py2 support? There's concern that not everyone knows that Py2 will no longer be supported.
    • We've been sharing this news for a year through an open SEP, blog posts, Open Hours, mailing lists, and announcements. Core Team is open to other suggestions for communication outlets and plans to make more effort to keep Salt's Twitter, Stack Overflow, and Reddit communities updated.
    • POP Build will help address this issue because Salt will be packaged with Python into a single binary
    • Python 2 is being end-of-lifed. The industry is moving forward with Python 3. It won't be removed from Salt all at once, but over time as releases progress
  • Question: There are people using modules in Jinja that use Py2 features that don't work without modification in Py3. Salt itself will work, but installations might not. How to resolve this?
    • Start by creating GitHub issues for those specific issues so we can fix them
  • Question: How would POP Build solve external states library dependency like Elastic Search?
    • User can still do a pip install into the POP Build environment to install extra dependencies
  • Question: Is it possible to add dependencies into the POP Build process? By the time Salt from POP Build starts can it have its dependencies installed? How does this apply to both third party dependencies and the underlying libraries?
    • With the pip install, hard dependencies are there and available right away. Optional dependencies will still need installed into POP Build.
    • Documentation about this is being updated. If you want to help with documentation, check out the Documentation Working Group. They'll be aware of this.
  • Question: Can users create their own installation packages to include those extra dependencies rather than needing to install them to each after Salt has been installed?
    • Yes. Users can create their own POP (Tiamat) Build of Salt and choose the dependencies they want rather than the standard dependencies. People can fork it and build their own self-contained versions.
  • Will users be able to use Tiamat to build on Windows as well as Linux platforms?
    • Yes. It's not in place for Windows yet, but it will be.
  • What's the status of PyTest deployment?
    • As of today, the Master branch has one test failure under PyTest which doesn't happen under run tests. This is because we weren't actually testing that test case using run tests. There's a PR to fix that. It may get into Sodium.
    • We haven't switched because it's getting so close to the release. We want them to run for a couple weeks to ensure that results are the same when we run tests.
    • After code freeze, we should be ready to make the full switch. The ground work to run PyTest and have it pass is done.

2020-05-21

Agenda

  • Salt news and updates
  • Closing SEP: Python version support
  • New SEP: Move unsupported releases to a separate URL
  • Feedback Item of the Week: CVE in salt_rc packages
  • Community-Requested Topic: New distribution support policy
  • Open discussion and questions

Salt News and Updates

Cassandra Faris

  • Thanks to community for suggested topics and conversation about previous and unsupported releases Sodium Release timeline: May 22 is the PR deadline. All PRs need passing tests-
  • Meetups back in place starting with SaltStack Germany UG with Elias Probst speaking on SaltStack Between Guardrails
  • YouTube uploads being updated
  • Community Wiki Page Updates: Changed title on Open Hour notes, moved most recent notes to top, created About the Open Hour page
  • Open Hour timeline
    • Agendas published on Tuesdays
    • If a contributor requests a topic, they'll be notified when the topic is going to be covered in an Open Hour so they don't miss their request

Closing SEP: Python Version Support

Cassandra Faris and Wayne Werner

New SEP: Unsupported Releases

Bryce Larson

  • SEP: ttps://github.com/saltstack/salt-enhancement-proposals/pull/28
  • The goal is to make it explicit that someone is at risk of installing an unsupported version
  • Much like Ubuntu and Debian move unsupported releases to another site, we are going to do the same thing
  • Any time a major release goes out of support, we'll move anything from that release to the archive website
  • Any time a point release is affected by a CVE, we'll move it to the archive site
  • PyPI's old releases will move to a SaltStack hosted location
  • This SEP has an expedited timeline due to CVE-related urgency. It takes effect on June 1

Feedback Item of the Week: CVE in salt_rc packages

David Murphy

  • Previously when Salt has had RCs, they available to download on the repo.saltstack.com directory. Those RCs were out-of-date. We usually leave them up until there's another RC. In this case, we removed them because they had CVE issues
  • When Salt releases Sodium, we'll update the page with the latest RCs

New Distribution Support Policy

Sebastien Blaisot and Core Team

  • Background: Each time a user deploys a new server, the first thing they want available on the server is a Salt minion because it's the start to deploy anything else on a new server. Can Salt packages be available as soon as a new version comes out? How can the community help make Salt packages available earlier?
  • When Core releases a RC, they test it internally. The more testing they get from the community, the more stable Salt will be which will lead to quicker releases
  • Core Team is in the process of moving packaging to Salt Bin (Salt's single binary distribution /directory) which should make it faster to target and support newer OS versions. It simplifies dependency management.
    • Suggestion: Create a SEP to discuss how to get this into an automated pipeline with Salt Bin
    • Salt Bin should allow us to create self-contained versions that have fixes to allow for a specific operating system before it's back into the main version of Salt. The result will be faster releases. Self-containment allows less hassle in dealing with dependencies.

Open Discussion and Questions

Community and Core Team

  • Question: Is the change to Salt bin happening for Sodium?
    • The current SaltStack Enterprise version uses Salt bin to create packages that get bundled into the installer. It uses GitLab and has pipelines set up to leverage Salt bin. This creates improvements and simplifications when bundling necessary dependencies
    • For SSE upgrade questions, contact [email protected]
    • During this Open Hour, acommunity member requested a deep dive on Salt bin. That deep dive will be part of the May 28 Open Hour agenda
  • Question: How is the work on test suite improvement going?
    • The plan was to switch to PyTest by Sodium. Due to the CVE and related support, we don’t have quite enough time to do that by Feature Freeze
    • Currently there's one additional PR that makes changes to PyTest. Users should be able to run PyTest going forward
    • We've identified test suite performance issues and marked some tests as slow. This gives helps PRs passing/not passing status sooner which makes merging faster.
    • Going forward, we'll address the slow tests by verifying their validity and rewriting them
    • If you're interested in helping with tests, reach out to the Core Team who can help with setup
    • Testing Working Group is working to help us run the test suite in 60 minutes or less

2020-05-14

Agenda

  • Salt news and updates
  • Feedback Item of the Week: RSS Feed
  • Salt 2019.2.5 and 3000.3 Releases
  • Approaching Vulnerable Releases
  • Open community discussion and questions

Salt News and Updates

Cassandra Faris

  • Announcement coming tomorrow with updated code freeze and release candidate dates. This will be the first time we get a release candidate out a few weeks before the GA
  • PR Merge Jam was placed on hold due to CVE-related time constraints. If you have a PR, it needs to have passing tests by the 19th to make it into Sodium
  • The Security RSS feed is live: https://www.saltstack.com/security-announcements/

2019.2.5 and 3000.3 Releases

Moe Anderson

Recalling Vulnerable Releases

Moe Anderson and Open Hour Attendees

  • This has been a common topic of discussion lately. Thanks to community for feedback about these releases. People want to know what to do with the out-of-support packages that are inherently vulnerable to the CVE. Unless they're patched, downloading those versions opens risk of exploitation.
  • Two schools of thought regarding unsupported versions
    • We should keep the unsupported versions available because people will need them, and find ways to flag and alert people that they need to install the packages.
    • Consistent with precedent in other projects, these versions should be removed and require people to make a special request to access them. That helps ensure that everyone who uses the older versions will know about and install the patch
  • Why not refresh all of the packages? It's not a viable option. Salt carries a custom setup with many distributions. It's not a reasonable approach to refresh them all
  • Will be making a decision on how to approach this, but needs further discussion. We'll share the decision that is made on all channels.
  • Security firm has told us that vulnerable packages are still being downloaded, but patch application hasn't been automated
  • Send thoughts on this topic to Moe

2020-05-07

Agenda

  • Salt news and updates
  • Feedback Item of the Week: additional communication channels
  • CVE and security patch release
  • POCs and API discussion led by Tai Groot
  • Open community discussion and questions

Salt news and updates

Cassandra Faris

  • PR Merge Jam and Test Clinic on hold to support CVE
  • Ongoing events: Hacks podcast, daily Twitch steam, weekly Open Hours and virtual meetups, and working group meetings. All can be found on the community events calendar.

Feedback Item of the Week: Additional communication channels

Cassandra Faris

  • The team has recently received requests for RSS feed and security email.
  • We will be implementing both of these things and will share when they're in place
  • We'll make sure that the security updates and emails are focused on security only and that they won't be connected to sales or marketing

CVE Release

Moe Anderson

  • Summary
    • As of May 7, we have seen an incredible amount of collaboration across the community, folks leveraging their expertise, and helping one another with patches and updates. Thank you!!
    • A sizable chunk of people still haven't applied the CVE patches. We're going to continue to share messaging and raise awareness
  • Blog post content clarification (https://community.saltstack.com/blog/active-saltstack-cve-critical-updates/)
    • When we say patch, we mean a patch that's applicable to all unsupported versions
    • When we refer to RPM or package, we mean something for the currently supported versions
  • Two waves of patches
    • 4/29: All of these patches contain all that is needed to address the CVE. If you picked up and applied the 4/29-4/30 patch, you're good
    • As the week went on, we found 2 issues that were functional regressions causing the master to throw exceptions and errors in the master. They are not part of the security update, but rather a fix for the master. Only applies to those on 2017 and earlier
    • For 2019 and 3000, we are going to proceed and update the packages. The new packages will be available next Wed. (2019.2.5 and 3000.3). These are being updated because we've identified two issues that introduce functional regressions, not because of security concerns
  • Docker images will be updated next week
  • Request for updates to document on how to harden Salt deployments and reinforce best practices for security
  • Communications in general
    • We appreciate the feedback on best practices and are considering all of the suggestions people make. Community will be updated as we implement these suggestions.
    • We'll also be reinforcing the communication plan so people know what to expect when there are releases or other major announcements

POCs and API Discussion

Moe Anderson and Tai Groot

  • POCs

    • We've been asking those who have found how to exploit to give us a grace period to allow people to patch
    • The common ethical method is to allow people time or create a tool that allows people ensure that they've been protected
    • We'll be partnering with experts during the blackout period in order to create some utilities to ensure
    • We can't control or force those who are sharing POCs. Thank you to the #salt-store-miner channel for working on this.

  • API Tool Overview (Tai Groot)

    • He's been updating saltexploit.com frequently - this is not affiliated with SaltStack in any official capacity. The SaltStack offline checker will be shared on this site as well as SaltStack's official tool.
    • What is the progression of exploits we've seen? What malware?
      • We can't guarantee that the first person to use this exploit was the mining application. The attacker used the exploit to grab the master's key and run a command against all the minions to run a shell script that deleted files, killed processes, and downloaded and run a payload. We have copies of those. That payload was called salt store.
    • The initial attack wasn't too malicious. All that was required to prevent this is to update or patch the salt master and restart
    • The next day, a second version was uploaded . We know exact times because bitbucket was used. It was used because it’s whitelisted
    • Version 4 was tricky to remove, then with version 5 the advice changed completely. An additional attacker became involved, likely using POCs that were published. The attack added a worm. Current advice:
      • Understand that there will be losses. The vulnerability has been shared widely now and made its way into media. It's the responsibility of admins to update the systems.
      • Shut down Salt Master (take a backup & snapshot for forensics state)
      • Set up a new Salt Master and make sure it's updated to the latest version of SaltStack
      • Consider using tools for ensuring you're not vulnerable created by Tai, SaltStack, and another community user
      • After verifying you're not vulnerable, you can attach minions to the master, but that's risky. If compromised minions are attached to compromised master, another master will need to be spun up
      • Once things are up and running, do some forensics and assume that if it was on the box it was taken
      • Consider changing API keys

  • Can you describe the API tool? How do we ensure it's not being exploited?

    • If you run the API, a new file is created that is called hacked.txt
    • He has built and provided this API with some caveats - linked back to part in GitHub when he first announces this. Concern that people are sending their IP addresses to a stranger on the internet. There is valid concern about whether to trust Tai. He's not going to share exactly how to exploit the POC.
    • Motivation: People wouldn't be interested in this until/unless it affects them. How can he make is easy for people to check whether they're vulnerable? He considered writing a POC, but acknowledged it would be very easy to weaponize
      • He sent this out for a peer review
      • When he set this up for his own system, he was going to run something local. He ran into issues importing Salt on Python 3.8. He created an API to support and be helpful
      • He's not logging any of this information

Community Discussion and Questions

Core Team

Q&A

  • To confirm, if you're on 2018 do you need to apply additional packages?
    • No. You're covered by the initial CVE patch
  • When is Salt breaking on Ubuntu 20.04?
    • Ubuntu 20 wasn't one of the supported areas. We're working toward making it supported. When we support it, we will fix it. We'll see if there's a way we can accelerate.
  • Is the sodium release date changing?
  • What's the latest on the PR Port Jam?
    • Community members want to help with PRs that will be ported into master, we've gone from 900 to ~350 in the backlog. The intent was to run through prioritized PRs with dedicated blocks of time and support
    • After tomorrow's planning meeting, we'll be making and sharing a decision on how to run the PR jam and make it a success

Closing Note

Moe Anderson and Cassandra Faris

Clone this wiki locally