Changes made during setup only effect the first user

[Jiff21]

Describe the bug

Values made during setup (regardless of if it's set on UserBehavior, WebsiteUser or on HttpLocust) are cleared for all locust except for the first one.

Expected behavior

Variable made during setup will effect all locust. Additionally default session should be editable. Or the documentation should be updated to reflect Setup only effecting one User.

Actual behavior

Variables made during setup will only effect the first TaskSet that runs. Other Users will not have access to things done in setup. For example file:
Stdout:

Yes this runs
on_start self.locust.bearer_header is #####################
on_start after setting headers self.client.headers are {'User-Agent': 'python-requests/2.20.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': '#####################'}
on_start self.locust.bearer_header is None
on_start after setting headers self.client.headers are {'User-Agent': 'python-requests/2.20.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': None}
first task self.client.headers are {'User-Agent': 'python-requests/2.20.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': '#####################'}
first task self.client.headers are {'User-Agent': 'python-requests/2.20.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': None}
second task self.client.headers are {'User-Agent': 'python-requests/2.20.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': '#####################'}
second task self.client.headers are {'User-Agent': 'python-requests/2.20.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': None}

Currently the documentation implies this should effect all users. If you don't plan on changing setup documentation should be updated.
From https://docs.locust.io/en/stable/writing-a-locustfile.html#setups-teardowns-on-start-and-on-stop:

setup and teardown, whether it’s run on Locust or TaskSet, are methods that are run only once. setup is run before tasks start running,...

Seems to be running before first TaskSet but not before tasks are run. Or

This enables you to perform some preparation before tasks start running (like creating a database) and to clean up before the Locust quits (like deleting the database).

Would be helpful if you mentioned you can't changes made here will only effect the first locust. Or show some sort of example of how to edit the session (besides cookies) in a way that will effect all Users if I'm not running setup on Locust correctly. Some of the confusion may be like previous ticket I read says in that Locust is both name of tool and name of User and I'm reading it as tool.

But it would make sense if there was a setup that ran once before the test and could effect session defaults and once before each locust runs. But once before a single locust is confusing. Currently it seems the built in setup functions force you to set something that effects all users you need to either do it on on_start (which may make you call an API per client which causes issues) or set it up outside Locust which just doesn't look clean.

Steps to reproduce

1. Run the file below with the "Locust command line that you ran"

Environment

• OS: OSX 10.14.6

• Python version: Python 3.6.8

• Locust version:locust 0.14.4

• Locust command line that you ran:
  locust -c 5 -r 5 -t 30s --no-web --csv=example.txt -H http://google.com -f example.py

• Locust file contents (anonymized if necessary):

from locust.core import HttpSession
from locust import Locust, TaskSet, task, between


def api_call_with_quota_that_does_not_set_cookie():
    return {'Authorization': '#####################'}

class UserBehavior(TaskSet):

    def on_start(self):
        print('on_start self.locust.bearer_header is %s' % self.locust.bearer_header)
        self.client.headers['Authorization']= self.locust.bearer_header
        print('on_start after setting headers self.client.headers are %s' % str(self.client.headers))

    def on_stop(self):
        pass


    @task(1)
    def first(self):
        self.client.get('/')
        print('first task self.client.headers are %s' % str(self.client.headers))

    @task(1)
    def second(self):
        self.client.get('/about')
        print('second task self.client.headers are %s' % str(self.client.headers))



class HttpLocust(Locust):

    client = None
    trust_env = None
    bearer_header = None

    def __init__(self):
        super(HttpLocust, self).__init__()
        if self.host is None:
            raise LocustError("You must specify the base host. Either in the host attribute in the Locust class, or on the command line using the --host option.")
        session = HttpSession(base_url=self.host)
        session.trust_env = self.trust_env
        self.client = session


    def setup(self):
        """ setup is called before running on_start """
        print('Yes this runs')
        header = api_call_with_quota_that_does_not_set_cookie()
        self.bearer_header = header['Authorization']


class WebsiteUser(HttpLocust):

    task_set = UserBehavior
    wait_time = between(5, 9)

Note: I do know you could work around this by running:

def api_call_with_quota_that_does_not_set_cookie():
    return {'Authorization': '#####################'}

bearer_header = api_call_with_quota_that_does_not_set_cookie()

at the top of the file, but it seems deceiving to have a Setup that doesn't work as Test Frameworks usually use Setup.

[cyberw]

I think the core of this confusion is that setup is only run once, for the first Locust/TaskSet.

@heyman How would you feel about changing setup/teardown to functions instead of object methods (so people cant access self and thus get these unexpected behaviours)

[heyman]

@cyberw I think that's a good idea. I don't think that global setup/teardown functions for a whole test run belongs on the user class at all.

I'm working on a branch where I'm refactoring quite a bit of Locust's internal code. I'm introducing an Environment class that is used to tie things together in a cleaner way, and get rid of the global stats and global locust_runner references, as well as the global event hooks (instead they will all be available through the Environment instance. It's still very much work in progress with a lot of broken stuff, but I'm aiming to get it to a somewhat working state as soon as possible so that I can push it in order to get feedback. Anyway, I think it would be a good idea to use locust events for the setup and teardown hooks. I'm imagining an API that could look something like this:

from locust import environment

def setup(**kwargs):
    # run setup code

def teardown(**kwargs):
    # my teardown

environment.events.test_start.add_listener(setup)
environment.events.test_stop.add_listener(teardown)

[cyberw]

Cool! Lets wait for that them!

[weitzman]

Seems like part of this got done? Is there an issue we can subscribe to for any open items?

[heyman]

@weitzman Yes, you're right! In the upcoming 1.0 version the setup and teardown methods has been replaces with the test_start and test_stop events (more info here: https://docs.locust.io/en/latest/writing-a-locustfile.html#test-start-and-test-stop-events)