Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PostgresUser to examples #2836

Merged
merged 16 commits into from
Aug 24, 2024
Merged

Conversation

guel-codes
Copy link
Contributor

Overview

This PR creates a new database/ sub-dir inside of examples where database related files can be stored. Currently this only supports PostgreSQL. Ideally in the future we can have support for Mongo, MySQL, Redis, etc.

Changes

  • examples/database/README.md
    • This will store instructions on how to run the load test on various databases
  • examples/database/postgres_test.py
    • This is the file where the PostgreSQL Client was created

@cyberw
Copy link
Collaborator

cyberw commented Aug 5, 2024

How well does this handle concurrent queries? For locust-plugins we use psycogreen.gevent.patch_psycopg() to make psycopg greenlet-capable: https://github.com/SvenskaSpel/locust-plugins/blob/master/locust_plugins/listeners/timescale.py

(its not a locust User but is used to write requests and other test log data to a database)

@cyberw
Copy link
Collaborator

cyberw commented Aug 5, 2024

Scratch that, I can see that psycopg3 has built in support for gevent! Awesome!

@cyberw cyberw changed the title Postgres Test Example Add PostgresUser to examples Aug 5, 2024
@cyberw
Copy link
Collaborator

cyberw commented Aug 5, 2024

Can you split this into two files? One that contains a base PostgresUser marked with abstract = True and a separate one (the actual locustfile, maybe called postgres_ex.py) that imports PostgresUser from that file?

Also, maybe you can read connection settings from class variables of the final User class defined in the actual locustfile? Another option is to just use the standard postgres env vars (PGHOST, PGUSER etc)

@guel-codes
Copy link
Contributor Author

@cyberw I broke these files out and nested them inside of their own database/postgresql/ directory. Let me know if you have any questions or preferences on how to proceed. Thanks.

examples/database/README.md Outdated Show resolved Hide resolved
examples/database/README.md Outdated Show resolved Hide resolved
examples/database/README.md Outdated Show resolved Hide resolved


class PostgresClient:
def __getattr__(self, name):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think you need the whole __getattr__ song and dance :)

Just define the execute_query method directly on the PostgresClient class.



def execute_query(conn_string, query):
db_conn = create_conn(conn_string)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this create a new connection on each query? That sounds very wasteful. Probably you wanna do that only when the client object is created.

response_length=0,
exception=e,
)
print(f"error: {e}")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think we should print the exception :)

import psycopg


def create_conn(conn_string):
Copy link
Collaborator

@cyberw cyberw Aug 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this method is too small to make sense! The code is easier to understand if you just call .connect directly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, it looks a lot cleaner calling the psycopg.connect directly

def execute_query(self, query):
start_time = time.time()
try:
cursor = self.connection.cursor()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

psycopg3 can do connection.execute(), no need to have the intermediate step of creating a cursor.


class PostgresLocust(PostgresUser):
tasks = [UserTasks]
min_wait = 0
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the wait_time stuff and just leave it as the default, to simplify the example.

You can skip the taskset and put the tasks directly under the user too.

@cyberw
Copy link
Collaborator

cyberw commented Aug 24, 2024

Real close now, just a few last simplifications.

@guel-codes
Copy link
Contributor Author

@cyberw I made the updates for this. I think it is A LOT better than when it started. Thanks for all your patience as I made changes 🙏🏽

@cyberw cyberw merged commit 4d050be into locustio:master Aug 24, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants