From 8ec449cdc7fb5a2b0996e7c4c0d5dc6501c641d7 Mon Sep 17 00:00:00 2001 From: Bishwas Praveen Date: Wed, 20 Sep 2023 12:53:18 -0500 Subject: [PATCH 1/3] Added instructions on setting up the project with a db backup and also instructions on pre-commit hooks --- README.md | 109 ++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 98 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 2b3efe6e..a262026b 100644 --- a/README.md +++ b/README.md @@ -11,21 +11,33 @@ Moved to [settings](http://cookiecutter-django.readthedocs.io/en/latest/settings ## Basic Commands +### Building The Project + ```bash + $ docker-compose -f local.yml build + ``` + +### Running The Necessary Containers + ```bash + $ docker-compose -f local.yml up + ``` + ### Setting Up Your Users - To create a **normal user account**, just go to Sign Up and fill out the form. Once you submit it, you'll see a "Verify Your E-mail Address" page. Go to your console to see a simulated email verification message. Copy the link into your browser. Now the user's email should be verified and ready to go. - To create a **superuser account**, use this command: - - $ python manage.py createsuperuser + ```bash + $ docker-compose -f local.yml run -rm django python manage.py createsuperuser + ``` For convenience, you can keep your normal user logged in on Chrome and your superuser logged in on Firefox (or similar), so that you can see how the site behaves for both kinds of users. ### Loading fixtures - To load collections - - docker-compose -f local.yml run --rm django python manage.py loaddata sde_collections/fixtures/collections.json + ```bash + $ docker-compose -f local.yml run --rm django python manage.py loaddata sde_collections/fixtures/collections.json + ``` ### Loading scraped URLs into CandidateURLs @@ -36,26 +48,74 @@ For convenience, you can keep your normal user logged in on Chrome and your supe - Run the crawler with `scrapy crawl -o scraped_urls//urls.jsonl - Then run this: + ```bash + $ docker-compose -f local.yml run --rm django python manage.py load_scraped_urls + ``` + +### Loading The DB From A Backup + +- If a database backup is made available, you wouldn't have to load the fixtures or the scrapped URLs anymore. This changes a few steps necessary to get the project running. + +- Step 1 : Build the project (Documented Above) + +- Step 2 : Run the necessary containers (Documented Above) + +- Step 3 : Clear Out Contenet Types Using Django Shell + + -- Enter the Django shell in your Docker container. + ```bash + $ docker-compose -f local.yml run --rm django python manage.py shell + ``` + + -- In the Django shell, you can now delete the content types. + ```bash + from django.contrib.contenttypes.models import ContentType + ContentType.objects.all().delete() + ``` + + -- Exit the shell. + +- Step 4 : Load Your Backup Database + + Assuming your backup is a `.json` file from `dumpdata`, you'd use `loaddata` command to populate your database. + + -- If the backup file is on the local machine, make sure it's accessible to the Docker container. If the backup is outside the container, you will need to copy it inside first. + ```bash + $ docker cp /path/to/your/backup.json container_name:/path/inside/container/backup.json + ``` + + -- Load the data from your backup. + ```bash + $ docker-compose -f local.yml run --rm django python manage.py loaddata /path/inside/the/container/backup.json + ``` + + -- Once loaded, you may want to run migrations to ensure everything is aligned. + ```bash + $ docker-compose -f local.yml run -rm django python manage.py migrate + ``` - $ docker-compose -f local.yml run --rm django python manage.py load_scraped_urls ### Type checks Running type checks with mypy: - + ```bash $ mypy sde_indexing_helper + ``` ### Test coverage To run the tests, check your test coverage, and generate an HTML coverage report: - + ```bash $ coverage run -m pytest $ coverage html $ open htmlcov/index.html + ``` #### Running tests with pytest + ```bash $ pytest + ``` ### Live reloading and Sass CSS compilation @@ -63,15 +123,17 @@ Moved to [Live reloading and SASS compilation](https://cookiecutter-django.readt ### Install Celery -Make sure Celery is installed in your environment. -To install, -pip install celery +Make sure Celery is installed in your environment. To install : + ```bash + $ pip install celery + ``` ### Install all requirements Install all packages listed in a 'requirements' file - + ```bash pip install -r requirements/*.txt + ``` ### Celery @@ -100,6 +162,31 @@ cd sde_indexing_helper celery -A config.celery_app worker -B -l info ``` +### Pre-Commit Hook Instructions + +Hooks have to be run on every commit to automatically take care of linting and structuring. + +To install pre-commit package manager : + + ```bash + $ pip install pre-commit + ``` + +Install the git hook scripts : + + ```bash + $ pre-commit install + ``` + +Run against the files : + + ```bash + $ pre-commit run --all-files + ``` + + It's usually a good idea to run the hooks against all of the files when adding new hooks (usually `pre-commit` will only run on the chnages files during git hooks). + + ### Sentry Sentry is an error logging aggregator service. You can sign up for a free account at or download and host it yourself. From 00a765d0ded194203f5c4fb687637608d4cfa740 Mon Sep 17 00:00:00 2001 From: Bishwas Praveen Date: Tue, 26 Sep 2023 10:38:07 -0500 Subject: [PATCH 2/3] Fixed the issue with the push collections to github button --- sde_collections/utils/github_helper.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/sde_collections/utils/github_helper.py b/sde_collections/utils/github_helper.py index b6234803..60688447 100644 --- a/sde_collections/utils/github_helper.py +++ b/sde_collections/utils/github_helper.py @@ -52,6 +52,19 @@ def _update_file_contents(self, collection): branch=self.github_branch, ) + def branch_exists(self, branch_name: str) -> bool: + try: + self.repo.get_branch(branch=branch_name) + return True + except GithubException: + return False + + def create_branch(self, branch_name: str): + # Get the SHA of the commit you want to branch from (basically the Dev branch) + base_sha = self.repo.get_branch(self.dev_branch).commit.sha + # Create the new branch + self.repo.create_git_ref(ref=f"refs/heads/{branch_name}", sha=base_sha) + def create_pull_request(self) -> None: title = "Webapp: Update config files" body = "\n".join(self.collections.values_list("name", flat=True)) @@ -66,6 +79,8 @@ def create_pull_request(self) -> None: print("PR exists") def push_to_github(self) -> None: + if not self.branch_exists(self.github_branch): + self.create_branch(self.github_branch) for collection in self.collections: print(f"Pushing {collection.name} to GitHub.") self._update_file_contents(collection) From 78ea04f498c3bc96416a27fd758e81bdee3fa10c Mon Sep 17 00:00:00 2001 From: Carson Davis Date: Thu, 28 Sep 2023 10:16:06 -0500 Subject: [PATCH 3/3] add note about fixtures --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a262026b..01efe654 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ Moved to [settings](http://cookiecutter-django.readthedocs.io/en/latest/settings For convenience, you can keep your normal user logged in on Chrome and your superuser logged in on Firefox (or similar), so that you can see how the site behaves for both kinds of users. ### Loading fixtures - +Please note that currently loading fixtures will not create a fully working database. If you are starting the project from scratch, it is probably preferable to skip to the Loading the DB from a Backup section. - To load collections ```bash $ docker-compose -f local.yml run --rm django python manage.py loaddata sde_collections/fixtures/collections.json