Improve + Fix Bandersnatch Docker Compose Setup: Volume Configuration…

… + HTTPS Support (#1653)

* Update nginx service configuration in docker-compose.yml

This commit updates the bandersnatch_nginx service in the docker-compose.yml file. Changes include:

1. Specifying the correct build context and Dockerfile path for the Nginx image. The build context has been updated to point to the '../banderx' directory, which contains the Dockerfile and nginx.conf as per the structure. (since docker-compose doesn't handle links well in some versions)

2. Mounting the correct nginx.conf file.
Previously, the nginx.conf file was assumed to be in the current directory. This update mounts the nginx.conf from '../banderx', following the instructions in '../banderx/README.md', ensuring that the Nginx server uses the appropriate configuration file for serving the mirrored data.

3. Updated comment has been updated to more accurately describe its purpose - serving the downloaded repository.

* Adding https support for docker-compose

Added commented lines for uncomment when HTTPS is wanted

* Add optional HTTPS support to nginx.conf

This commit introduces optional HTTPS support in the nginx.conf file. The changes are as follows:

1. Add of a new server block for handling HTTPS (port 443).
This block is commented out by default and can be enabled by uncommenting the lines. It includes settings for SSL certificates and keys, which are expected to be mounted into the container at specified paths.

2. Additional server block for redirecting HTTP to HTTPS, also commented out by default.
When enabled, this block will redirect all HTTP traffic to HTTPS, enhancing security.

3. The existing HTTP server block (port 80) remains unchanged and will continue to function as the default server.

This is to allow users to easily enable HTTPS by uncommenting the relevant sections and ensuring that the SSL certificates and keys are correctly mounted into the Nginx container.

* Update README with HTTPS setup instructions

This commit updates the README in the bandersnatch_docker_compose directory to include detailed instructions for enabling HTTPS support. The key additions are:

1. Steps to uncomment relevant sections in the `docker-compose.yml` for mounting SSL certificate and key files, and exposing the HTTPS port 44300.

2. Steps for placing SSL certificate and key files in `banderx` directory.

3. Guide on uncommenting the HTTPS server block in the `nginx.conf` file to enable SSL support and optional HTTP to HTTPS redirection.

4. Steps to rebuild and restart the Docker containers after making these changes.

5. Simple methods to test the HTTPS connection using a web or the `curl`.

* Added table of contents

* Add PR #1653 to CHANGES.md

* committing pre-commit fixes

* Update docker-compose.yml

* Fix bandersnatch command in docker-compose file

* Add guide for generating SSL certificate for HTTPS support

* Update README.md

* Fix parameter suggested by pre-commit

* pre-commit README.md fix

* Added fixes done by pre-commit

* Fix Table of Contents

* Added pre-commit fixes

* added pre-commit fixes

* pre-commit fixes

* "Added a function to convert paths to Windows compatible format

* pre-commit fix

* Revert "pre-commit fix"

This reverts commit c01977b.

* Revert ""Added a function to convert paths to Windows compatible format"

This reverts commit 79b73bd.

* excluded 'bandersnatch_docker_compose' from test discovery

* pre-commit fix

* Update src/bandersnatch_docker_compose/README.md

added -v, --verbose - to make curl verbose during the operation. Useful for debugging and seeing what's going on "under the hood". A line starting with '>' means "header data" sent by curl, '<' means "header data" received by curl that is hidden in normal cases, and a line starting with '*' means additional info provided by curl.

Co-authored-by: Cooper Lees <[email protected]>

* removed .* from norecursedirs

* Update CHANGES.md

Move change entry to Unreleased

---------

Co-authored-by: Cooper Lees <[email protected]>

CHANGES.md

@@ -1,3 +1,9 @@
+# Unreleased
+
+## New Features
+
+- Added HTTPS support in Docker Compose + Enabled bind-mount volume for Nginx config + add documentation in README.md (PR #1653)
+
 # 6.5.0
 
 ## New Features

pytest.ini

@@ -2,3 +2,4 @@
 log_cli_level = DEBUG
 log_level = DEBUG
 asyncio_mode=strict
+norecursedirs = src/bandersnatch_docker_compose

src/bandersnatch_docker_compose/Dockerfile

@@ -1 +1 @@
-../banderx/Dockerfile
+../banderx/Dockerfile

src/bandersnatch_docker_compose/README.md

@@ -1,5 +1,25 @@
 # Bandersnatch with docker compose
 
+# Table of Contents
+
+[Introduction](#introduction)
+
+[Preparation](#preparation)
+
+[Pull the Docker Image](#pull-the-docker-image)
+
+[Run with Docker Compose](#run-with-docker-compose-v2)
+
+[Watching Live Logs](#watching-live-logs)
+
+[(Optional) - Enabling HTTPS Support](#optional---enabling-https-support)
+
+[Removing the Repository](#removing-the-repository)
+
+[Caveats](#caveats)
+
+## Introduction
+
 Similar to [banderx](https://github.com/pypa/bandersnatch/tree/main/src/banderx) but with docker compose.
 
 ## Preparation
@@ -33,24 +53,105 @@ docker compose logs -f bandersnatch
 docker compose logs -f bandersnatch_nginx
 ```
 
+### (Optional) - Enabling HTTPS Support
+
+The `bandersnatch_docker_compose` setup now includes optional HTTPS support for the Nginx server.
+
+To enable HTTPS:
+
+- **Uncomment the HTTPS sections**:
+
+In `docker-compose.yml` related to SSL certificate and key volumes, as well as the exposed HTTPS port.
+
+- **Provide SSL Certificates**:
+
+Place your SSL certificate and key files in the `src/banderx` directory and name them `certificate.crt` and `private.key`, respectively.
+
+- Ensure that these files are not publicly accessible.
+
+- In case you need just a Self-Signed certificate just use the guide [here](https://github.com/pypa/bandersnatch/blob/main/src/banderx/README.md)
+
+- **Uncomment the HTTPS sections in `nginx.conf`**:
+
+In the `nginx.conf` file located in the `banderx` directory, uncomment the server block for HTTPS and the server block for redirecting HTTP to HTTPS.
+
+- **Rebuild and Restart the Containers**:
+
+After making these changes, rebuild and restart your Docker containers.
+
+```bash
+docker-compose down
+docker-compose up --build -d
+```
+
+#### Integrating with Docker Compose
+
+```yaml
+services:
+   bandersnatch_nginx:
+      # ... other configurations ...
+      volumes:
+      # ... Other volunes ...
+      - "../banderx/certificate.crt:/etc/ssl/certs/nginx.crt:ro" # SSL certificate
+      - "../banderx/private.key:/etc/ssl/private/nginx.key:ro" # SSL key
+```
+
+#### Test HTTPS Connection
+
+Ensure that your Nginx server is correctly serving content over HTTPS and access it using https.
+
+##### Using a Web Browser
+
+- Open your web browser.
+
+- Navigate to `https://pypi-repo-domain>:44300`.
+
+- You should see your site served over HTTPS.
+
+> Note - Your browser may display a security warning if you are using a self-signed certificate.
+
+##### Using `curl` Command
+
+- Open a terminal or command prompt.
+
+- Run the following command:
+
+  ```bash
+  curl -vk https://pypi-repo-domain:44300
+  ```
+
+  - Replace `<your-server-domain-or-IP>` with your server's domain name or IP address.
+
+  - The `-k` option allows `curl` to perform "insecure" SSL connections and transfers, useful if you're using a self-signed certificate.
+
+- If the Nginx server is correctly serving content over HTTPS, you should see the HTML content of your website in the terminal output.
+
+> Note
+>
+> - Ensure that the port `44300` is open and accessible from your network.
+>
+> - If you are using a self-signed certificate, web browsers and tools like `curl` may show a warning because the certificate is not signed by a recognized Certificate Authority.
+>
+> - This is expected behavior for self-signed certificates. For a production environment, it's recommended to use a certificate from a recognized Certificate Authority.
+
 ## Removing the Repository
 
 To remove the Bandersnatch repository that you've set up using Docker Compose, follow these steps. Please be aware that this process will delete all the packages and configuration files you have downloaded or created. Ensure you have backups if necessary.
 
-1. **Stop the Docker Containers**: Before removing any files, it's important to stop the running Docker containers to prevent any file corruption or data loss. Use the command:
+- **Stop the Docker Containers**: Before removing any files, it's important to stop the running Docker containers to prevent any file corruption or data loss. Use the command:
 
 ```bash
 docker compose down
 ```
 
-2. **Remove the Packages and Configuration Files**: To delete all the downloaded packages and configuration files, run the following command. This will remove the `packages` and entire `config` directory directory inside your `config` folder, which contains all the mirrored Python packages.
+- **Remove the Packages and Configuration Files**: To delete all the downloaded packages and configuration files, run the following command. This will remove the `packages` and entire `config` directory directory inside your `config` folder, which contains all the mirrored Python packages.
 
 ```bash
 rm -rf ./data/
 rm -rf ./config/
 ```
 
-3. **Clean up Docker Artifacts**: Finally, to remove any Docker volumes, networks, or other artifacts that were created with the Docker Compose file, you can use the following command:
+- **Clean up Docker Artifacts**: Finally, to remove any Docker volumes, networks, or other artifacts that were created with the Docker Compose file, you can use the following command:
 
 ```bash
 docker system prune --volumes

src/bandersnatch_docker_compose/docker-compose.yml

@@ -7,34 +7,42 @@ services:
       - "./data:/srv/pypi/web"  # Mount the local data directory where the mirror will be stored
     # To run bandersnatch in mirror mode.
     # It will mirror all the packages in the mirror list of the config file.
-    command: bandersnatch mirror --config-file /conf/bandersnatch.conf
+    command: bandersnatch --config /conf/bandersnatch.conf mirror
     # To run bandersnatch in --force mode, uncomment the following line.
     # It will delete all packages that are not in the mirror list.
-    # command: bandersnatch mirror --config-file /conf/bandersnatch.conf --force
+    # command: bandersnatch --config /conf/bandersnatch.conf --force mirror
     # To run bandersnatch in --debug mode, uncomment the following line.
     # It will print out all the debug messages. This is useful for debugging.
-    # command: bandersnatch mirror --config-file /conf/bandersnatch.conf --debug
+    # command: bandersnatch --config /conf/bandersnatch.conf --debug mirror
     # To run bandersnatch in --verbose mode, uncomment the following line.
     # It will print out all the verbose messages. This is useful for debugging.
-    # command: bandersnatch mirror --config-file /conf/bandersnatch.conf --verbose
+    # command: bandersnatch --config /conf/bandersnatch.conf --verbose mirror
     # To run bandersnatch in verify mode, uncomment the following line.
     # It will verify all the packages in the mirror list and delete the ones that are not in the list anymore.
-    # command: bandersnatch verify --config-file /conf/bandersnatch.conf
+    # command: bandersnatch --config /conf/bandersnatch.conf verify
     # To run bandersnatch in once mode, uncomment the following line.
     # It will run bandersnatch once and exit. This is useful for testing.
-    # command: bandersnatch once --config-file /conf/bandersnatch.conf
+    # command: bandersnatch --config ./conf/bandersnatch.conf once
     networks:
       - bandersnatch_net  # Connects the service to the custom bridge network
 
   # The nginx service configuration for serving the mirror
   bandersnatch_nginx:
     image: bandersnatch_nginx
     # If the image not built yet, it will be built from the nginx image from the Dockerfile in the current directory
-    build: .
+    build:
+      context: ../banderx
+      dockerfile: Dockerfile
     volumes:
-      - "./data:/data/pypi/web:ro"  # Mount the data directory as read-only for nginx
+      - "./data:/data/pypi/web:ro"  # Mount the data directory as read-only for nginx for serving the downloaded repository
+      - "../banderx/nginx.conf:/config/nginx.conf:ro"  # Mount custom nginx.conf as recommended in ../banderx/README.md
+      # Uncomment for HTTPS support
+      #- "../banderx/certificate.crt:/etc/ssl/certs/nginx.crt:ro" # Uncomment for HTTPS support
+      #- "../banderx/private.key:/etc/ssl/private/nginx.key:ro" # Uncomment for HTTPS support
     ports:
       - "40080:80"  # Map port 40080 on the host to port 80 in the container
+      # Uncomment for HTTPS support
+      #- "44300:443"  # Uncomment for expose port 44300 on the host and map it to port 443 in the container
     networks:
       - bandersnatch_net
 

src/bandersnatch_docker_compose/nginx.conf

@@ -1 +1 @@
-../banderx/nginx.conf
+../banderx/nginx.conf

src/banderx/README.md

@@ -11,3 +11,37 @@ Simple nginx PEP 501 + 691 mirror filesystem serving example.
 ## Run
 
 - `docker run --detach --name banderx banderx`
+
+## (Optional) - Generate a Self-Signed Certificate for Bandersnatch
+
+> Note - Use this only for HTTPS support
+
+- **Generate a 2048-bit RSA Private Key**:
+
+  This key is used to decrypt traffic. Run the following command in PowerShell or Command Prompt (on Windows):
+
+  ```bash
+  openssl genrsa -out ./src/banderx/private.key 2048
+  ```
+
+  This will create a private key file `key.key`.
+
+- **Generate a Certificate S igning Request (CSR)**:
+
+  Using the private key from the previous step, generate a CSR:
+
+  ```bash
+  openssl req -new -key ./src/banderx/private.key -out ./src/banderx/certificate.csr
+  ```
+
+  You will be prompted to enter details for the certificate; you can fill these out as needed or leave them blank.
+
+- **Self-Sign the Certificate**:
+
+  Sign the CSR with the private key, setting the certificate's validity period. For example, for a validity of 365 days:
+
+  ```bash
+  openssl x509 -req -days 180 -in ./src/banderx/certificate.csr -signkey ./src/banderx/private.key -out ../banderx/certificate.crt
+  ```
+
+  This creates a self-signed certificate `cert.crt` in the `banderx` directory.

src/banderx/nginx.conf

@@ -35,6 +35,45 @@ http {
         "~*text/html" ".html";
     }
 
+    # Uncomment the following lines to enable HTTPS
+    #server {
+    #    listen 443 ssl;
+    #    server_name banderx;
+    #    ssl_certificate /etc/ssl/certs/nginx.crt;
+    #    ssl_certificate_key /etc/ssl/private/nginx.key;
+    #    root /data/pypi/web;
+    #    autoindex on;
+    #    charset utf-8;
+
+    #    location /simple/ {
+    #        index index$mirror_suffix;
+    #        try_files $uri$mirror_suffix $uri $uri/ =404;
+    #    }
+
+    #    location /json/ {
+    #        default_type  application/json;
+    #    }
+
+    #    location /pypi/ {
+    #        default_type  application/json;
+    #    }
+
+    #    error_page 404 /404.html;
+    #    location = /404.html {
+    #    }
+
+    #    error_page 500 502 503 504 /50x.html;
+    #    location = /50x.html {
+    #    }
+    #}
+
+    # Uncomment the following lines to redirect HTTP to HTTPS
+    #server {
+    #    listen 80;
+    #    server_name banderx;
+    #    return 301 https://$host$request_uri;
+    #}
+
     server {
         listen       80 default_server;
         listen       [::]:80 default_server;