linters do not ignore `target/` directory after `make dev` #6852

cfm · 2023-06-16T00:53:23Z

Description

After #6828, make dev now yields a target/ folder, which fails several linters under make lint. Like Bandit in #6843, they should ignore target/.

Steps to Reproduce

make dev
make lint

Expected Behavior

Linters pass.

Actual Behavior

Linters fail on generated code in the target/ tree.

The text was updated successfully, but these errors were encountered:

legoktm · 2023-06-16T03:49:55Z

Which linters are failing for you?

user@dev ~/g/f/securedrop> USE_PODMAN=1 make dev
███ Starting development server...
STEP 1/19: FROM ubuntu@sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61dbd
STEP 2/19: ARG USER_NAME
--> Using cache b7805eaa383de86bc3bc04662dcbfc218a285842b198397d54e70703a1c02382
--> b7805eaa383d
STEP 3/19: ENV USER_NAME ${USER_NAME:-root}
--> Using cache a6eac2f0441d5d752c6586e42a62cf7014ae18e04c7757e686e21854b7616bec
--> a6eac2f0441d
STEP 4/19: ARG USER_ID
--> Using cache d8bce6cc088afca85316e4a785ae8720815d7ec1f426bb51e4ab057f4659ae0b
--> d8bce6cc088a
STEP 5/19: ENV USER_ID ${USER_ID:-0}
--> Using cache 19cd52cecb5fc37aceb38a473047466a2e6ffaf992a5cc7451490cc440a497ad
--> 19cd52cecb5f
STEP 6/19: RUN apt-get update && DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y                        apache2-dev coreutils vim                        python3-pip python3-all python3-venv virtualenv python3-dev libssl-dev                        gnupg2 redis-server git curl wget                        enchant libffi-dev sqlite3 gettext sudo tor basez pkg-config
--> Using cache 8a05d613c124c0e252f8d60d572a77bc02226af4cb5bcdeec727a79fa2b65690
--> 8a05d613c124
STEP 7/19: ENV RUST_VERSION 1.69.0
--> Using cache 8ca7ce23af91716b50edfbb7a0a94d1e24099862aef5e68c6c335e6713597d3b
--> 8ca7ce23af91
STEP 8/19: ENV RUSTUP_VERSION 1.24.3
--> Using cache fcd544c159b4d3b275d7bc1404d29c713ae1e639086d8ded6b9f004d3412be8d
--> fcd544c159b4
STEP 9/19: ENV RUSTUP_INIT_SHA256 3dc5ef50861ee18657f9db2eeb7392f9c2a6c95c90ab41e45ab4ca71476b4338
--> Using cache 96a267f2d15183493777a6d43692297ef5077aabc2757054200e1a1272103987
--> 96a267f2d151
STEP 10/19: ENV RUSTUP_HOME /opt/rustup
--> Using cache 5dd74e681a9ba18f421b1f985a7a4e0d51895fbc5f4eebc9de23e239408a9a60
--> 5dd74e681a9b
STEP 11/19: ENV CARGO_HOME /opt/cargo
--> Using cache 5b14b182b0b00c10812a84415ecac30135a33361127ee514ab085e8c842beaa9
--> 5b14b182b0b0
STEP 12/19: RUN TMPDIR=`mktemp -d` && cd ${TMPDIR}         && curl --proto '=https' --tlsv1.2 -OO -sSf https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/x86_64-unknown-linux-gnu/rustup-init         && echo "${RUSTUP_INIT_SHA256} *rustup-init" | sha256sum -c -         && chmod +x rustup-init         && ./rustup-init --default-toolchain=${RUST_VERSION} --profile minimal -y         && cd && rm -rf ${TMPDIR}
--> Using cache 1334bafc6c054106a0905cc358fa7db8aebbc36d9c344d32ebfe1a0c2a115d4b
--> 1334bafc6c05
STEP 13/19: ENV CARGO_REGISTRIES_CRATES_IO_PROTOCOL sparse
--> Using cache 850988c669c5f8bf17423228178626930aa654572598868168f3e82c5899bbdc
--> 850988c669c5
STEP 14/19: COPY requirements requirements
--> Using cache b22e2ea68d865b2a1ab3cbfe54e6048160bc720be46dd6132c774f8c4166f7be
--> b22e2ea68d86
STEP 15/19: RUN python3 -m venv /opt/venvs/securedrop-build &&     /opt/venvs/securedrop-build/bin/pip3 install --no-deps --require-hashes -r requirements/python3/build-requirements.txt
--> Using cache 0246bea5ac3095a6cd0be6fbb3ca1ae7a2d4f273d01caddfce8a726c7ae52a37
--> 0246bea5ac30
STEP 16/19: RUN python3 -m venv /opt/venvs/securedrop-app-code &&     /opt/venvs/securedrop-app-code/bin/pip3 install --no-deps --require-hashes -r requirements/python3/bootstrap-requirements.txt &&     /opt/venvs/securedrop-app-code/bin/pip3 install --no-deps --require-hashes -r requirements/python3/test-requirements.txt &&     /opt/venvs/securedrop-app-code/bin/pip3 install --no-deps --require-hashes -r requirements/python3/requirements.txt
--> Using cache 457fa9cabda0f5d8c355eb6da5fb6bc9714c3d0cfad257373bf3bf9d5423bd2c
--> 457fa9cabda0
STEP 17/19: RUN if test $USER_NAME != root ; then useradd --no-create-home --home-dir /tmp --uid $USER_ID $USER_NAME && echo "$USER_NAME ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers ; fi &&     chown -R $USER_NAME.$USER_NAME /opt/venvs/
--> Using cache 66985a954e11284c88841d952010ecb714a699959630efabfbafeb5a48363bff
--> 66985a954e11
STEP 18/19: STOPSIGNAL SIGKILL
--> Using cache 850681fde8d38bff7316fe83258d05929a5b9add743a835c07cecc2a48d94fce
--> 850681fde8d3
STEP 19/19: EXPOSE 8080 8081 5909
--> Using cache 8ee0bc24988c2ef88be5cd7a02f20f7bcc938eeee815d8b73a485a4ca3007c34
COMMIT securedrop-slim-focal-py3
--> 8ee0bc24988c
Successfully tagged localhost/securedrop-slim-focal-py3:latest
8ee0bc24988c2ef88be5cd7a02f20f7bcc938eeee815d8b73a485a4ca3007c34
************************************************************
Exposed services will be available on localhost at
Source interface: http://127.0.0.1:8080
Journalist interface: http://127.0.0.1:8081
************************************************************
rm: cannot remove '/dev/random': Device or resource busy
ln: failed to create symbolic link '/dev/random': File exists
Unable to replace /dev/random
🔗 Found pyo3 bindings
🐍 Found CPython 3.8 at /opt/venvs/securedrop-app-code/bin/python3
📡 Using build options compatibility from pyproject.toml
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
📖 Found type stub file at redwood.pyi
📦 Built wheel for CPython 3.8 to /home/user/github/freedomofpress/securedrop/target/dev/wheels/redwood-0.1.0-cp38-cp38-linux_x86_64.whl
Processing /home/user/github/freedomofpress/securedrop/target/dev/wheels/redwood-0.1.0-cp38-cp38-linux_x86_64.whl
Installing collected packages: redwood
Successfully installed redwood-0.1.0
WARNING: You are using pip version 21.1.1; however, version 23.1.2 is available.
You should consider upgrading via the '/opt/venvs/securedrop-app-code/bin/python3 -m pip install --upgrade pip' command.
main: /var/lib/securedrop/db.sqlite
^Cmake: *** [Makefile:239: dev] Error 2
(2) user@dev ~/g/f/securedrop> make lint
███ Linting Ansible configuration...
/bin/bash: line 1: molecule: command not found
make: *** [Makefile:92: ansible-config-lint] Error 127
(2) user@dev ~/g/f/securedrop> source .venv/bin/activate.fish
(.venv) user@dev ~/g/f/securedrop> make lint
███ Linting Ansible configuration...
--> Test matrix
    
└── ansible-config
    └── verify
    
--> Inventory /home/user/github/freedomofpress/securedrop/molecule/ansible-config/../../install_files/ansible-base/group_vars linked to /home/user/.cache/molecule/securedrop/ansible-config/inventory/group_vars
--> Inventory /home/user/github/freedomofpress/securedrop/molecule/ansible-config/../../install_files/ansible-base/host_vars linked to /home/user/.cache/molecule/securedrop/ansible-config/inventory/host_vars
--> Scenario: 'ansible-config'
--> Action: 'verify'
--> Executing Testinfra tests found in /home/user/github/freedomofpress/securedrop/molecule/ansible-config/tests/...
    ============================= test session starts ==============================
    platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-0.13.1
    rootdir: /home/user/github/freedomofpress/securedrop/molecule, configfile: pytest.ini
    plugins: testinfra-5.3.1, xdist-3.0.2, forked-1.3.0
collected 27 items                                                             
    
    tests/test_play_configuration.py ...........................             [100%]
    
    ============================== 27 passed in 0.79s ==============================
Verifier completed successfully.

███ Linting application code...

███ Running black check...
All done! ✨ 🍰 ✨
222 files would be left unchanged.

███ Running isort check...
Skipped 11 files

███ Running flake8...

███ Linting application templates...

███ Linting shell scripts...
ShellCheck - shell script analysis tool
version: 0.9.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net

███ Running mypy type checking...
./securedrop/bin/dev-shell: line 44: nc: command not found
🔗 Found pyo3 bindings
🐍 Found CPython 3.8 at /opt/venvs/securedrop-app-code/bin/python3
📡 Using build options compatibility from pyproject.toml
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
📖 Found type stub file at redwood.pyi
📦 Built wheel for CPython 3.8 to /home/user/github/freedomofpress/securedrop/target/dev/wheels/redwood-0.1.0-cp38-cp38-linux_x86_64.whl
Processing ./target/dev/wheels/redwood-0.1.0-cp38-cp38-linux_x86_64.whl
Installing collected packages: redwood
Successfully installed redwood-0.1.0
WARNING: You are using pip version 21.1.1; however, version 23.1.2 is available.
You should consider upgrading via the '/opt/venvs/securedrop-app-code/bin/python3 -m pip install --upgrade pip' command.
Success: no issues found in 170 source files

███ Linting YAML files...

cfm · 2023-06-20T23:39:57Z

Via both Docker and USE_PODMAN=1, including after a refreshing rm -rf .venv && make venv:

user@sd-dev:~/securedrop$ make dev
[...]
^C
(.venv) user@sd-dev:~/securedrop$ source .venv/bin/activate
(.venv) user@sd-dev:~/securedrop$ make lint
[...]
would reformat target/cargo-dev/registry/src/index.crates.io-6f17d22bba15001f/unicode-normalization-0.1.22/scripts/unicode.py

Oh no! 💥 💔 💥
3 files would be reformatted, 220 files would be left unchanged.
make: *** [Makefile:72: check-black] Error 1

legoktm · 2023-06-21T15:00:14Z

Ohh, black had already reformatted it so it didn't show up for me. Once I rm -rf target/ then it started failing again.

legoktm · 2023-06-21T16:07:16Z

Both black and isort have flags to automatically use rules defined in .gitignore, will submit a PR for that in a bit.

For black, setting `exclude` unintentionally overrode black's default of ignoring files listed in .gitignore. isort requires setting an explicit `skip_gitignore` option. Notably this will automatically have both tools ignore target/, which in some cases does contain Python code. Fixes #6852.

cfm added build process goals: improve developer workflow labels Jun 16, 2023

legoktm added the Rust Issues that touch Rust code label Jun 16, 2023

legoktm self-assigned this Jun 21, 2023

legoktm added this to the SecureDrop 2.7.0 milestone Jun 21, 2023

legoktm mentioned this issue Jun 22, 2023

Have black and isort respect .gitignore #6871

Merged

4 tasks

zenmonkeykstop closed this as completed in #6871 Jun 30, 2023

zenmonkeykstop added this to SecureDrop dev cycle Jun 30, 2023

zenmonkeykstop moved this to Done in SecureDrop dev cycle Jun 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

linters do not ignore `target/` directory after `make dev` #6852

linters do not ignore `target/` directory after `make dev` #6852

cfm commented Jun 16, 2023

legoktm commented Jun 16, 2023

cfm commented Jun 20, 2023 •

edited

Loading

legoktm commented Jun 21, 2023

legoktm commented Jun 21, 2023

linters do not ignore target/ directory after make dev #6852

linters do not ignore target/ directory after make dev #6852

Comments

cfm commented Jun 16, 2023

Description

Steps to Reproduce

Expected Behavior

Actual Behavior

legoktm commented Jun 16, 2023

cfm commented Jun 20, 2023 • edited Loading

legoktm commented Jun 21, 2023

legoktm commented Jun 21, 2023

linters do not ignore `target/` directory after `make dev` #6852

linters do not ignore `target/` directory after `make dev` #6852

cfm commented Jun 20, 2023 •

edited

Loading