Domain Scanner

[kshitijk4poor]

🚀 Implement domain validation and task addition for efficient scanning

📝 Description

This PR introduces two complementary methods in the ArtemisBase class: check_domain_exists for domain validation and add_valid_domains_task for adding tasks for valid domains. The primary purpose of these additions is to ensure that only valid domains are sent for further scanning, improving efficiency and reducing unnecessary processing.

🔧 Changes

• ✨ Added check_domain_exists and add_valid_domains_task method to ArtemisBase class
• 🔍 Implemented domain validation logic using NS and A record lookups
• 🔄 Created a method to filter and add tasks only for valid domains
• Fix mock setup in ArtemisModuleTestCase to handle multiple arguments in lookup function calls, resolving TypeError in tests.
  • This change is important because:
    • It fixes a breaking error in the test suite, allowing tests to run successfully.
    • It more accurately mimics the real lookup function's signature, improving test fidelity.
    • It prevents false negatives in tests that could mask real issues in the code being tested.
    • It ensures that the mock can handle various lookup calls throughout the codebase, increasing test coverage and reliability.

🌟 Benefits

• 🎯 Ensures only valid domains proceed to further scanning stages
• 🚀 Improves overall system efficiency by reducing unnecessary scans
• 🧩 Provides reusable domain validation logic across modules
• 🔄 Centralizes the process of adding tasks for valid domains
• 🛡️ Reduces potential false positives from invalid or non-existent domains

🤝 How the methods complement each other

1. check_domain_exists verifies the validity of a domain by checking for NS or A records
2. add_valid_domains_task uses check_domain_exists to filter out invalid domains
3. Only domains that pass the validation are added as new tasks for further scanning

This combination ensures a streamlined and efficient domain processing pipeline.

📝Testing

• Updated unit tests for the check_domain_exists method to ensure all scenarios are covered, including:
  • Domains with only NS records
  • Domains with only A records
  • Domains with both NS and A records
  • Non-existent domains
  • Scenarios where DNS resolution fails

TODO

• remove NS records for subdomains maybe? since most of them will inherit it from the parent domains and for subdomains only A records will be checked.

📚 Related Issues

Closes #1094

[kazet]

if we have open ports, we can be sure the domain exists ;) but checking in classifier is a good idea as well as it processes incoming tasks

[kazet]

waits for #1261 for the tests to pass