Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Directly download commercial ip geolocation databases from providers #110844

Merged
merged 79 commits into from
Jul 18, 2024
Merged

Directly download commercial ip geolocation databases from providers #110844

merged 79 commits into from
Jul 18, 2024

Commits on Jul 8, 2024

  1. A small tidiness refactor of the GeoIpTaskState's Metadata (elastic#1… 

    …10553)
    @joegallo
    joegallo committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    64fbdf4 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. Adding a unit test for GeoIpDownloader.cleanDatabases (elastic#110650) ( 

    elastic#110675)

Co-authored-by: Joe Gallo <[email protected]>
    @masseyke @joegallo
    masseyke and joegallo committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    a86555c View commit details
    Browse the repository at this point in the history

  2. Removing the use of Stream::peek from GeoIpDownloader::cleanDatabases (… 

    …elastic#110666) (elastic#110679)
    @masseyke @joegallo
    masseyke authored and joegallo committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    f9e0607 View commit details
    Browse the repository at this point in the history

  3. WIP squashed history of dev work so far

    @joegallo
    joegallo committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    c62f63b View commit details
    Browse the repository at this point in the history

  4. Rename this class

    @joegallo
    joegallo committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    87bba46 View commit details
    Browse the repository at this point in the history

  5. Tweak these constants to better shape the state 

    Some cluster state customs have singular nouns and some have plural,
and I think plural is more natural for this case.
    @joegallo
    joegallo committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    8f86909 View commit details
    Browse the repository at this point in the history

  6. Make it possible to override the endpoint for tests

    @joegallo
    joegallo committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    29f9db2 View commit details
    Browse the repository at this point in the history

  7. Log and skip if there are no credentials

    @joegallo
    joegallo committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    a0e009e View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. Updating EnterpriseGeoIpDownloaderLicenseListener to not start task i… 

    …f no database metadata is present
    @masseyke
    masseyke committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    fc8dd0c View commit details
    Browse the repository at this point in the history

  2. Schedule the next run at the very beginning of the run 

    to work around a race condition on the first run (when there is no
next run already scheduled).
    @joegallo
    joegallo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    4ae4171 View commit details
    Browse the repository at this point in the history

  3. Guard these contains operations for the no-change case

    @joegallo
    joegallo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    8c447c6 View commit details
    Browse the repository at this point in the history

  4. Adding EnterpriseGeoIpTaskStateSerializationTests

    @masseyke
    masseyke committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    18ec1e4 View commit details
    Browse the repository at this point in the history

  5. Remove the conflict between the two geoip downloaders

    @joegallo
    joegallo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    ec64555 View commit details
    Browse the repository at this point in the history

  6. Rename this setting and the associated constant 

    From ingest.geoip.downloader.maxmind.default.license_key to just
ingest.geoip.downloader.maxmind.license_key.
    @joegallo
    joegallo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    7110f42 View commit details
    Browse the repository at this point in the history

  7. Address todos and comments, and tidy up logging

    @joegallo
    joegallo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    3102a7a View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. Bump to Priority.NORMAL and passed hallway review

    @joegallo
    joegallo committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    dd72464 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'enterprise-downloader' of github.com:joegallo/elasticse… 

    …arch into enterprise-downloader
    @masseyke
    masseyke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    b5e44b2 View commit details
    Browse the repository at this point in the history

  3. Adding IngestGeoIpMetadataTests (and fixing a bug it found)

    @masseyke
    masseyke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    102db55 View commit details
    Browse the repository at this point in the history

  4. This reads slightly better in terms of intent

    @joegallo
    joegallo committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    e921255 View commit details
    Browse the repository at this point in the history

  5. Switch from a setting to a field for the account_id 

    In this version Maxmind is very much a hardcoded configuration -- I
think we might want a NamedWriteable here rather than merely a
Writeable, but this allows us to keep making progress.
    @joegallo
    joegallo committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    8adcabc View commit details
    Browse the repository at this point in the history

  6. Adding yaml rest tests

    @masseyke
    masseyke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    528a0a4 View commit details
    Browse the repository at this point in the history

  7. Adjust the new NodeFeature slightly

    @joegallo
    joegallo committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    ad55ec3 View commit details
    Browse the repository at this point in the history

  8. Collapse the two versions and add an alias

    @joegallo
    joegallo committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    1d05673 View commit details
    Browse the repository at this point in the history

  9. beginnings of a unit test for EnterpriseGeoIpDownloaderTests

    @masseyke
    masseyke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    8d5518a View commit details
    Browse the repository at this point in the history

  10. Override the default maxmind endpoint for the yaml tests

    @joegallo
    joegallo committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    38aa680 View commit details
    Browse the repository at this point in the history

  11. Merge remote-tracking branch 'masseyke/enterprise-downloader' into en… 

    …terprise-downloader
    @joegallo
    joegallo committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    36e3060 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'main' into enterprise-downloader

    @joegallo
    joegallo committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    b4ef611 View commit details
    Browse the repository at this point in the history

  13. Update docs/changelog/110844.yaml

    @joegallo
    joegallo committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    9178de4 View commit details
    Browse the repository at this point in the history

  14. These are non-operator actions

    @joegallo
    joegallo committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    d3c6172 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. Moving NodeFeature and FeatureSpecification to server to avoid build … 

    …errors
    @masseyke
    masseyke committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    e02b780 View commit details
    Browse the repository at this point in the history

  2. adding EnterpriseGeoIpDownloaderIT

    @masseyke
    masseyke committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    4cb5000 View commit details
    Browse the repository at this point in the history

  3. Enhance this test 

    This is not necessary for this PR, but we had a failure here and I've
always hated the failure messages from this class because it makes me
compare the lists of files myself and do the mental accounting of
what's right and what's wrong, etc. Anyway, using `assertThat` gives
us a nicer error message about exactly what was wrong with the
returned key set.
    @joegallo
    joegallo committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    c90abf3 View commit details
    Browse the repository at this point in the history

  4. cleaning up EnterpriseGeoIpDownloaderIT

    @masseyke
    masseyke committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    cd8ac3f View commit details
    Browse the repository at this point in the history

  5. Merge branch 'enterprise-downloader' of github.com:joegallo/elasticse… 

    …arch into enterprise-downloader
    @masseyke
    masseyke committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    0106544 View commit details
    Browse the repository at this point in the history

  6. Fix a typo

    @joegallo
    joegallo committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    9b461ed View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into enterprise-downloader

    @joegallo
    joegallo committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    494d653 View commit details
    Browse the repository at this point in the history

  8. creating a new EnterpriseGeoIpHttpFixture

    @masseyke
    masseyke committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    5eca85b View commit details
    Browse the repository at this point in the history

  9. merging

    @masseyke
    masseyke committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    2385afa View commit details
    Browse the repository at this point in the history

  10. removing enterprise logic from standard geo fixture

    @masseyke
    masseyke committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    d08cc63 View commit details
    Browse the repository at this point in the history

  11. removing whitespace change

    @masseyke
    masseyke committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    ba56d3d View commit details
    Browse the repository at this point in the history

  12. Drop unnecessary scope

    @joegallo
    joegallo committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    0110dac View commit details
    Browse the repository at this point in the history

  13. Add a comment about why the ReindexPlugin is necessary

    @joegallo
    joegallo committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    97afb24 View commit details
    Browse the repository at this point in the history

  14. Remove a comment

    @joegallo
    joegallo committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    a5906b3 View commit details
    Browse the repository at this point in the history

  15. Rename this variable

    @joegallo
    joegallo committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    314bd25 View commit details
    Browse the repository at this point in the history

  16. Drop this unnecessary class

    @joegallo
    joegallo committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    daa7759 View commit details
    Browse the repository at this point in the history

  17. Reword this comment

    @joegallo
    joegallo committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    8145a58 View commit details
    Browse the repository at this point in the history

  18. small changes to EnterpriseDownloaderPlugin and EnterpriseGeoIpDownlo… 

    …aderLicenseListener for code review feedback
    @masseyke
    masseyke committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    4b0649a View commit details
    Browse the repository at this point in the history

  19. fixing javadoc link

    @masseyke
    masseyke committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    8b15c67 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. undoing accidental revert of commeent change

    @masseyke
    masseyke committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    e53e38b View commit details
    Browse the repository at this point in the history

  2. Improving and testing database configuration validation

    @masseyke
    masseyke committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    932b52d View commit details
    Browse the repository at this point in the history

  3. Adding TransportPutDatabaseConfigurationActionTests

    @masseyke
    masseyke committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    4394dce View commit details
    Browse the repository at this point in the history

  4. cleaning up and commenting EnterpriseGeoIpHttpFixture

    @masseyke
    masseyke committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    07f0864 View commit details
    Browse the repository at this point in the history

  5. fixing EnterpriseGeoIpTaskStates minimalSupportedVersion

    @masseyke
    masseyke committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    2822d43 View commit details
    Browse the repository at this point in the history

  6. avoiding keeping a collection of Map.Entry values in DatabaseNodeService

    @masseyke
    masseyke committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    14195e2 View commit details
    Browse the repository at this point in the history

  7. Cleanup of EnterpriseGeoIpDownloader -- adding comments, adding asser… 

    …tion messages, and incorporating code review feedback
    @masseyke
    masseyke committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    225513a View commit details
    Browse the repository at this point in the history

  8. renaming GeoIpTaskState.isValid() to isNewEnough()

    @masseyke
    masseyke committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    57997d2 View commit details
    Browse the repository at this point in the history

  9. Allow unknown fields on this parser

    @joegallo
    joegallo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    f9dc894 View commit details
    Browse the repository at this point in the history

  10. Add a javadoc for PasswordAuthenticationHolder

    @joegallo
    joegallo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    77a76ae View commit details
    Browse the repository at this point in the history

  11. Remove unused method

    @joegallo
    joegallo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    dcbb249 View commit details
    Browse the repository at this point in the history

  12. Simplify serialization and deserialization

    @joegallo
    joegallo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    c48f91b View commit details
    Browse the repository at this point in the history

  13. Rewrite the get logic slightly

    @joegallo
    joegallo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    8d43e18 View commit details
    Browse the repository at this point in the history

  14. Use the correct transport version here

    @joegallo
    joegallo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    552eef7 View commit details
    Browse the repository at this point in the history

  15. Handle deletes slightly differently

    @joegallo
    joegallo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    2b56371 View commit details
    Browse the repository at this point in the history

  16. Don't check for no-ops outside the cluster state thread

    @joegallo
    joegallo committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    bc7e4eb View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. Remove my comments about this logic being mistaken 

    It's still true that I do think the code is mistaken, but I'll move
that notion into a github issue and link it for followup.
    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    43cfe46 View commit details
    Browse the repository at this point in the history

  2. Switch to Tuples here, too

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    e8b4f36 View commit details
    Browse the repository at this point in the history

  3. Only set the flag if we actually added something 

    That is, don't consider a skip-due-to-credentials to be 'added
something'.
    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    404bb01 View commit details
    Browse the repository at this point in the history

  4. Drop the copyOf and simplify this slightly

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    39416f0 View commit details
    Browse the repository at this point in the history

  5. Add back two missing early returns

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    4fc4f46 View commit details
    Browse the repository at this point in the history

  6. Expand javadoc slightly

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    3f3a7c0 View commit details
    Browse the repository at this point in the history

  7. Don't handle the empty state case specially

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    90a749c View commit details
    Browse the repository at this point in the history

  8. No treble

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    9f54396 View commit details
    Browse the repository at this point in the history

  9. Extend the fixture to cover an edge case

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    e9e2c1c View commit details
    Browse the repository at this point in the history

  10. Merge branch 'main' into enterprise-downloader

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    234a166 View commit details
    Browse the repository at this point in the history

  11. Switch exception types and messages

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    572991a View commit details
    Browse the repository at this point in the history

  12. Rename inner variable

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    31af5d6 View commit details
    Browse the repository at this point in the history

  13. Use a record instead of a Tuple

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    12655ed View commit details
    Browse the repository at this point in the history

  14. Wrap and rethrow rather than logging

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    95a820b View commit details
    Browse the repository at this point in the history

  15. Merge branch 'main' into enterprise-downloader

    @joegallo
    joegallo committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    22c93bd View commit details
    Browse the repository at this point in the history