Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Integrations API endpoint /api/fleet/epm/packages returns different data randomly #166921

Closed
colin-stubbs opened this issue Sep 21, 2022 · 5 comments
Closed

[Fleet] Integrations API endpoint /api/fleet/epm/packages returns different data randomly #166921

colin-stubbs opened this issue Sep 21, 2022 · 5 comments
Labels
Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project stalled Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@colin-stubbs
Copy link

colin-stubbs commented Sep 21, 2022 •
edited
Loading

The Fleet API endpoint for integrations randomly returns different data regarding available integrations.

No apparent cause of even pattern is obvious.

URI example: /api/fleet/epm/packages?excludeInstallStatus=false

Response examples isolated to the 1password integration simply for sanity.

1password is installed on this deployment.

This request confirms it, 1.6.0 is available and we have it installed.

{
    "items": [
        {
            "name": "1password",
            "title": "1Password",
            "version": "1.6.0",
            "release": "ga",
            "description": "Collect logs from 1Password with Elastic Agent.",
            "type": "integration",
            "download": "/epr/1password/1password-1.6.0.zip",
            "path": "/package/1password/1.6.0",
            "icons": [
                {
                    "src": "/img/1password-logo-light-bg.svg",
                    "path": "/package/1password/1.6.0/img/1password-logo-light-bg.svg",
                    "title": "1Password",
                    "size": "116x116",
                    "type": "image/svg+xml"
                }
            ],
            "policy_templates": [
                {
                    "name": "1password",
                    "title": "1Password Events",
                    "description": "Collect events from 1Password Events Reporting"
                }
            ],
            "conditions": {
                "kibana": {
                    "version": "^7.16.0 || ^8.0.0"
                }
            },
            "owner": {
                "github": "elastic/security-external-integrations"
            },
            "categories": [
                "security"
            ],
            "id": "1password",
            "status": "installed",
            "savedObject": {
                "type": "epm-packages",
                "id": "1password",
                "attributes": {
                    "installed_kibana": [
                        {
                            "id": "1password-item-usages-full-dashboard",
                            "type": "dashboard"
                        },
                        {
                            "id": "1password-signin-attempts-full-dashboard",
                            "type": "dashboard"
                        },
                        {
                            "id": "1password-item-usages-hot-items",
                            "type": "visualization"
                        },
                        {
                            "id": "1password-item-usages-hot-users",
                            "type": "visualization"
                        },
                        {
                            "id": "1password-item-usages-hot-vaults",
                            "type": "visualization"
                        },
                        {
                            "id": "1password-item-usages-over-time",
                            "type": "visualization"
                        },
                        {
                            "id": "1password-signin-attempts-categories-over-time",
                            "type": "visualization"
                        },
                        {
                            "id": "1password-signin-attempts-count-over-time",
                            "type": "visualization"
                        },
                        {
                            "id": "1password-signin-attempts-failed-gauge",
                            "type": "visualization"
                        },
                        {
                            "id": "1password-signin-attempts-hot-users",
                            "type": "visualization"
                        },
                        {
                            "id": "1password-all-events",
                            "type": "search"
                        },
                        {
                            "id": "1password-item-usages",
                            "type": "search"
                        },
                        {
                            "id": "1password-signin-attempts",
                            "type": "search"
                        },
                        {
                            "id": "1password-item-usages-source-IPs-map",
                            "type": "map"
                        },
                        {
                            "id": "1password-signin-attempts-source-IPs-map",
                            "type": "map"
                        }
                    ],
                    "installed_es": [
                        {
                            "id": "logs-1password.item_usages-1.6.0",
                            "type": "ingest_pipeline"
                        },
                        {
                            "id": "logs-1password.signin_attempts-1.6.0",
                            "type": "ingest_pipeline"
                        },
                        {
                            "id": "logs-1password.item_usages",
                            "type": "index_template"
                        },
                        {
                            "id": "logs-1password.item_usages@package",
                            "type": "component_template"
                        },
                        {
                            "id": "logs-1password.item_usages@custom",
                            "type": "component_template"
                        },
                        {
                            "id": "logs-1password.signin_attempts",
                            "type": "index_template"
                        },
                        {
                            "id": "logs-1password.signin_attempts@package",
                            "type": "component_template"
                        },
                        {
                            "id": "logs-1password.signin_attempts@custom",
                            "type": "component_template"
                        }
                    ],
                    "package_assets": [
                        {
                            "id": "2e1894db-44c4-5eab-9b18-4e6cb3d565ff",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "ee0c8ec4-cf8a-5a90-a046-dd66c1f6674b",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "c3022fba-f79d-5630-950b-bf411aa092bb",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "b30e9285-ef4c-5bd9-bb22-778ff095647f",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "9ad3c493-c7c8-5531-8357-80618edb1162",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "00332687-bff6-5e12-a180-07d56f122419",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "d6579330-2483-5472-82c9-b5610e213c65",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "82b4a885-73b9-56e6-b4cb-fd7718873ce5",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "eaa1a2c1-57f4-5df9-8347-72485e18059f",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "2d820c2d-64ac-5391-92ed-3ae203812c72",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "7d9001df-aec4-5004-9087-fa928b272c10",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "42a852e8-04ca-5300-9238-41aa8f67f39d",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "a2b7f136-f382-5739-ac43-016a69543a8c",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "59f844e0-6cfc-5c20-bce9-3da202ee7bfe",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "d9bd0a15-6b8c-5914-8e79-1e657b12f024",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "f9972506-d91e-5c91-a3cc-8c73ff983eaf",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "4b3c2def-025b-5d9d-8e09-77d62406b6a4",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "a19f4f86-21ed-53ba-99c5-d3e91d8ee4ed",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "b9aaeb91-bf4d-5dc8-b7c3-a05e0ca9be9b",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "c25efa10-7748-52b6-8c1d-f17d34d522f3",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "5f3b8766-d8a2-5975-a487-d7b1bae8f4f5",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "afd978bf-15ef-559c-92f6-8d5be0baf841",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "5757ddda-7b71-58d6-91b7-fb7c387b4b6b",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "fd32c195-db0e-5ed8-8211-e059ed1b61f8",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "88a64970-ad29-5f17-a55f-eab513a6eddd",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "8599f2dc-f004-555d-bf6e-44aa97ec96fb",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "6c832ea2-4d44-503b-bce9-94dc88bb42e4",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "49bfa8f3-04bb-5330-9b07-545704f7093d",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "a0cbcbe7-3a9b-5e77-ae01-c10da50af42f",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "a117f63f-791f-51c4-9f9d-224a6cc8c5d3",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "6ffe8bc5-586a-5ea2-830e-46c9eb8fee0a",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "b7bd6020-ee1c-5fcb-b426-5b265a8eddc0",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "2a4a5169-1140-507c-a838-097f6b94ead4",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "042cb8ea-bd8b-55fc-8cd7-88b15f1a03a0",
                            "type": "epm-packages-assets"
                        },
                        {
                            "id": "e3d956ac-be32-56bc-91f2-2aa31aece8f8",
                            "type": "epm-packages-assets"
                        }
                    ],
                    "es_index_patterns": {
                        "item_usages": "logs-1password.item_usages-*",
                        "signin_attempts": "logs-1password.signin_attempts-*"
                    },
                    "name": "1password",
                    "version": "1.6.0",
                    "internal": false,
                    "install_version": "1.6.0",
                    "install_status": "installed",
                    "install_started_at": "2022-09-14T00:04:17.399Z",
                    "install_source": "registry",
                    "keep_policies_up_to_date": false
                },
                "references": [],
                "migrationVersion": {
                    "epm-packages": "8.3.0"
                },
                "coreMigrationVersion": "8.3.2",
                "updated_at": "2022-09-14T00:04:34.874Z",
                "version": "WzEzNzEyMjU4OCw0XQ==",
                "score": 0
            }
        },

But a subsequent request to the same endpoint with the parameters returns this, where it claims that 1password is NOT installed even though it is.

{
    "items": [
        {
            "name": "1password",
            "title": "1Password",
            "version": "1.6.0",
            "release": "ga",
            "description": "Collect logs from 1Password with Elastic Agent.",
            "type": "integration",
            "download": "/epr/1password/1password-1.6.0.zip",
            "path": "/package/1password/1.6.0",
            "icons": [
                {
                    "src": "/img/1password-logo-light-bg.svg",
                    "path": "/package/1password/1.6.0/img/1password-logo-light-bg.svg",
                    "title": "1Password",
                    "size": "116x116",
                    "type": "image/svg+xml"
                }
            ],
            "policy_templates": [
                {
                    "name": "1password",
                    "title": "1Password Events",
                    "description": "Collect events from 1Password Events Reporting"
                }
            ],
            "conditions": {
                "kibana": {
                    "version": "^7.16.0 || ^8.0.0"
                }
            },
            "owner": {
                "github": "elastic/security-external-integrations"
            },
            "categories": [
                "security"
            ],
            "id": "1password",
            "status": "not_installed"
        },

This also results in the Kibana webUI randomly listing integrations as available for upgrade, then not listing them on next page refresh/navigation event.

We are on version 8.3.2, this issue has existed in all 8.x versions that we have used, I am only just now working out what the actual cause is at this API level.
@botelastic
Copy link

botelastic bot commented Sep 21, 2023

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

@botelastic botelastic bot added the stalled label Sep 21, 2023
@andrewkroh andrewkroh transferred this issue from elastic/integrations Sep 21, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Sep 21, 2023
@andrewkroh andrewkroh added Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project and removed needs-team Issues missing a team label labels Sep 21, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Feature:EPM)

@botelastic botelastic bot added the needs-team Issues missing a team label label Sep 21, 2023
@jughosta jughosta added the Team:Fleet Team label for Observability Data Collection Fleet team label Oct 4, 2023
@botelastic botelastic bot removed the needs-team Issues missing a team label label Oct 4, 2023
@jen-huang jen-huang changed the title Fleet API integrations endpoint /api/fleet/epm/packages returns different data randomly [Fleet] Integrations API endpoint /api/fleet/epm/packages returns different data randomly Nov 8, 2023
@criamico
Copy link
Contributor

criamico commented Nov 9, 2023

One thing that comes to mind is that in 8.6.0 we introduced a change in the way we get the information from registry or archive (PR), so it would be interesting to know if it still happens in newer versions.

@amolnater-qasource could you try reproducing this bug on latest version? We are not sure if it's still occurring.

@amolnater-qasource
Copy link

Hi @criamico

We have revalidated this issue on latest 8.11.0 and 8.12.0 SNAPSHOT kibana cloud environment and found it not reproducible there.

Observations:

  • Integration remains installed in the API request api/fleet/epm/packages?excludeInstallStatus=false and no change is observed on multiple attempts.

Screen Recording:
8.11.0:

r.kb.europe-west1.gcp.cloud.es.io_9243_api_fleet_epm_packages_excludeInstallStatus.false.-.Google.Chrome.2023-11-10.15-19-37.mp4

8.12.0:

s.kb.us-west2.gcp.elastic-cloud.com_9243_api_fleet_epm_packages_excludeInstallStatus.false.-.Google.Chrome.2023-11-10.15-39-45.mp4

Build details:
8.12.0 SNAPSHOT
BUILD: 68947
COMMIT: 6bb46fd

Please let us know if anything else is required from our end.
Thanks!!

@jlind23
Copy link
Contributor

jlind23 commented Nov 10, 2023

Thanks @amolnater-qasource and @criamico for chiming in. Closing this as not reproducible then.

@jlind23 jlind23 closed this as not planned Won't fix, can't repro, duplicate, stale Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project stalled Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jughosta @colin-stubbs @andrewkroh @elasticmachine @criamico @jlind23 @amolnater-qasource