migrate `obs-infraobs-integrations` owned packages to `format_version: 3.0.0` #8028

tommyers-elastic · 2023-09-29T14:33:41Z

This issue captures all the work done and additional work required to migrate obs-infraobs-integrations owned packages to format_version 3.0.0.

In order to get some packages passing the new package-spec validations, several validation exclude rules were added. For these packages, the underlying issues should be fixed as soon as possible and the exclude rules removed. The fixes required for this are also captured in this issue.

The packages being migrated as part of this issue are:

PRs:

[apache_tomcat] - Migrate to package-spec v3 #8159
[salesforce] Update package to format_version 3.0.0 #8215
Migrate obs-infraobs-integrations to package-spec v3 #1 #8170 (activemq -> etcd)
Add dimension field previously missed during package-spec v3 migration #8260
Migrate obs-infraobs-integrations to package-spec v3 #2 #8171 (golang -> nginx)
Migrate obs-infraobs-integrations to package-spec v3 #3 #8203 (oracle -> zookeeper)
Migrate obs-infraobs-integrations to package-spec v3 #4 #8216 (leftovers - azure_app_service, citrix_adc, redis, tomcat)
Migrate obs-infraobs-integrations to package-spec v3 #5 #8234 (more leftovers - influxdb, mongodb)
Migrate obs-infraobs-integrations to package-spec v3 #6 #8291 (statsd_input, sql_input)
[memcached]: Address skipped validation SVR00003 #8347
elastic/obs-infraobs-integrations: Address skipped validation SVR00002 #8348

Issues blocking migration to format_version 3.0.0

[apache] convert legacy visualizations to lens #8209
[azure_app_service]: resolving external fields failed: can't resolve fields: field geo.continent_name cannot be reused at top level (see: #1)
[citrix_adc]: resolving external fields failed: can't resolve fields: field interface.id cannot be reused at top level
[haproxy] convert legacy visualizations to lens #8210
[IBMMQ]: [IBM MQ] Migrate Top 5 Errors [Logs IBM MQ] panel to the lens #5662
[redis]: resolving external fields failed: can't resolve fields: field os.full cannot be reused at top level
[tomcat]: resolving external fields failed: can't resolve fields: field geo.city_name cannot be reused at top level
[nagios_xi]: multiple set processors setting event.dataset in this file. (see: #1). Probable fix could be a breaking change.

Issues that need fixing but are currently skipped in 'validation.yml'

Package name	Validation skipped	Progress
activemq	SVR00004,SVR00002
airflow	SVR00002
apache_spark	SVR00004,SVR00002
apache_tomcat	SVR00004
azure_functions	SVR00004,SVR00002
cassandra	SVR00004,SVR00002
ceph	SVR00002
cockroachdb	SVR00002
coredns	SVR00002
couchbase	SVR00002
couchdb	SVR00002
etcd	SVR00002
golang	SVR00002
hadoop	SVR00004,SVR00002
iis	SVR00002,SVR00004
influxdb	SVR00002
kafka	SVR00002,SVR00004
memcached	SVR00003,SVR00002
microsoft_sqlserver	SVR00004,SVR00002
mongodb	SVR00004,SVR00002
mysql	SVR00002,SVR00004
nginx	SVR00004,SVR00002
oracle	SVR00002
oracle_weblogic	SVR00002,SVR00004
postgresql	SVR00001,SVR00002,SVR00004
rabbitmq	SVR00002
redisenterprise	SVR00002
salesforce	SVR00002
spring_boot	SVR00004,SVR00002
system (upgrade by different team)	SVR00004,SVR00002
traefik	SVR00004,SVR00002
vsphere	SVR00002
websphere_application_server	SVR00004,SVR00002
zookeeper	SVR00002
citrix_adc	SVR00002
redis	SVR00002,SVR00004

Other non-blocking issues that should be fixed in the future

[etcd]: While migrating we noticed the type {backend_commit_duration,wal_fsync_duration}.ns.{count,sum} is set to long but it should be float instead. (ref)
[azure_app_service]: geo mappings are not allowed at root level in ECS (ref)
[citrix_adc]: interface mappings are not allowed at root level in ECS (ref). Also, interface be nested like observer.egress.interface.*. Please refer ECS docs.
[redis]: os mappings are not allowed at root level in ECS (ref)
[tomcat]: geo mappings are not allowed at root level in ECS (ref)

The text was updated successfully, but these errors were encountered:

ishleenk17 · 2023-10-16T09:40:37Z

@shmsr : I know there are couple integrations where the TSDB fields are touched.
Would recommend to get TSDB testing done for some of those integrations to ensure there is no loss of data.

shmsr · 2023-10-16T09:52:15Z

@shmsr : I know there are couple integrations where the TSDB fields are touched. Would recommend to get TSDB testing done for some of those integrations to ensure there is no loss of data.

mongodb is the only package where I had to move dimension: true (for multiple datastreams) to ecs.yml to dedup the fields. Then I'll hold the PR until mongodb is tested properly.

tommyers-elastic · 2023-10-16T09:54:32Z

can we pull the mongodb changes into it's own PR so it doesn't block anything?

shmsr · 2023-10-16T19:12:09Z

Migrated 4 more packages. See: #8216

shmsr · 2023-10-16T19:47:03Z

Here's a script that'd make reviewing these PRs easier. Because of elastic-package test pipeline -g the *expected.json are modified (mostly with timestamp changes which is making the PR harder to review). See: #8203 (comment)

Here is something that might help:

Go to your browser's bookmark manager. Right click and create a new bookmark and paste the following script there and save it.

javascript:(function() {
  const headers = document.querySelectorAll('div[class*="file-header"][data-path*="log-expected.json"]');

  headers.forEach(header => {
    const checkbox = header.querySelector('.js-reviewed-checkbox');

    if (checkbox && !checkbox.checked) {
      checkbox.click();
    }
  });
})();

See the data-path in the above script? It contains the path of the file. *= means contains. You want to further modify the script; refer: https://api.jquery.com/category/selectors/attribute-selectors/
So in this script, I'm marking the files as viewed (when in GitHub PR Review) which contains the string log-expected.json in data-path attribute.
Now when doing PR review click on the bookmark, those files will get marked as viewed and it'd make it much easier for you to review the changes that matters.

Hope this helps.

cc: @ishleenk17 @tommyers-elastic

shmsr · 2023-10-18T10:03:00Z

Should I pull out mongodb and influxdb from #8171? As influxdb ci failure is blocking us; I think it better to pull it out from that PR.

cc: @tommyers-elastic

tommyers-elastic · 2023-10-18T10:20:13Z

yep sounds good - thanks!

shmsr · 2023-10-18T11:12:11Z

Should I pull out mongodb and influxdb from #8171? As influxdb ci failure is blocking us; I think it better to pull it out from that PR.

cc: @tommyers-elastic

Done. See: #8234

shmsr · 2023-10-18T16:21:45Z

@andresrc mentioned a tool that helps in transforming the by-reference visualizations into by-value panels. Here's what I found:

elastic/package-spec#316
elastic/kibana#129303 (related issue to the tool)
elastic/elastic-package#791 (related issue to the tool)

Repository that contains the tool: https://github.com/flash1293/legacy_vis_analyzer (need to explore; at first glance looks simple to use). With this tool, we'd hopefully be able to address SVR00004 (currently for 19 packages we have skipped this validation).

lalit-satapathy · 2023-10-19T05:04:04Z

Summarising below the beta in fields.yml for each of the packages as part of the v3 migration. This will help to discuss any package specific unique scenarios.

Package name	Data stream Name with Beta in fields.yml	Summary
hadoop	namenode	Since hadoop all data streams is made GA, we should just remove beta from fields.yml. @milan-elastic to re-confirm
influxdb	advstatus and status streams	Since the package is in beta, we dont need separate beta in fields.yml
mysql	performance and status	This one still beats data streams is showing beta. We can temporarily keep data streams aligned as to beats and track a backlog separately. Some further analysis is needed.
oracle	memory, performance, system_statistics and tablespace	@agithomas can you confirm which of the data streams of oracle need to be kept beta.
redisenterprise	node and proxy	Since the package is in beta, we can remove beta from fields.yml

agithomas · 2023-10-19T05:17:11Z

@agithomas can you confirm which of the data streams of oracle need to be kept beta.

I don't find there is any reason to keep it beta anymore considering Oracle Integration package is used by a few customers and there exist no SDHs.

shmsr · 2023-10-19T07:44:50Z

@lalit-satapathy For mysql, I see:

https://github.com/elastic/beats/blob/main/metricbeat/module/mysql/status/_meta/fields.yml#L5 (status is ga)
all other data streams are in beta

So to be consistent, marking them beta except status i.e., galera_status and performance

ali786XI · 2023-10-19T09:27:19Z

@shmsr Tested the TSDB related scenarios for MongoDB package in this PR and it seems to be working as expected.

milan-elastic · 2023-10-19T11:33:21Z

Since hadoop all data streams is made GA, we should just remove beta from fields.yml. @milan-elastic to re-confirm

Yes, I agree we should remove it since Hadoop is GA

drewdaemon · 2023-11-07T19:00:39Z

Repository that contains the tool: https://github.com/flash1293/legacy_vis_analyzer (need to explore; at first glance looks simple to use). With this tool, we'd hopefully be able to address SVR00004 (currently for 19 packages we have skipped this validation).

This is the old repository owned by a former Elastic employee. The maintained version is this fork.

We should be using that fork instead!

shmsr · 2023-11-08T07:15:04Z

As discussed in the last sprint, I am creating separate issues to track things not done yet as part of this meta issue.

Separate issue for "Issues blocking migration to format_version 3.0.0"
Separate issue for "Other non-blocking issues that should be fixed in the future"
One meta issue for all validations to address:
- Individual issue for each for validation: SVR0000{1,2,3,4,5} (optional; as per need)

After create the aforementioned issues I will close this issue. We will look at those specific issues for better trackability.

shmsr · 2023-11-09T19:17:50Z

As I have created separate issues for blocking and non-blocking issues, and skipped validations during package-spec upgrade; I am closing this issue. Please feel free to re-open the issue if any discussion is required.

agithomas · 2023-11-23T04:56:18Z

[etcd]: While migrating we noticed the type {backend_commit_duration,wal_fsync_duration}.ns.{count,sum} is set to long but it should be float instead. (ref)

@shmsr , the above mentioned fields will be changed to type histogram with this PR. I hope, this would solve the problem.

shmsr · 2023-11-23T07:13:55Z

[etcd]: While migrating we noticed the type {backend_commit_duration,wal_fsync_duration}.ns.{count,sum} is set to long but it should be float instead. (ref)

@shmsr , the above mentioned fields will be changed to type histogram with this PR. I hope, this would solve the problem.

Yep, looks good.

jsoriano · 2024-05-08T11:41:09Z

@tommyers-elastic please refresh my memory, why was a blocker for spec v3 to migrate dashboards to lens? I have tried to migrate the Apache package and it seems to be possible #9818

shmsr · 2024-05-08T17:48:29Z

@jsoriano While migrating packages to spec v3, we encountered issues with multiple packages, including Apache. Error example:

found legacy visualization "Uptime [Metrics Apache]" (metric, TSVB)

Find the actual error here. Noted down some errors while migrating to spec v3.

Later, this PR was opened to address the error above: #8173. After that, we did not try migrating to spec v3.

jsoriano · 2024-05-08T21:30:48Z

Later, this PR was opened to address the error above: #8173. After that, we did not try migrating to spec v3.

So #8209 could be already closed?

For the migration to v3, please take a look to #9818.

Thanks!

shmsr · 2024-05-09T17:08:18Z

Later, this PR was opened to address the error above: #8173. After that, we did not try migrating to spec v3.

So #8209 could be already closed?

For the migration to v3, please take a look to #9818.

Thanks!

Yes. Closed it. Thanks for the PR!

shmsr · 2024-05-31T10:17:41Z

The checklist in the issue description tracks what packages have been migrated or not from v1/v2 -> v3 format. There were some blockers earlier because of which the migration of some packages could not be done but most of them are hopefully unblocked now.

From the checklist, you see [BLOCKED] unticked boxes against package names that are yet to be migrated like haproxy, ibmmq, and nagios_xi. The input packages could not be migrated because we did not arrive at a conclusion then; see:
#8291 (comment)

tommyers-elastic added the Team:Service-Integrations Label for the Service Integrations team label Sep 29, 2023

tommyers-elastic assigned shmsr Oct 10, 2023

tommyers-elastic changed the title ~~infraobs-integrations package upgrades to format_version 3.0.0~~ migrate obs-infraobs-integrations owned packages to format_version: 3.0.0 Oct 16, 2023

tommyers-elastic mentioned this issue Oct 16, 2023

[apache_tomcat] - Migrate to package-spec v3 #8159

Merged

4 tasks

tommyers-elastic mentioned this issue Oct 16, 2023

Bulk package-spec v3 migration of packages for serverless #8147

Closed

shmsr mentioned this issue Oct 16, 2023

Migrate obs-infraobs-integrations to package-spec v3 #3 #8203

Merged

22 tasks

lalit-satapathy mentioned this issue Oct 19, 2023

Migrate obs-infraobs-integrations to package-spec v3 #2 #8171

Merged

19 tasks

shmsr mentioned this issue Oct 19, 2023

Migrate obs-infraobs-integrations to package-spec v3 #5 #8234

Merged

8 tasks

This was referenced Oct 31, 2023

[memcached]: Address skipped validation SVR00003 #8347

Closed

elastic/obs-infraobs-integrations: Address skipped validation SVR00002 #8348

Closed

shmsr closed this as completed Nov 9, 2023

shmsr mentioned this issue Nov 9, 2023

elastic/obs-infraobs-integrations: Address skipped validation SVR00004 #8423

Merged

14 tasks

shmsr mentioned this issue May 9, 2024

[Apache] Migrate package to spec v3 #9818

Merged

4 tasks

shmsr mentioned this issue Aug 28, 2024

{,x-pack/}metricbeat/module/prometheus/{collector,remote_write}: Add metrics_count to Prometheus module if metrics_count is true elastic/beats#40411

Merged

6 tasks

muthu-mps mentioned this issue Oct 17, 2024

Migrate integrations with package-spec v1/v2 to package-spec v3 #11449

Open

5 tasks

shmsr mentioned this issue Dec 3, 2024

packages/prometheus: Upgrade to package format_version v3 #11988

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate `obs-infraobs-integrations` owned packages to `format_version: 3.0.0` #8028

migrate `obs-infraobs-integrations` owned packages to `format_version: 3.0.0` #8028

tommyers-elastic commented Sep 29, 2023 •

edited by shmsr

Loading

ishleenk17 commented Oct 16, 2023

shmsr commented Oct 16, 2023

tommyers-elastic commented Oct 16, 2023

shmsr commented Oct 16, 2023

shmsr commented Oct 16, 2023 •

edited

Loading

shmsr commented Oct 18, 2023

tommyers-elastic commented Oct 18, 2023

shmsr commented Oct 18, 2023

shmsr commented Oct 18, 2023 •

edited

Loading

lalit-satapathy commented Oct 19, 2023

agithomas commented Oct 19, 2023

shmsr commented Oct 19, 2023 •

edited

Loading

ali786XI commented Oct 19, 2023 •

edited

Loading

milan-elastic commented Oct 19, 2023

drewdaemon commented Nov 7, 2023 •

edited

Loading

shmsr commented Nov 8, 2023 •

edited

Loading

shmsr commented Nov 9, 2023

agithomas commented Nov 23, 2023

shmsr commented Nov 23, 2023

jsoriano commented May 8, 2024

shmsr commented May 8, 2024 •

edited

Loading

jsoriano commented May 8, 2024

shmsr commented May 9, 2024

shmsr commented May 31, 2024

migrate obs-infraobs-integrations owned packages to format_version: 3.0.0 #8028

migrate obs-infraobs-integrations owned packages to format_version: 3.0.0 #8028

Comments

tommyers-elastic commented Sep 29, 2023 • edited by shmsr Loading

PRs:

Issues blocking migration to format_version 3.0.0

Issues that need fixing but are currently skipped in 'validation.yml'

Other non-blocking issues that should be fixed in the future

ishleenk17 commented Oct 16, 2023

shmsr commented Oct 16, 2023

tommyers-elastic commented Oct 16, 2023

shmsr commented Oct 16, 2023

shmsr commented Oct 16, 2023 • edited Loading

shmsr commented Oct 18, 2023

tommyers-elastic commented Oct 18, 2023

shmsr commented Oct 18, 2023

shmsr commented Oct 18, 2023 • edited Loading

lalit-satapathy commented Oct 19, 2023

agithomas commented Oct 19, 2023

shmsr commented Oct 19, 2023 • edited Loading

ali786XI commented Oct 19, 2023 • edited Loading

milan-elastic commented Oct 19, 2023

drewdaemon commented Nov 7, 2023 • edited Loading

shmsr commented Nov 8, 2023 • edited Loading

shmsr commented Nov 9, 2023

agithomas commented Nov 23, 2023

shmsr commented Nov 23, 2023

jsoriano commented May 8, 2024

shmsr commented May 8, 2024 • edited Loading

jsoriano commented May 8, 2024

shmsr commented May 9, 2024

shmsr commented May 31, 2024

migrate `obs-infraobs-integrations` owned packages to `format_version: 3.0.0` #8028

migrate `obs-infraobs-integrations` owned packages to `format_version: 3.0.0` #8028

tommyers-elastic commented Sep 29, 2023 •

edited by shmsr

Loading

shmsr commented Oct 16, 2023 •

edited

Loading

shmsr commented Oct 18, 2023 •

edited

Loading

shmsr commented Oct 19, 2023 •

edited

Loading

ali786XI commented Oct 19, 2023 •

edited

Loading

drewdaemon commented Nov 7, 2023 •

edited

Loading

shmsr commented Nov 8, 2023 •

edited

Loading

shmsr commented May 8, 2024 •

edited

Loading