Performance issue between version 1.11.0 and 1.13.0

[vinaykumar4s]

I see considerable Performance degradation between the versions 1.11.0 and 1.13.0. I do not know which section this issue belongs to so i am opening issue in opendistro-build section. I tried to sample data in both versions on the same machine and view the the dashboard. version 1.11.0 is loading very fast when compared with 1.13.0. The difference is lot. its in seconds.
i have not tested with actual data.

Open distro version 1.11.0

Open distro version 1.13.0

[bbarani]

@vinaykumar4s, Can you provide more details on your use case? It would be great if you can list the steps to better understand the root case of this issue.

[vinaykumar4s]

Hi @bbarani ,
I have currently dev server running opendistro 1.11.0 version on a single node. It was installed from tar.gz distribution of opendistro. I am running as independent process using /bin/elasticsearch. As far as data, we don’t have so much data and it’s not very heavy user base. Performance is good in that node. I don’t see any lags in that server. Now. I am trying to setup in stage env. So I thought of using latest version opendistro in so I can get all the features including reports. I have setup latest in stage env using yum package manager it is RHEL v8.

OS version:
NAME="Red Hat Enterprise Linux"
VERSION="8.2 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.2"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.2 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.2:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.2"

Reproduce steps:
Opendistro version 1.11.0

1. Download tar.gz version - 1.11.0
2. Extract distribution
3. Execute ./opendistro-tar-install.sh
4. Download kibana tar.gz version - 1.11.0
5. Extract kibana distribution
6. Execute ./bin/kibana
7. Go to local host localhost:5601
8. Username and password: admin:admin
9. Add sample data
10. Select Sample eCommerce orders
11. Open dashboard [eCommerce] Revenue Dashboard in dashboard section
12. In browser dev/inspect tools check the request loading timings.

Follow the same steps for version 1.13.0 or you can install using yum (https://opendistro.github.io/for-elasticsearch-docs/docs/install/rpm/). You can see the difference in loading pages. Not only dashboard, I see the differences in Discover mode also. Please note that I have not customized any of the internal configurations in both situations. I kept default confutations as it is.

On the side note, Can you tell me how can i download previous versions of tar.gz distributions? In the webisite i only see the link to download latest versions.

Thank you.

[mxmorin]

I've same issue in all the masters (3 nodes). These nodes are ingest nodes too and receive bulks.
No effect on data nodes only.

First peak in after the ugrade. Bulk operations failed with timeout and masters became instable.
Bulks have been redirected to data node and cpu usage decreased but still be more than previous version.

With hot_threads, we have this result.
Is it due to BouncyCastle upgrade ?

[bbarani]

Hi @bbarani ,
I have currently dev server running opendistro 1.11.0 version on a single node. It was installed from tar.gz distribution of opendistro. I am running as independent process using /bin/elasticsearch. As far as data, we don’t have so much data and it’s not very heavy user base. Performance is good in that node. I don’t see any lags in that server. Now. I am trying to setup in stage env. So I thought of using latest version opendistro in so I can get all the features including reports. I have setup latest in stage env using yum package manager it is RHEL v8.

OS version:
NAME="Red Hat Enterprise Linux"
VERSION="8.2 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.2"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.2 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.2:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.2"

Reproduce steps:
Opendistro version 1.11.0

1. Download tar.gz version - 1.11.0
2. Extract distribution
3. Execute ./opendistro-tar-install.sh
4. Download kibana tar.gz version - 1.11.0
5. Extract kibana distribution
6. Execute ./bin/kibana
7. Go to local host localhost:5601
8. Username and password: admin:admin
9. Add sample data
10. Select Sample eCommerce orders
11. Open dashboard [eCommerce] Revenue Dashboard in dashboard section
12. In browser dev/inspect tools check the request loading timings.

Follow the same steps for version 1.13.0 or you can install using yum (https://opendistro.github.io/for-elasticsearch-docs/docs/install/rpm/). You can see the difference in loading pages. Not only dashboard, I see the differences in Discover mode also. Please note that I have not customized any of the internal configurations in both situations. I kept default confutations as it is.

On the side note, Can you tell me how can i download previous versions of tar.gz distributions? In the webisite i only see the link to download latest versions.

Thank you.

@vinaykumar4s ,Thanks for the steps. We will look in to this issue and update our findings in this thread.

You can download the previous versions using the below cloudfront link (Just replace the version placeholder with the one you want to download).

https://d3g5vo6xdbdb9a.cloudfront.net/tarball/opendistro-elasticsearch/opendistroforelasticsearch-[version].tar.gz

[mxmorin]

I confirm performance issue since upgrade to 1.13 from 1.12 version.
Queries and bulks are very slow

We have 6 data nodes (32 cpu, 190 Go Ram, 1T disk) and 3 masters (4 CPU, 32 Go RAM) on Centos7
Bulks are sent to data nodes.

All was fine before ugrade

Is there a way to downgrade to 1.12 ?

[mxmorin]

In my staging env, i've restarted all nodes with security plugin disabled.
Cpu on master and response time decreased immediatly (0.500ms to 0.100ms for the same query via curl)

[bbarani]

@vinaykumar4s @mxmorin we have zeroed in on the root cause of this performance issue and currently testing our fix. We are working on pushing the fix in a patch release (by this week or next). You can track the progress here - #653

[bbarani]

@vinaykumar4s We have added a new page to download previous versions. Let me know if you have any questions.

[vinaykumar4s]

@bbarani thank you. I will wait for the patch and I will update my results after applying patch. Right now I have downgraded to 1.11.0 and everything great on 1.11.0.

[bbarani]

@vinaykumar4s @mxmorin The patch version (1.13.1) is out and you can download it now from downloads page. Please provide your feedback after testing it.

[vinaykumar4s]

@bbarani i confirm the issues is resolved in 1.13.1. Thank you for quick turnaround.

[vinaykumar4s]

I confirm the issues is resolved in 1.13.1. Thank you for quick turnaround.