Skip to content

Commit

Permalink
[PLAT-15279] Add gzip compression to core dumps from DB.
Browse files Browse the repository at this point in the history
Summary:
Add gzip compression to core dumps on DB nodes.
We are updating core pattern to contain a script that takes coredump on stdout and writes a gz compressed core dump to /home/yugabyte/cores directory.
Sample core

```
/home/yugabyte/cores/core.yb-master.47462.51733.gz
```

Test Plan:
Tested on the node by deploying on aws provider.

```
-rw-r--r--. 1 root   root   6.8M Sep 26 01:37 core.yb-master.186280.189968.gz
drwxr-xr-x. 6 yugabyte yugabyte  83 Sep 26 01:39 ..
drwxr-xr-x. 2 yugabyte yugabyte 273 Sep 26 01:40 .
[yugabyte@ip-10-9-71-23 cores]$ fie core.yb-master.186280.189968.gz^C
[yugabyte@ip-10-9-71-23 cores]$ file core.yb-master.186280.189968.gz
core.yb-master.186280.189968.gz: gzip compressed data, last modified: Thu Sep 26 01:37:51 2024, from Unix, original size 195506176
```
After unzip

```
-rw-r--r--. 1 yugabyte yugabyte 187M Sep 26 01:37 core.yb-master.186280.189968
```

Triggered a core on yb-master manually.
```
yugabyte@ip-10-9-75-132 ~]$ ps -ef | grep master
yugabyte   47462   30356  0 02:47 ?        00:00:01 /home/yugabyte/master/bin/yb-master --flagfile /home/yugabyte/master/conf/server.conf
yugabyte   51729   51683  0 02:51 pts/0    00:00:00 grep --color=auto master
[yugabyte@ip-10-9-75-132 ~]$ kill -11 47462
[yugabyte@ip-10-9-75-132 ~]$ ls -lahtr
total 6.6M
drwxr-xr-x. 6 yugabyte yugabyte   83 Sep 26 02:48 ..
drwxr-xr-x. 2 yugabyte yugabyte   43 Sep 26 02:51 .
-rw-r--r--. 1 root     root     6.6M Sep 26 02:51 core.yb-master.47462.51733.gz
[yugabyte@ip-10-9-75-132 ~]$ kill -11 47462
```

Reviewers: sanketh, svarshney, nbhatia, muthu

Reviewed By: svarshney, muthu

Subscribers: svc_phabricator, yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D38105
  • Loading branch information
amannijhawan committed Sep 26, 2024
1 parent 12b2c40 commit b1e6329
Show file tree
Hide file tree
Showing 2 changed files with 42 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@ pg_max_mem_mb: 0
install_retry_count: 5
vm_max_map_count: 262144
yb_metrics_dir: "{{ yb_home_dir }}/metrics"
bin_path: "/usr/local/bin"
42 changes: 41 additions & 1 deletion managed/devops/roles/provision-cluster-server/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,50 @@
state: present
tags: yb-prebuilt-ami

- name: Create directory for core dumps
file:
path: "{{ yb_home_dir }}/cores/"
state: directory
mode: '0755'
owner: "{{ user_name }}"
group: "{{ user_name }}"
tags: yb-prebuilt-ami

- name: Ensure gzip is installed
command: which gzip
register: gzip_installed
ignore_errors: yes
tags: yb-prebuilt-ami

- name: Fail if gzip is not installed
fail:
msg: "gzip is not installed, please install it manually."
when: gzip_installed.rc != 0
tags: yb-prebuilt-ami

- name: Create core dump compression script
become: yes
become_method: sudo
copy:
dest: "{{ bin_path }}/compress_core.sh"
mode: '0755'
content: |
#!/bin/bash
DUMP_DIR="/home/yugabyte/cores/"
CORE_FILE="$DUMP_DIR/$(basename $1).$$.gz"
GZIP_PATH=$(PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" which gzip)
if [[ -z "$GZIP_PATH" ]]; then
echo "gzip not found, aborting." >> /home/yugabyte/cores/custom_core_dump.log
exit 1
fi
/usr/bin/gzip >> "$CORE_FILE"
chmod 644 "$CORE_FILE"
tags: yb-prebuilt-ami

- name: Provision | Create core dump kernel pattern
sysctl:
name: kernel.core_pattern
value: "{{ yb_home_dir }}/cores/core_%p_%t_%E"
value: "|{{ bin_path }}/compress_core.sh {{ yb_home_dir }}/cores/cores/core.%e.%p"
state: present
tags: yb-prebuilt-ami

Expand Down

0 comments on commit b1e6329

Please sign in to comment.