PMM-5680 store agents logs #197

qwest812 · 2022-04-02T15:29:30Z

PMM-5680

Build: SUBMODULES-2425

PMM-5680 store logs pmm-agent#342

BupycHuk · 2022-04-12T22:30:02Z

@qwest812 please make PR is ready for review if it's ready

# Conflicts: # go.mod # go.sum

github-actions · 2022-04-13T21:07:41Z

commands/summary.go

+			logrus.Debugf("%s", err)
+			b = []byte(err.Error())
+		}
+		b = append(b, '\n')


🚫 [golangci-lint] _{reported by reviewdog 🐶}
append only allowed to cuddle with appended value (wsl)

BupycHuk · 2022-04-18T23:36:05Z

commands/summary.go

+			b = []byte(err.Error())
+		}
+		b = append(b, '\n')
+		addData(zipW, "logs/"+pointer.GetString(agent.AgentType)+".json", now, bytes.NewReader(b))


what is the reason to make it json?
I think it would be much easier to read if it would be just a text with some separator.
something like

agent id: /agent_id/xxxx1 agent type: mongodb_exporter lorem ipsum log texts begin ... lorem ipsum log texts =============================================== agent id: /agent_id/xxxx2 agent type: mysqld_exporter lorem ipsum log texts begin ... lorem ipsum log texts

# Conflicts: # go.mod # go.sum

BupycHuk · 2022-05-03T14:29:09Z

commands/summary.go

+	fileName := "pmm-admin-summary.zip"
+	err = downloadFile(fmt.Sprintf("http://%s:%d/logs.zip", agentlocal.Localhost, agentlocal.DefaultPMMAgentListenPort), fileName)
+	if err != nil {
+		logrus.Debugf("%s", err)
+		b = []byte(err.Error())
+	}
+	logrus.Info("pmm-admin-summary.zip created")
+


we are on a right track, as a next step let's include files from logs.zip into main zip file

you can find example of it in addServerData

BupycHuk

LGTM, a few minor changes

BupycHuk · 2022-05-04T10:12:35Z

commands/summary.go

@@ -145,6 +143,11 @@ func addClientData(ctx context.Context, zipW *zip.Writer) {

 	addData(zipW, "client/pmm-admin-version.txt", now, bytes.NewReader([]byte(version.FullInfo())))

+	err = downloadFile(zipW, fmt.Sprintf("http://%s:%d/logs.zip", agentlocal.Localhost, agentlocal.DefaultPMMAgentListenPort), "pmm-admin logs")


Suggested change

err = downloadFile(zipW, fmt.Sprintf("http://%s:%d/logs.zip", agentlocal.Localhost, agentlocal.DefaultPMMAgentListenPort), "pmm-admin logs")

err = downloadFile(zipW, fmt.Sprintf("http://%s:%d/logs.zip", agentlocal.Localhost, agentlocal.DefaultPMMAgentListenPort), "pmm-agent")

BupycHuk · 2022-05-04T10:13:47Z

commands/summary.go

@@ -145,6 +143,11 @@ func addClientData(ctx context.Context, zipW *zip.Writer) {

 	addData(zipW, "client/pmm-admin-version.txt", now, bytes.NewReader([]byte(version.FullInfo())))

+	err = downloadFile(zipW, fmt.Sprintf("http://%s:%d/logs.zip", agentlocal.Localhost, agentlocal.DefaultPMMAgentListenPort), "pmm-admin logs")
+	if err != nil {
+		logrus.Debugf("%s", err)


let's write it in warn level

BupycHuk · 2022-05-04T10:14:30Z

commands/summary.go

@@ -220,6 +223,43 @@ func getURL(ctx context.Context, url string) ([]byte, error) {
 	return b, nil
 }

+// downloadFile download file to local destination


Suggested change

// downloadFile download file to local destination

// downloadFile download file and includes into zip file

BupycHuk · 2022-05-04T10:15:26Z

commands/summary.go

@@ -220,6 +223,43 @@ func getURL(ctx context.Context, url string) ([]byte, error) {
 	return b, nil
 }

+// downloadFile download file to local destination
+func downloadFile(zipW *zip.Writer, url, fileName string) error {
+


please remove this empty line, it makes CI fail

github-actions · 2022-05-04T10:24:05Z

commands/summary.go

+	if err != nil {
+		return err
+	}
+	defer response.Body.Close()


🚫 [golangci-lint] _{reported by reviewdog 🐶}
Error return value of response.Body.Close is not checked (errcheck)

github-actions · 2022-05-04T10:24:05Z

commands/summary.go

+		}
+		addData(zipW, path.Join(fileName, rf.Name), rf.Modified, rc)
+
+		rc.Close()


🚫 [golangci-lint] _{reported by reviewdog 🐶}
Error return value of rc.Close is not checked (errcheck)

github-actions · 2022-05-04T10:24:05Z

commands/summary.go

@@ -220,6 +223,42 @@ func getURL(ctx context.Context, url string) ([]byte, error) {
 	return b, nil
 }

+// downloadFile download file and includes into zip file
+func downloadFile(zipW *zip.Writer, url, fileName string) error {
+	response, err := http.Get(url)


🚫 [golangci-lint] _{reported by reviewdog 🐶}
G107: Potential HTTP request made with variable url (gosec)

github-actions · 2022-05-04T10:24:05Z

commands/summary.go

+			logrus.Errorf("%s", err)
+			continue
+		}
+		addData(zipW, path.Join(fileName, rf.Name), rf.Modified, rc)


🚫 [golangci-lint] _{reported by reviewdog 🐶}
G305: File traversal when extracting zip/tar archive (gosec)

github-actions · 2022-05-04T10:24:05Z

commands/summary.go

+	}
+	defer response.Body.Close()
+
+	if response.StatusCode != 200 {


🚫 [golangci-lint] _{reported by reviewdog 🐶}
mnd: Magic number: 200, in detected (gomnd)

github-actions · 2022-05-04T10:24:05Z

commands/summary.go

+		rc, err := rf.Open()
+		if err != nil {
+			logrus.Errorf("%s", err)
+			continue


🚫 [golangci-lint] _{reported by reviewdog 🐶}
continue with no blank line before (nlreturn)

commands/summary.go

github-actions · 2022-05-04T10:24:05Z

commands/summary.go

+func downloadFile(zipW *zip.Writer, url, fileName string) error {
+	response, err := http.Get(url)
+	if err != nil {
+		return err


🚫 [golangci-lint] _{reported by reviewdog 🐶}
error returned from external package is unwrapped: sig: func net/http.Get(url string) (resp *net/http.Response, err error) (wrapcheck)

PMM-5680 store agents logs

3bc9d37

qwest812 requested a review from BupycHuk as a code owner April 2, 2022 15:29

qwest812 marked this pull request as draft April 2, 2022 15:33

This was referenced Apr 2, 2022

PMM-5680 store logs Percona-Lab/pmm-submodules#2425

Closed

PMM-5680 store logs percona/pmm-agent#342

Open

PMM-5680 add log to struct AgentInfo percona/pmm#857

Closed

YaroslavPodorvanov approved these changes Apr 12, 2022

View reviewed changes

qwest812 mentioned this pull request Apr 13, 2022

PMM-5680 add log to struct AgentInfo percona/pmm#866

Closed

2 tasks

qwest812 added 2 commits April 14, 2022 00:04

Merge branch 'main' into PMM-5680-store-logs

2db5722

# Conflicts: # go.mod # go.sum

PMM-5680 store agents logs

e1df266

github-actions bot reviewed Apr 13, 2022

View reviewed changes

qwest812 added 2 commits April 14, 2022 00:13

PMM-5680 update dependency

1fc2698

PMM-5680 update dependency

df0f287

qwest812 mentioned this pull request Apr 14, 2022

PMM-5680 store logs percona/pmm#868

Closed

2 tasks

qwest812 marked this pull request as ready for review April 14, 2022 21:20

YaroslavPodorvanov approved these changes Apr 14, 2022

View reviewed changes

BupycHuk reviewed Apr 18, 2022

View reviewed changes

ShashankSinha252 self-requested a review April 21, 2022 10:45

qwest812 added 5 commits April 30, 2022 09:50

Merge branch 'main' into PMM-5680-store-logs

65cb23f

# Conflicts: # go.mod # go.sum

PMM-5680 create zip file with pmm-agent logs

1a9381c

Merge branch 'main' into PMM-5680-store-logs

4a86454

# Conflicts: # go.mod # go.sum

PMM-5680 pretty

ed34aaf

PMM-5680 remove TODO

1e5e1ab

qwest812 requested review from YaroslavPodorvanov and BupycHuk May 3, 2022 11:04

YaroslavPodorvanov approved these changes May 3, 2022

View reviewed changes

BupycHuk reviewed May 3, 2022

View reviewed changes

qwest812 added 3 commits May 3, 2022 22:28

PMM-5680 save logs to file zip

3d9236e

PMM-5680 pretty

253991c

change request

3b5020a

qwest812 requested a review from BupycHuk May 4, 2022 09:44

BupycHuk approved these changes May 4, 2022

View reviewed changes

BupycHuk reviewed May 4, 2022

View reviewed changes

fix bug

e23d761

github-actions bot reviewed May 4, 2022

View reviewed changes

Merge branch 'main' into PMM-5680-store-logs

f69bd1e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PMM-5680 store agents logs #197

PMM-5680 store agents logs #197

qwest812 commented Apr 2, 2022 •

edited

Loading

BupycHuk commented Apr 12, 2022

github-actions bot Apr 13, 2022

BupycHuk Apr 18, 2022 •

edited

Loading

BupycHuk May 3, 2022

BupycHuk May 3, 2022

qwest812 May 4, 2022

BupycHuk left a comment

BupycHuk May 4, 2022

BupycHuk May 4, 2022

BupycHuk May 4, 2022

BupycHuk May 4, 2022

github-actions bot May 4, 2022

github-actions bot May 4, 2022

github-actions bot May 4, 2022

github-actions bot May 4, 2022

github-actions bot May 4, 2022

github-actions bot May 4, 2022

github-actions bot May 4, 2022

		@@ -145,6 +143,11 @@ func addClientData(ctx context.Context, zipW *zip.Writer) {

		addData(zipW, "client/pmm-admin-version.txt", now, bytes.NewReader([]byte(version.FullInfo())))

		err = downloadFile(zipW, fmt.Sprintf("http://%s:%d/logs.zip", agentlocal.Localhost, agentlocal.DefaultPMMAgentListenPort), "pmm-admin logs")

	// downloadFile download file to local destination
	// downloadFile download file and includes into zip file

PMM-5680 store agents logs #197

Are you sure you want to change the base?

PMM-5680 store agents logs #197

Conversation

qwest812 commented Apr 2, 2022 • edited Loading

BupycHuk commented Apr 12, 2022

github-actions bot Apr 13, 2022

Choose a reason for hiding this comment

BupycHuk Apr 18, 2022 • edited Loading

Choose a reason for hiding this comment

BupycHuk May 3, 2022

Choose a reason for hiding this comment

BupycHuk May 3, 2022

Choose a reason for hiding this comment

qwest812 May 4, 2022

Choose a reason for hiding this comment

BupycHuk left a comment

Choose a reason for hiding this comment

BupycHuk May 4, 2022

Choose a reason for hiding this comment

BupycHuk May 4, 2022

Choose a reason for hiding this comment

BupycHuk May 4, 2022

Choose a reason for hiding this comment

BupycHuk May 4, 2022

Choose a reason for hiding this comment

github-actions bot May 4, 2022

Choose a reason for hiding this comment

github-actions bot May 4, 2022

Choose a reason for hiding this comment

github-actions bot May 4, 2022

Choose a reason for hiding this comment

github-actions bot May 4, 2022

Choose a reason for hiding this comment

github-actions bot May 4, 2022

Choose a reason for hiding this comment

github-actions bot May 4, 2022

Choose a reason for hiding this comment

github-actions bot May 4, 2022

Choose a reason for hiding this comment

qwest812 commented Apr 2, 2022 •

edited

Loading

BupycHuk Apr 18, 2022 •

edited

Loading