3437 improve handling of slow archives #4090

ThomasBrady · 2023-12-13T19:40:20Z

Description

Resolves #3437

Exposes config to control number of retries when fetching archives
Adds metrics for retries and throughput of files downloaded
Logs what commands are run as part of GetRemoteFileWork

Checklist

Reviewed the contributing document
Rebased on top of master (no merge commits)
Ran clang-format v8.0.0 (via make format or the Visual Studio extension)
Compiles
Ran all tests
If change impacts performance, include supporting evidence per the performance document

SirTyson

Off to a good start! I think it may be useful to step up a level and do the metrics and timings in GetHistoryArchiveStateWork. What you could do is still set the number of retries for GetHistoryArchiveStateWork. In the else if (state == State::WORK_FAILURE && archive) branch after you see a failure, you can then update the metrics with the time and count. This would also avoid noisy metrics GetAndUnzipRemoteFileWork, as I think we're only interested in GetHistoryArchiveStateWork (I'm not sure on this point though and need to double check).

src/work/BasicWork.h

src/main/Config.h

src/test/test.cpp

docs/stellar-core_example.cfg

docs/metrics.md

src/catchup/CatchupWork.cpp

src/history/test/HistoryTests.cpp

src/historywork/GetRemoteFileWork.cpp

src/ledger/readme.md

src/historywork/GetRemoteFileWork.cpp

src/historywork/GetRemoteFileWork.h

docs/stellar-core_example.cfg

src/catchup/CatchupWork.cpp

src/historywork/GetRemoteFileWork.cpp

src/historywork/GetRemoteFileWork.h

src/historywork/GetRemoteFileWork.cpp

SirTyson · 2023-12-21T23:59:30Z

I’m not sure if the semantics of HISTORY_ARCHIVE_RETRY_LIMIT are quite right. Currently, if we try to download from an archive and it fails, we try another archive in a round robin fashion. I don’t know if the HISTORY_ARCHIVE_RETRY_LIMIT would be better as a config that determines how many times you try another archive before failing vs. trying to download from the same archive multiple times. Looking at the original issue, I don’t know if this config setting addresses the archive pattern variables that Nico originally mentioned. Sorry about that, I think I lost track of the original issue when going back and forth on the design.

I think what makes the most sense is to keep the metrics in this PR (which we definitely want), but move the config issues to a follow up issue where we can discuss a little more what we want. I agree with Marta’s DownloadWorkWithMetrics suggestion and we can keep the metrics moving. Again, sorry about getting confused on the issue.

SirTyson

Looking pretty close! A few minor nits, and I think another class needs to inherit from DownloadWorkWithMetrics.

SirTyson · 2024-01-02T22:48:57Z

src/historywork/DownloadWorkWithMetrics.cpp

@@ -0,0 +1,37 @@
+// Copyright 2015 Stellar Development Foundation and contributors. Licensed


Nit: Copyright should be 2024

Happy new year 🎉

SirTyson · 2024-01-03T00:01:45Z

src/historywork/DownloadWorkWithMetrics.h

@@ -0,0 +1,28 @@
+// Copyright 2015 Stellar Development Foundation and contributors. Licensed


Nit: update copyright

SirTyson · 2024-01-03T00:06:56Z

src/historywork/GetRemoteFileWork.h

@@ -4,7 +4,7 @@

 #pragma once

-#include "historywork/RunCommandWork.h"


This include should not have changed.

SirTyson · 2024-01-03T00:08:08Z

src/historywork/GetAndUnzipRemoteFileWork.h

@@ -13,7 +14,7 @@ namespace stellar
 class HistoryArchive;
 class GetRemoteFileWork;

-class GetAndUnzipRemoteFileWork : public Work


I think GetHistoryArchiveStateWork should also inherit from DownloadWorkWithMetrics.

SirTyson · 2024-01-03T17:39:11Z

src/historywork/GetHistoryArchiveStateWork.h


 namespace stellar
 {

 class HistoryArchive;
 class GetRemoteFileWork;

-class GetHistoryArchiveStateWork : public Work
+class GetHistoryArchiveStateWork : public DownloadWorkWithMetrics


Sorry, I didn't catch this initially, but it looks like there's an inconsistency here with the mBytesPerSecond metric. The issue is the meter assumes that the download finishes whenever the work that inherits DownloadWorkWithMetrics finishes. For GetHistoryArchiveStateWork, this is fine because the entire class is effectively a wrapper around GetRemoteFileWork and does no additional work. However, GetAndUnzipRemoteFileWork first downloads the file, then unzips it, then finishes. This means that even if the speed of the download is identical, files downloaded via GetAndUnzipRemoteFileWork will always appear to be slower because you count unzip time.

I think the easiest way to fix this is probably just to do your metering in GetRemoteFileWork. There's no non-history related work that uses this, so we can probably just do this metering directly in the class. A wrapper class like DownloadWithMetrics would be useful if GetRemoteFileWork was used for use cases outside of history downloads, but I don't see any. You can mark the bytesPerSecond value in the onSuccess function still. Since we never retry the work directly, you can mark the retry/failure meter in onFailureRaise.

Ah right, good catch on the unzip time being included in the denominator. Regarding moving the metrics to GetRemoteFileWork, it doesn't make sense to call the metric history.get.retry (as its not exactly a retry when incremented from a failing GetRemoteFileWork), perhaps history.get.failure?

SirTyson

Looks good to me! A nice clean change after all the back and forth.

marta-lokhova · 2024-01-03T20:01:28Z

r+ c81c740

ThomasBrady marked this pull request as draft December 13, 2023 19:40

ThomasBrady marked this pull request as ready for review December 13, 2023 21:50

ThomasBrady force-pushed the 3437-improve-handling-of-slow-archives branch from edb4496 to 0987437 Compare December 14, 2023 01:28

SirTyson requested changes Dec 14, 2023

View reviewed changes

marta-lokhova self-requested a review December 20, 2023 23:04

marta-lokhova requested changes Dec 21, 2023

View reviewed changes

ThomasBrady force-pushed the 3437-improve-handling-of-slow-archives branch from 0987437 to 39cc61c Compare December 21, 2023 19:07

ThomasBrady force-pushed the 3437-improve-handling-of-slow-archives branch from 39cc61c to bc1c430 Compare December 23, 2023 01:14

SirTyson requested changes Jan 3, 2024

View reviewed changes

ThomasBrady force-pushed the 3437-improve-handling-of-slow-archives branch 2 times, most recently from 0e39ed1 to 8bd904e Compare January 3, 2024 16:37

SirTyson requested changes Jan 3, 2024

View reviewed changes

Adds history.get.{failure,throughput} metrics to GetRemoteFileWork

c81c740

ThomasBrady force-pushed the 3437-improve-handling-of-slow-archives branch from 8bd904e to c81c740 Compare January 3, 2024 19:15

ThomasBrady requested a review from SirTyson January 3, 2024 19:16

SirTyson approved these changes Jan 3, 2024

View reviewed changes

latobarita merged commit 58697e8 into stellar:master Jan 3, 2024
15 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3437 improve handling of slow archives #4090

3437 improve handling of slow archives #4090

ThomasBrady commented Dec 13, 2023 •

edited

Loading

SirTyson left a comment

SirTyson commented Dec 21, 2023

SirTyson left a comment

SirTyson Jan 2, 2024

ThomasBrady Jan 3, 2024

SirTyson Jan 3, 2024

SirTyson Jan 3, 2024

SirTyson Jan 3, 2024

SirTyson Jan 3, 2024

ThomasBrady Jan 3, 2024 •

edited

Loading

SirTyson Jan 3, 2024

SirTyson left a comment

marta-lokhova commented Jan 3, 2024

		@@ -0,0 +1,37 @@
		// Copyright 2015 Stellar Development Foundation and contributors. Licensed

		@@ -0,0 +1,28 @@
		// Copyright 2015 Stellar Development Foundation and contributors. Licensed

		@@ -4,7 +4,7 @@

		#pragma once

		#include "historywork/RunCommandWork.h"

3437 improve handling of slow archives #4090

3437 improve handling of slow archives #4090

Conversation

ThomasBrady commented Dec 13, 2023 • edited Loading

Description

Checklist

SirTyson left a comment

Choose a reason for hiding this comment

SirTyson commented Dec 21, 2023

SirTyson left a comment

Choose a reason for hiding this comment

SirTyson Jan 2, 2024

Choose a reason for hiding this comment

ThomasBrady Jan 3, 2024

Choose a reason for hiding this comment

SirTyson Jan 3, 2024

Choose a reason for hiding this comment

SirTyson Jan 3, 2024

Choose a reason for hiding this comment

SirTyson Jan 3, 2024

Choose a reason for hiding this comment

SirTyson Jan 3, 2024

Choose a reason for hiding this comment

ThomasBrady Jan 3, 2024 • edited Loading

Choose a reason for hiding this comment

SirTyson Jan 3, 2024

Choose a reason for hiding this comment

SirTyson left a comment

Choose a reason for hiding this comment

marta-lokhova commented Jan 3, 2024

ThomasBrady commented Dec 13, 2023 •

edited

Loading

ThomasBrady Jan 3, 2024 •

edited

Loading