Skip to content
This repository has been archived by the owner on Mar 9, 2023. It is now read-only.

Failed messages: Can't manage allocations (223 over 10 hrs) #253

Open
davaymne opened this issue Dec 11, 2020 · 8 comments
Open

Failed messages: Can't manage allocations (223 over 10 hrs) #253

davaymne opened this issue Dec 11, 2020 · 8 comments

Comments

@davaymne
Copy link

[2020-12-11 00:01:51.485 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 00:01:51.485 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 00:02:49.591 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 00:11:14.347 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 00:11:46.617 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 00:50:44.509 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 01:00:15.978 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 01:00:15.979 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 01:01:14.347 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 01:02:14.340 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 01:03:14.341 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 01:34:44.638 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 01:37:49.425 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 01:37:49.425 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:02:14.350 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:02:15.745 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 02:02:15.745 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:03:14.371 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:03:48.737 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 02:03:48.738 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:04:14.345 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:05:14.346 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:06:14.346 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:06:15.745 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 02:06:15.746 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:07:14.352 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:08:14.342 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:08:15.760 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 02:08:15.761 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:09:14.345 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:10:14.349 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:10:15.763 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 02:10:15.763 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:11:14.342 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:12:14.355 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:12:44.627 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 02:12:44.627 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:13:14.348 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:14:17.391 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 02:14:17.391 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:15:14.345 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:16:15.753 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 02:16:15.753 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:18:14.380 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:18:15.753 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 02:18:15.753 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:38:17.790 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 02:38:17.790 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:40:14.426 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:40:15.826 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 02:40:15.826 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:41:14.424 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 02:42:15.827 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 02:42:15.828 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 02:44:15.826 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 02:44:15.826 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:00:14.344 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:00:17.343 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 03:00:17.344 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:04:14.345 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:05:14.350 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:08:25.157 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 03:08:25.157 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:12:14.364 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:12:15.764 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 03:12:15.764 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:13:14.373 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:13:53.703 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 03:13:53.704 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:14:14.354 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:15:14.350 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:18:16.983 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 03:18:16.983 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:19:14.351 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:20:14.361 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:20:42.472 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 03:20:42.472 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:21:14.351 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:22:14.356 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:22:15.802 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 03:22:15.802 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:24:14.360 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:24:42.826 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 03:24:42.826 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:26:14.364 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:26:38.426 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 03:26:38.426 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:28:14.364 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:28:15.760 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 03:28:15.761 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:29:14.370 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:34:41.162 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 03:34:41.163 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:36:41.976 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 03:36:41.976 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:38:14.363 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 03:40:20.115 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 03:40:20.115 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:44:19.896 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 03:44:19.897 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:58:16.807 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 03:58:16.808 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 03:59:56.771 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 03:59:56.772 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 04:01:14.351 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 04:02:21.839 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 04:02:21.839 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 04:04:17.315 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 04:04:17.316 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 04:06:14.352 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 04:12:15.775 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 04:12:15.776 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 04:20:15.765 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 04:20:15.766 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 04:21:14.386 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 04:22:14.384 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 04:22:31.632 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 04:22:31.633 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 04:24:14.386 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:10:14.378 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:10:15.778 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 05:10:15.778 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:12:56.455 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:14:14.391 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:15:14.404 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:18:14.428 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:18:17.198 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 05:18:17.198 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:20:15.784 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 05:20:15.784 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:22:14.387 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:22:15.794 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 05:22:15.794 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:26:44.623 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:30:15.804 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 05:30:15.804 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:32:23.457 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 05:32:23.457 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:34:14.372 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:34:18.550 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 05:34:18.550 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:40:15.803 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 05:40:15.803 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:41:14.369 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:42:14.381 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:42:15.762 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 05:42:15.762 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:44:14.382 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:44:15.773 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 05:44:15.773 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:46:18.539 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 05:46:18.539 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:47:45.915 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 05:47:45.915 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:50:38.248 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 05:50:38.249 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 05:51:14.387 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:53:14.387 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 05:54:15.789 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 05:54:15.789 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 06:00:14.406 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 06:00:15.785 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 06:00:15.785 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 06:21:55.414 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 06:21:55.415 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 06:30:45.704 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 06:33:14.431 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 06:36:15.804 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 06:36:15.804 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 06:38:21.732 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 06:38:21.732 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 06:40:18.506 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 06:40:18.506 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 06:43:44.610 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 07:03:45.774 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 07:27:45.715 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 07:28:15.848 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 07:28:15.848 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 07:32:14.574 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 07:32:43.333 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 07:32:43.333 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 07:57:56.017 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 07:57:56.018 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 08:08:15.886 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 08:08:15.886 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 08:09:14.414 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 08:10:14.436 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 08:10:15.863 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 08:10:15.864 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 08:12:14.411 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 08:12:15.800 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 08:12:15.800 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 08:14:16.935 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 08:14:16.935 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 08:15:14.438 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 08:16:14.412 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 08:16:15.847 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 08:16:15.847 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 08:30:15.772 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 08:30:15.772 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 08:59:34.247 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 08:59:34.248 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 09:05:24.027 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 09:11:21.148 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 09:11:21.148 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 09:15:20.965 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations
[2020-12-11 09:15:20.965 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 09:17:21.869 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 09:17:21.869 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 09:21:55.709 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 09:23:23.993 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 09:45:04.703 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 09:45:04.703 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 09:46:50.385 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 09:46:50.386 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 09:49:20.260 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 09:49:20.261 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 09:55:24.088 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed
[2020-12-11 09:57:20.250 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing
[2020-12-11 09:57:20.251 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
[2020-12-11 09:59:24.718 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations
[2020-12-11 09:59:24.719 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network
@Jannis
Copy link
Contributor

Jannis commented Dec 11, 2020

/network request stats (last 6 hours)

| Status | Requests (total) | Percent |
|--------+------------------+---------|
|        |           778000 |  100.00 |
|    200 |           679700 |   87.37 |
|    204 |            24500 |    3.15 |
|    404 |            23000 |    2.96 |
|    499 |              300 |    0.04 |
|    502 |            50000 |    6.43 |

About 10% of the overall requests to =/network= are failing with a 404, 499 or 502 error. That is neither great nor terrible, since the agent and service both try again after a minute or two.

@Jannis
Copy link
Contributor

Jannis commented Dec 11, 2020 •
edited
Loading

The total number of requests I'd expect in 6 hours is something like:

total_requests = n_indexers * (3600 * 6 / 120) * (n_services + 1) * c

where

n_indexers ~ 100
n_services ~ 4
c          ~ 5 (the number of queries we make in each syncing cycle)

That adds up to 450000 requests made by indexers alone (the gateways also make some).

@Jannis
Copy link
Contributor

Jannis commented Dec 11, 2020

I've seen about 200 IE010 errors over 5 minutes on average, over the last 6 hours. That's 0.6/s, which adds up to 14,400 failed IE010s over 6 hours. Compared to about ~ 90,000 expected requests over that period.

That makes it an estimated 16% error rate on the IE010 requests alone. Too high, for sure.

@davaymne
Copy link
Author

davaymne commented Dec 11, 2020 •
edited
Loading

Full log for day 2020-12-11 is here

agent.b5b624ffdfc41715fff3935284ef42a6a.log

@davaymne
Copy link
Author

davaymne commented Dec 11, 2020 •
edited
Loading

Number of Errors per minute:
Screenshot 2020-12-11 at 11 57 10

@WayneWayner
Copy link

Same issue on my side.
https://gist.github.com/WayneWayner/7b9765209f1c69fc629a378afa4a5a0e

@davaymne
Copy link
Author

davaymne commented Dec 13, 2020 •
edited
Loading

Below you can find number of Failed messages by day:
20201209 - 128
20201210 - 266
20201211 - 418
20201212 - 194
20201213 - 207

Please, let me know if you need more details.

@davaymne
Copy link
Author

Allocations are not getting updated, hence incorrect Indexer Reword is displayed on Account Card.
Screenshot 2020-12-14 at 13 31 35

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Jannis @davaymne @WayneWayner