Windows: False "Harvester appears offline" notifications when farm is fine

[suckatlife]

Describe the bug

I'm running the Windows version of chiadog. Every now and then I'll get the "Your harvester appears offline! No events for the past xxxx seconds" notification, and I will continue getting this notification until I restart chiadog. When I check my farm (using chia farm summary) everything is fine. I actually won a block during one of these fake outages.

Looking at timestamps, these outages seem to always coincide with a log rotation, but not every rotation. My log rotates every 40-50 minutes, but this only happens around once a day or so - so it's definitely not with every rotation.

It was suggested I turn off any other processes tailing (using get-content) debug.log, which I've done, but it didn't help.

Environment:

• OS: Windows 10
• Python version: 3.9.4
• PIP version: 21.1
• Chia version: 1.1.3

[centrd]

This is a known issue. There is a fix in the works.
It is indeed related to log rotation and the way logs are being accessed.

[pieterhelsen]

I wonder if we can write a Windows-specific monitor that hooks into the keep alive monitor thread and then tries to reset the _consume_loop in the log_consumer if there's no activity for 60 seconds...

Or might be even better to write a monitor that monitors the size of the debug.log (on Windows) and resets the _consume_loop if the size goes from X MB to < 1 MB.

What do you think @martomi ?

[pieterhelsen]

Apologies @skrustev ; I should've updated the issue.
I've been working on a fix as well, which has been tested successfully by a few people on the Keybase channel.
https://github.com/martomi/chiadog/tree/windows-rotating-log (initial version; needs cleaning up)

However, your solution ( #87 ) would be preferable I think, as it's not so invasive. Thoughts, @martomi ?

[greimela]

If #87 fixes the issue and has no side effects it would be really nice!

[skrustev]

Yeah, It does not fix the issue, that is why I closed it. It doesn't happen as often but I still got the false offline state.
You can still try it out in the meantime but I don't think its enough.

[pieterhelsen]

OK! Thanks anyway, it would've been a graceful solution :)
Feel free to try out my branch instead. I've only implemented it for local File Consumers right now, but will implement the fix for the NetworkLogConsumer tonight.

[pieterhelsen]

I have moved the work on this feature to a new branch:
https://github.com/martomi/chiadog/tree/windows-rotation

This ticket relates to #102

[tschechniker]

@pieterhelsen i see the same issue on linux too. It works fine for a while then harvester offline messages pops up. Everytime i check the logs it's totally fine. From what i can see it's related

[DrHou83]

Hey guys are we waiting on a fix here. Does it help if I switch the log back from INFO to stop the log rotation.

[24601]

I can confirm this also occurs on Mac OS X (Big Sur)

[gilgm12]

This fixed up yet? I'm resetting chiadog daily.

[pieterhelsen]

We have a fix ready that uses the Pygtail module to provide a more pythonic way of reading the logfiles and has better handling of log rotations.

[martomi]

Closing this now as the fix is in the new release. Please reopen if issue persists.

[runechronos]

Are you sure ? I exactly have the same error, after the update, keeps telling me harvester is offline ????
Here is my daily recap of yesterday for example, does it seems regular to u ?

ℹ️ Chia DAILY: Hello farmer! 👋 Here's what happened in the last 24 hours:

Received ☘️: 0.00 XCH
Proofs 🧾: None
Search 🔍:
- average: 0.10s over 9124 searches
- over 5s: 3 occasions (0.0%)
- over 15s: 0 occasions (0.0%)
Plots 🌱: 132, new: 4
Eligible plots 🥇: 0.25 average
Skipped SPs ⚠️: Unknown

[pieterhelsen]

Daily stats look OK. Some additional questions for you:

• How often do you get the message telling you your harvester is offline?
• Are you monitoring a local harvester or a remote harvester (using SSH)?

If the problem still persists, please change this line temporarily

chiadog/src/chia_log/log_consumer.py

Line 75 in 429b90c

for log_line in Pygtail(self._expanded_log_path, read_from_end=True, offset_file=self._offset_path):

from

for log_line in Pygtail(self._expanded_log_path, read_from_end=True, offset_file=self._offset_path):

to this

for log_line in Pygtail(self._expanded_log_path, read_from_end=True, offset_file=self._offset_path, paranoid=True):

Please report back if this fixes your problem.

[gilgm12]

After a pull the other day, things seem much better for me - no false 'offline' since.

…

On Fri, May 28, 2021 at 1:55 AM pieterhelsen ***@***.***> wrote: Daily stats look OK. Some additional questions for you: - How often do you get the message telling you your harvester is offline? - Are you monitoring a local harvester or a remote harvester (using SSH)? If the problem still persists, please change this line temporarily https://github.com/martomi/chiadog/blob/429b90cf2d3f74a885d4ffcb748fb63b1521e48b/src/chia_log/log_consumer.py#L75 from for log_line in Pygtail(self._expanded_log_path, read_from_end=True, offset_file=self._offset_path): to this for log_line in Pygtail(self._expanded_log_path, read_from_end=True, offset_file=self._offset_path, paranoid=True): Please report back if this fixes your problem. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#72 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEX77TLN7OSUYJENQQIE73TP5D7ZANCNFSM437ZVWAQ> .

[runechronos]

@pieterhelsen I'm monitoring a local harvester. I create my plots with a computer 1, send them to a computer 2 via my local network, and i'm farming and running the full node on computer 2. I get the error mesage every 5 min. So it's a huge spam on pushover app ^^

What will it change to add the "paranoid=true" parameter ? (only removing harvesters notifications maybe ?)

[gilgm12]

@runechronos - that was exactly how it was for me until I did git pull in master. I think this has been fixed.

…

-M

On Mon, May 31, 2021 at 7:14 AM runechronos ***@***.***> wrote: @pieterhelsen <https://github.com/pieterhelsen> I'm monitoring a local harvester. I create my plots with a computer 1, send them to a computer 2 via my local network, and i'm farming and running the full node on computer 2. I get the error mesage every 5 min. So it's a huge spam on pushover app ^^ — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#72 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEX77VFD5IBAJEYYJQQAB3TQODTBANCNFSM437ZVWAQ> .

[runechronos]

@gilgm12 I think you are right, i did a new pull, and it seems to be ok now :)

Just can someone explain to me how the different "search tries are working ?

Search 🔍:

• average: 0.46s in 9500 tries
• over 5s: 0
• over 15s: 0
  (for example)
  What has to be achieved to get the block reward (i guess one of your plot has to pass all 3 and maybe more ?)

Or if you have some link explaining how Chia harvesting is working exactly, would like to be less dumb ^^

Thanks in advance.

[gilgm12]

Generally you want the scan times to be as close to zero as possible. Mine look like this: Search 🔍: - average: 0.34s over 9307 searches - over 5s: 42 occasions (0.5%) - over 15s: 16 occasions (0.2%) The majority of my scans are fast, but sometimes when plots are transferring to almost-full disks, the search time is slower. Chia rewards are sort of like a bingo ticket, and the search times are how fast you yell bingo. Under 5s recommended, over 30 and you lose that opportunity to win. This is what I've gathered at least - there might be subtleties I'm omitting. I'd also be interested in knowing if I 'missed' a win - this is out of scope for this particular ticket though. Glad it's working for you!

…

On Thu, Jun 3, 2021, 3:21 AM runechronos ***@***.***> wrote: @gilgm12 <https://github.com/gilgm12> I think you are right, i did a new pull, and it seems to be ok now :) Just can someone explain to me how the different "search tries are working ? Search 🔍: - average: 0.46s in 9500 tries - over 5s: 0 - over 15s: 0 (for example) What has to be achieved to get the block reward (i guess one of your plot has to pass all 3 and maybe more ?) Or if you have some link explaining how Chia harvesting is working exactly, would like to be less dumb ^^ Thanks in advance. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#72 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEX77XRBRJF4S2SS7LT5O3TQ5CRNANCNFSM437ZVWAQ> .