Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce log verbosity #1523

Closed
Stebalien opened this issue Oct 12, 2021 · 6 comments · Fixed by #1530
Closed

Reduce log verbosity #1523

Stebalien opened this issue Oct 12, 2021 · 6 comments · Fixed by #1530

Comments

@Stebalien
Copy link
Member

Enabling info logging currently causes nodes to fall out of sync due to a flood of messages like:

2021-10-08T00:09:24.757 INFO filecoin_proofs::pieces > verifying 1 pieces
2021-10-08T00:09:24.757 INFO filecoin_proofs::api::seal > compute_comm_d:finish
2021-10-08T00:09:24.757 INFO filcrypto::proofs::api > generate_data_commitment: finish
2021-10-08T00:09:24.757 INFO filcrypto::proofs::api > generate_data_commitment: start
2021-10-08T00:09:24.757 INFO filecoin_proofs::api::seal > compute_comm_d:start
2021-10-08T00:09:24.757 INFO filecoin_proofs::pieces > verifying 1 pieces
2021-10-08T00:09:24.757 INFO filecoin_proofs::api::seal > compute_comm_d:finish
2021-10-08T00:09:24.757 INFO filcrypto::proofs::api > generate_data_commitment: finish
2021-10-08T00:09:24.758 INFO filcrypto::proofs::api > generate_data_commitment: start
2021-10-08T00:09:24.758 INFO filecoin_proofs::api::seal > compute_comm_d:start
2021-10-08T00:09:24.758 INFO filecoin_proofs::pieces > verifying 1 pieces
2021-10-08T00:09:24.758 INFO filecoin_proofs::api::seal > compute_comm_d:finish
2021-10-08T00:09:24.758 INFO filcrypto::proofs::api > generate_data_commitment: finish
2021-10-08T00:09:24.758 INFO filcrypto::proofs::api > generate_data_commitment: start
2021-10-08T00:09:25.105 INFO filecoin_proofs::api::seal > got verifying key (68719476736) while verifying seal
2021-10-08T00:09:25.105 INFO filecoin_proofs::api::seal > verify_seal:finish: SectorId(15312)
2021-10-08T00:09:25.106 INFO filcrypto::proofs::api > verify_seal: finish
2021-10-08T00:09:25.106 INFO filcrypto::proofs::api > verify_seal: start
2021-10-08T00:09:25.106 INFO filecoin_proofs::api::seal > verify_seal:start: SectorId(1512883)
2021-10-08T00:09:25.106 INFO filecoin_proofs::caches > trying parameters memory cache for: STACKED[68719476736]-verifying-key
2021-10-08T00:09:25.106 INFO filecoin_proofs::caches > found params in memory cache for STACKED[68719476736]-verifying-key
2021-10-08T00:09:25.106 INFO filecoin_proofs::api::seal > got verifying key (68719476736) while verifying seal
2021-10-08T00:09:25.106 INFO filecoin_proofs::api::seal > verify_seal:finish: SectorId(143174)
2021-10-08T00:09:25.106 INFO filcrypto::proofs::api > verify_seal: finish
2021-10-08T00:09:25.106 INFO filcrypto::proofs::api > verify_seal: start
2021-10-08T00:09:25.106 INFO filecoin_proofs::api::seal > verify_seal:start: SectorId(44431)
2021-10-08T00:09:25.106 INFO filecoin_proofs::caches > trying parameters memory cache for: STACKED[34359738368]-verifying-key
2021-10-08T00:09:25.106 INFO filecoin_proofs::caches > found params in memory cache for STACKED[34359738368]-verifying-key
2021-10-08T00:09:25.106 INFO filecoin_proofs::api::seal > got verifying key (34359738368) while verifying seal
2021-10-08T00:09:25.106 INFO filecoin_proofs::api::seal > verify_seal:finish: SectorId(78605)
2021-10-08T00:09:25.106 INFO filecoin_proofs::api::seal > verify_seal:finish: SectorId(16609)
2021-10-08T00:09:25.106 INFO filcrypto::proofs::api > verify_seal: finish
2021-10-08T00:09:25.106 INFO filcrypto::proofs::api > verify_seal: start
2021-10-08T00:09:25.106 INFO filecoin_proofs::api::seal > verify_seal:start: SectorId(242314)
2021-10-08T00:09:25.106 INFO filecoin_proofs::caches > trying parameters memory cache for: STACKED[68719476736]-verifying-key
2021-10-08T00:09:24.973 INFO filecoin_proofs::api::seal > compute_comm_d:finish
2021-10-08T00:09:24.973 INFO filcrypto::proofs::api > generate_data_commitment: finish
2021-10-08T00:09:24.974 INFO filcrypto::proofs::api > generate_data_commitment: start
2021-10-08T00:09:24.974 INFO filecoin_proofs::api::seal > compute_comm_d:start
2021-10-08T00:09:24.974 INFO filecoin_proofs::pieces > verifying 1 pieces
2021-10-08T00:09:24.974 INFO filecoin_proofs::api::seal > compute_comm_d:finish
2021-10-08T00:09:24.974 INFO filcrypto::proofs::api > generate_data_commitment: finish
2021-10-08T00:09:24.974 INFO filcrypto::proofs::api > generate_data_commitment: start
2021-10-08T00:09:24.974 INFO filecoin_proofs::api::seal > compute_comm_d:start
2021-10-08T00:09:24.974 INFO filecoin_proofs::pieces > verifying 1 pieces
2021-10-08T00:09:24.974 INFO filecoin_proofs::api::seal > compute_comm_d:finish
2021-10-08T00:09:24.974 INFO filcrypto::proofs::api > generate_data_commitment: finish
2021-10-08T00:09:24.975 INFO filcrypto::proofs::api > generate_data_commitment: start
2021-10-08T00:09:24.975 INFO filecoin_proofs::api::seal > compute_comm_d:start
2021-10-08T00:09:24.975 INFO filecoin_proofs::pieces > verifying 1 pieces
2021-10-08T00:09:24.975 INFO filecoin_proofs::api::seal > compute_comm_d:finish
2021-10-08T00:09:24.975 INFO filcrypto::proofs::api > generate_data_commitment: finish
2021-10-08T00:09:24.978 INFO filcrypto::proofs::api > generate_data_commitment: start
2021-10-08T00:09:24.978 INFO filecoin_proofs::api::seal > compute_comm_d:start
2021-10-08T00:09:24.978 INFO filecoin_proofs::pieces > verifying 1 pieces

These should all be debug.

@arajasek
Copy link
Contributor

Note we're not sure that that's what's causing nodes to fall out of sync, but agree with the suggestion

@cryptonemo cryptonemo transferred this issue from filecoin-project/filecoin-ffi Oct 12, 2021
@cryptonemo
Copy link
Collaborator

It sounds very unlikely a culprit to sync issues (or at least hasn't changed much recently). But we can tweak some logging and levels.

@Stebalien
Copy link
Member Author

@cryptonemo the default log level appears to be warn (I think?). The user in question enabled debug logging, then their node started falling out of sync. I'd expect debug logging to cause that regardless, but these messages still shouldn't be logged as infos.

@Stebalien
Copy link
Member Author

Or maybe just off?

@Stebalien
Copy link
Member Author

I guess the question is: do we run any nodes with this level of logging enabled?

@cryptonemo
Copy link
Collaborator

I guess the question is: do we run any nodes with this level of logging enabled?

Fair question. I don't run it on my node. Generally, we only advise enabling logging when there are failures being investigated. I agree that the info/debug/trace levels need adjustment though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants