Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unexpected plot log and process count #161

Open
altendky opened this issue Apr 24, 2021 · 5 comments
Open

Unexpected plot log and process count #161

altendky opened this issue Apr 24, 2021 · 5 comments

Comments

@altendky
Copy link
Collaborator

+1. Running plotman in WSL2 with 1 tmp drive and 1 dst drive without archiving:

directories:
        log: /home/jay/chia-blockchain/plotman-logs
        tmp:
                - /mnt/p
        tmp_overrides:
                "/mnt/p":
                        tmpdir_max_jobs: 8

        # tmp2: /mnt/tmp/a

        dst:
                - /mnt/d
                - 
        # archive:
                # rsyncd_module: plots
                # rsyncd_path: /plots
                # rsyncd_bwlimit: 80000  # Bandwidth limit in KB/s
                # rsyncd_host: myfarmer
                # rsyncd_user: chia

Here is my configuration for scheduling:

        tmpdir_stagger_phase_major: 2
        tmpdir_stagger_phase_minor: 1
        # Optional: default is 1
        tmpdir_stagger_phase_limit: 2

        # Don't run more than this many jobs at a time on a single temp dir.
        tmpdir_max_jobs: 8

        # Don't run more than this many jobs at a time in total.
        global_max_jobs: 10

        # Don't run any jobs (across all temp dirs) more often than this.
        global_stagger_m: 10

        # How often the daemon wakes to consider starting a new plot job
        polling_time_s: 20

My understanding is that maximum of 8 jobs will be created with this configuration, and should be able to display the status of all the 8 jobs. However, there are 10 plotting logs created and the only shows 2 jobs：

(venv) root@JAY-DESKTOP:/home/jay/chia-blockchain/plotman-logs# ls
2021-04-23-16:44:25.log  2021-04-23-17:23:34.log  2021-04-23-18:22:51.log  2021-04-23-20:01:25.log
2021-04-23-16:54:26.log  2021-04-23-17:38:59.log  2021-04-23-18:54:41.log
2021-04-23-17:09:51.log  2021-04-23-17:53:23.log  2021-04-23-19:28:13.log
(venv) root@JAY-DESKTOP:/home/jay/chia-blockchain/plotman-logs# plotman status
 plot id    k      tmp      dst   wall   phase    tmp    pid   stat    mem   user    sys   io
3bef6f46   32   /mnt/p   /mnt/d   0:20     1:2    68G   8034    SLP   5.4G   0:19   0:03   0s
b8a0387a   32   /mnt/p   /mnt/d   0:54     1:3   111G   7943    SLP   5.4G   0:47   0:04   0s

Did I miss anything or mis-understand the configuration? Please help, thanks!

Originally posted by @zrjaa1 in #149 (comment)
@altendky
Copy link
Collaborator Author

altendky commented Apr 24, 2021 •
edited
Loading

From this information I'm not sure what is going on. The log files aren't proof that plotman has started 10 simultaneous processes and the two found processes doesn't show that plotman is missing processes that are running. Could you share the following to help debug this please?

  • all of the log files here, maybe in a .zip
  • the complete configuration file
  • the prompt, command, and output for each of:
    • tail /home/jay/chia-blockchain/plotman-logs/*.log
    • ps aux | grep 'chia plots create'
    • plotman status

My guess is that the plotting processes are terminating for some reason. I vaguely guess based on the timing that it isn't due to completion. Usually plots that fail do so fairly immediately, in my experience. But... That's the best guess I've got until we can look over the additional information.

@altendky
Copy link
Collaborator Author

Also, let's add the full configuration file to the list. (edited above)

@zrjaa1
Copy link

zrjaa1 commented Apr 24, 2021

Hi @altendky, thanks for the help. I inspected the log files and looks like plots were not terminated, I'll attach the zip below. Unfortunately, I already killed the plot jobs since my plotting drive was full. Here is the output of the command:

config.yml:

/root/.config/plotman/plotman.yaml
(venv) root@JAY-DESKTOP:/home/jay/chia-blockchain/plotman-logs# cat /root/.config/plotman/plotman.yaml
# Default/example plotman.yaml configuration file

# Options for display and rendering
user_interface:
        # Call out to the `stty` program to determine terminal size, instead of
        # relying on what is reported by the curses library.   In some cases,
        # the curses library fails to update on SIGWINCH signals.  If the
        # `plotman interactive` curses interface does not properly adjust when
        # you resize the terminal window, you can try setting this to True.
        use_stty_size: True

# Where to plot and log.
directories:
        # One directory in which to store all plot job logs (the STDOUT/
        # STDERR of all plot jobs).  In order to monitor progress, plotman
        # reads these logs on a regular basis, so using a fast drive is
        # recommended.
        log: /home/jay/chia-blockchain/plotman-logs

        # One or more directories to use as tmp dirs for plotting.  The
        # scheduler will use all of them and distribute jobs among them.
        # It assumes that IO is independent for each one (i.e., that each
        # one is on a different physical device).
        #
        # If multiple directories share a common prefix, reports will
        # abbreviate and show just the uniquely identifying suffix.
        tmp:
                - /mnt/p

        # Optional: Allows overriding some characteristics of certain tmp
        # directories. This contains a map of tmp directory names to
        # attributes. If a tmp directory and attribute is not listed here,
        # it uses the default attribute setting from the main configuration.
        #
        # Currently support override parameters:
        #     - tmpdir_max_jobs
        tmp_overrides:
                # In this example, /mnt/tmp/00 is larger than the other tmp
                # dirs and it can hold more plots than the default.
                "/mnt/p":
                        tmpdir_max_jobs: 8

        # Optional: tmp2 directory.  If specified, will be passed to
        # chia plots create as -2.  Only one tmp2 directory is supported.
        # tmp2: /mnt/tmp/a

        # One or more directories; the scheduler will use all of them.
        # These again are presumed to be on independent physical devices,
        # so writes (plot jobs) and reads (archivals) can be scheduled
        # to minimize IO contention.
        dst:
                - /mnt/d

        # Archival configuration.  Optional; if you do not wish to run the
        # archiving operation, comment this section out.
        #
        # Currently archival depends on an rsync daemon running on the remote
        # host, and that the module is configured to match the local path.
        # See code for details.
        # archive:
                # rsyncd_module: plots
                # rsyncd_path: /plots
                # rsyncd_bwlimit: 80000  # Bandwidth limit in KB/s
                # rsyncd_host: myfarmer
                # rsyncd_user: chia
                # Optional index.  If omitted or set to 0, plotman will archive
                # to the first archive dir with free space.  If specified,
                # plotman will skip forward up to 'index' drives (if they exist).
                # This can be useful to reduce io contention on a drive on the
                # archive host if you have multiple plotters (simultaneous io
                # can still happen at the time a drive fills up.)  E.g., if you
                # have four plotters, you could set this to 0, 1, 2, and 3, on
                # the 4 machines, or 0, 1, 0, 1.
                #   index: 0


# Plotting scheduling parameters
scheduling:
        # Run a job on a particular temp dir only if the number of existing jobs
        # before tmpdir_stagger_phase_major tmpdir_stagger_phase_minor
        # is less than tmpdir_stagger_phase_limit.
        # Phase major corresponds to the plot phase, phase minor corresponds to
        # the table or table pair in sequence, phase limit corresponds to
        # the number of plots allowed before [phase major, phase minor]
        tmpdir_stagger_phase_major: 2
        tmpdir_stagger_phase_minor: 1
        # Optional: default is 1
        tmpdir_stagger_phase_limit: 2

        # Don't run more than this many jobs at a time on a single temp dir.
        tmpdir_max_jobs: 8

        # Don't run more than this many jobs at a time in total.
        global_max_jobs: 10

        # Don't run any jobs (across all temp dirs) more often than this.
        global_stagger_m: 10

        # How often the daemon wakes to consider starting a new plot job
        polling_time_s: 20


# Plotting parameters.  These are pass-through parameters to chia plots create.
# See documentation at
# https://github.com/Chia-Network/chia-blockchain/wiki/CLI-Commands-Reference#create
plotting:
        k: 32
        e: False             # Use -e plotting option
        n_threads: 4         # Threads per job
        n_buckets: 128       # Number of buckets to split data into
        job_buffer: 4608     # Per job memory


tail /home/jay/chia-blockchain/plotman-logs/*.log

(venv) root@JAY-DESKTOP:/home/jay/chia-blockchain/plotman-logs# tail /home/jay/chia-blockchain/plotman-logs/*.log
==> /home/jay/chia-blockchain/plotman-logs/2021-04-23-16:44:25.log <==
        Bucket 123 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.281GiB.
        Bucket 124 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Bucket 125 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Bucket 126 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Bucket 127 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Total matches: 4294973037
Forward propagation table time: 1269.568 seconds. CPU (138.040%) Fri Apr 23 17:09:34 2021
Computing table 3
        Bucket 0 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 1 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.

==> /home/jay/chia-blockchain/plotman-logs/2021-04-23-16:54:26.log <==
        Bucket 122 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Bucket 123 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Bucket 124 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.281GiB.
        Bucket 125 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.281GiB.
        Bucket 126 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Bucket 127 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Total matches: 4295028031
Forward propagation table time: 1516.240 seconds. CPU (129.380%) Fri Apr 23 17:23:29 2021
Computing table 3
        Bucket 0 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.

==> /home/jay/chia-blockchain/plotman-logs/2021-04-23-17:09:51.log <==
        Bucket 125 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Bucket 126 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.281GiB.
        Bucket 127 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Total matches: 4295098237
Forward propagation table time: 1443.906 seconds. CPU (130.910%) Fri Apr 23 17:37:59 2021
Computing table 3
        Bucket 0 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 1 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 2 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 3 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.

==> /home/jay/chia-blockchain/plotman-logs/2021-04-23-17:23:34.log <==
        Bucket 124 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.281GiB.
        Bucket 125 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Bucket 126 uniform sort. Ram: 4.434GiB, u_sort min: 0.563GiB, qs min: 0.281GiB.
        Bucket 127 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.281GiB.
        Total matches: 4294975976
Forward propagation table time: 1456.793 seconds. CPU (126.730%) Fri Apr 23 17:51:46 2021
Computing table 3
        Bucket 0 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 1 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 2 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.

==> /home/jay/chia-blockchain/plotman-logs/2021-04-23-17:38:59.log <==
        Bucket 24 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 25 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 26 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 27 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 28 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 29 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 30 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 31 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 32 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 33 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.

==> /home/jay/chia-blockchain/plotman-logs/2021-04-23-17:53:23.log <==
        Bucket 70 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 71 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 72 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 73 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 74 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 75 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 76 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 77 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 78 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 79 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.

==> /home/jay/chia-blockchain/plotman-logs/2021-04-23-18:22:51.log <==
        Bucket 68 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 69 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 70 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 71 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 72 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 73 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 74 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 75 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 76 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 77 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.

==> /home/jay/chia-blockchain/plotman-logs/2021-04-23-18:54:41.log <==
        Bucket 67 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 68 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 69 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 70 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 71 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 72 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 73 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 74 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 75 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 76 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.

==> /home/jay/chia-blockchain/plotman-logs/2021-04-23-19:28:13.log <==
        Bucket 71 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 72 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 73 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 74 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 75 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 76 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 77 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 78 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 79 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 80 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.

==> /home/jay/chia-blockchain/plotman-logs/2021-04-23-20:01:25.log <==
        Bucket 70 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 71 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 72 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 73 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 74 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 75 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 76 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 77 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.
        Bucket 78 uniform sort. Ram: 4.434GiB, u_sort min: 2.250GiB, qs min: 0.563GiB.
        Bucket 79 uniform sort. Ram: 4.434GiB, u_sort min: 1.125GiB, qs min: 0.562GiB.


(venv) root@JAY-DESKTOP:/home/jay/chia-blockchain/plotman-logs# ps aux | grep 'chia plots create'
root       278  0.0  0.0   8164   724 pts/0    S+   12:26   0:00 grep --color=auto chia plots create

zip of log files:
plotman-logs.zip

@altendky
Copy link
Collaborator Author

There are incomplete logs. They don't indicate why. If it happens again go ahead and collect the info again. Sorry.

@zrjaa1
Copy link

zrjaa1 commented Apr 24, 2021 •
edited
Loading

No worries. I'll add more details if I can re-produce the issue. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@altendky @zrjaa1