Big overhead in parallel transpile for large target systems

[nonhermitian]

Environment

• Qiskit Terra version: 0.19.2
• Python version: 3.8
• Operating system: Linux

What is happening?

There is a large overhead when doing parallel transpilation for circuits targeting larger systems. In the case below they are targeting the 127Q system, but the circuits themselves consist of only x gates and are already the correct width. Thus transpilation should just be a pass-through. However, in parallel mode (quad-core machine) it takes 330sec to transpile. If I turn off parallel transpile it takes 3sec.

from qiskit import *
from mthree.circuits import balanced_cal_strings, balanced_cal_circuits

b_strs = balanced_cal_strings(127)
cal_circs = balanced_cal_circuits(b_strs)

backend = provider.backend.ibm_washington
trans_circs = transpile(cal_circs, backend, optimization_level=0)

How can we reproduce the issue?

Run above

What should happen?

Parallel should not give 100x overhead.

Any suggestions?

I am guessing this is copying overhead from forking when the systems are large.

[mtreinish]

I did a quick profile of a modified version of your script, just running transpile() on [QuantumCircuit() for _ in range(n)] targeting ibm_washington. and it is indeed spending most of the time in pickle to serialize the all the transpile arguments to send them to the subprocesses. Then each subprocess is spending the vast majority of their time deserializing the transpiler arguments payload. I'm assuming it's because there are a lot more properties for washington, it's only 1.5MB of data (for BackendProperties and the instruction schedule map) but when you multiply that 127 times for each circuit in the cal circuits list that time is going to be large.

Just thinking out loud we might need to look if we can use shared memory somehow between the processes to avoid the serialization overhead because this problem is only going to get worse as the size of devices increases. Maybe we can leverage https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory although it's new enough that I haven't tried it before (and only available starting in python 3.8).

[mtreinish]

While I was investigating adding the same approach from #7789 to the parallel dispatch in passmanager.run() I realized that using shared memory explicitly there doesn't seem to have any benefit. I'm guess the overhead we're seeing in transpiler is partially for serializing a long list of different arguments instead of a single pass manager instance. The tl;dr is if you replace your code example with:

import time

from qiskit.transpiler.preset_passmanagers import generate_preset_pass_manager
from qiskit.providers.fake_provider import FakeWashington
from qiskit.circuit.quantumcircuit import QuantumCircuit

from mthree.circuits import balanced_cal_strings, balanced_cal_circuits

backend = FakeWashington()
b_strs = balanced_cal_strings(127)
cal_circs = balanced_cal_circuits(b_strs, list(range(127)), 127)
# generate level 0 pass manager for backend
pm = generate_preset_pass_manager(0, backend)
start = time.perf_counter()
pm.run(cal_circs)
stop = time.perf_counter()
print(stop - start)

(this will only work on main because generate_preset_pass_manager() only merged recently)

that is running in parallel and taking ~3-4 seconds on python 3.7 (and ~1.6 seconds on python 3.10) for me.

This is making me think to close this issue we might need to revisit the list inputs for arguments on transpile(). If we can move transpile() to only supporting single arguments for all circuits in the input than we can move to creating a single pass manager for all the circuits and get this benefit (this will also make it substantially easier to make transpile() pluggable).

[nonhermitian]

Ahh this is interesting and seems to be the correct direction going forward.

[mtreinish]

This is only partially addressed by the recently merged pr the full performance isn't there yet

[mtreinish]

Just to write out the current plan here for completeness:

After 0.24.0 when the transpile() argument dispatch's deprecation window has completed we can remove all the dispatch support and drop the unique vs shared argument handling and passing extra arguments to the parallel workers as everything will be shared. Once that's done we should update transpile() to call generate_preset_pass_manager() inside transpile() and then just use the .run() method with all the provided circuits.