Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cluster backend #1281

Merged
merged 28 commits into from
Aug 23, 2021
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
bea9481
Add ClusterBackend and related utilities
Nov 19, 2020
1ce8807
Fix docs, pylint, tests
Dec 1, 2020
5371bb4
Added statevector and unitary tests
Dec 2, 2020
44ed808
Simplified clusterjob logic
Dec 14, 2020
dc56af3
Fixing pylint
Dec 16, 2020
fb9da38
Merge branch 'master' into cluster_backend
vvilpas Jan 14, 2021
b2f6565
Merge branch 'master' into cluster_backend
chriseclectic Mar 30, 2021
92fa323
Merge remote-tracking branch 'origin' into cluster_backend
hitomitak May 20, 2021
fce954d
Support the latest qiskit
hitomitak May 25, 2021
c5b2f3c
Refactoring cluster backend code
hitomitak Jun 10, 2021
f0fcbac
Update code to add the whole execution time to time_taken value
hitomitak Jun 24, 2021
b152345
Merge branch 'Qiskit:main' into cluster_backend
hitomitak Jun 24, 2021
4a17d16
fix pylint
hitomitak Jun 25, 2021
d715065
Move executor option to run_options
hitomitak Jul 21, 2021
a06090a
Merge branch 'main' into cluster_backend
hitomitak Jul 27, 2021
c6edd4c
Refactor executor and job splitting
chriseclectic Aug 12, 2021
10c8bb8
Add AerSimulator executor tests
chriseclectic Aug 12, 2021
878ff18
Revert "Add AerSimulator executor tests"
chriseclectic Aug 12, 2021
e82cd86
Revert "Refactor executor and job splitting"
chriseclectic Aug 12, 2021
9ca5f93
Refactor executor and job splitting
chriseclectic Aug 12, 2021
6205f5a
Add AerSimulator executor tests
chriseclectic Aug 12, 2021
5fc18fb
Merge pull request #2 from chriseclectic/chris_cluster_backend
hitomitak Aug 16, 2021
1d6ee6c
Change location to get executor options from qobj.config to option
hitomitak Aug 18, 2021
e0e7f99
Merge branch 'main' into cluster_backend
hitomitak Aug 18, 2021
86d7981
fix lint
hitomitak Aug 19, 2021
3e21fb9
Merge branch 'main' into cluster_backend
chriseclectic Aug 23, 2021
a3fb874
Update CONTRIBUTING.md
chriseclectic Aug 23, 2021
332b7aa
Apply suggestions from code review
chriseclectic Aug 23, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions qiskit/providers/aer/aerjob.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,11 @@ def __init__(self, backend, job_id, fn, qobj, *args):
self._args = args
self._future = None

def submit(self):
def submit(self, executor=None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

executor should be an arg of __init__ like with AerJobSet, not to submit

"""Submit the job to the backend for execution.

Args:
executor (futures.Executor or None): executor to handle asynchronous jobs
Raises:
QobjValidationError: if the JSON serialization of the Qobj passed
during construction does not validate against the Qobj schema.
Expand All @@ -75,11 +77,11 @@ def submit(self):
"""
if self._future is not None:
raise JobError("We have already submitted the job!")

self._future = self._executor.submit(self._fn,
self._qobj,
self._job_id,
*self._args)
_exec = executor or self._executor
self._future = _exec.submit(self._fn,
self._qobj,
self._job_id,
*self._args)

@requires_submit
def result(self, timeout=None):
Expand Down
90 changes: 71 additions & 19 deletions qiskit/providers/aer/backends/aerbackend.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,16 @@
from qiskit.providers.models import BackendStatus
from qiskit.result import Result
from qiskit.utils import deprecate_arguments
from qiskit.circuit import QuantumCircuit
from qiskit.pulse import Schedule
from qiskit.qobj import QasmQobj, PulseQobj
from qiskit.compiler import assemble

from ..aerjob import AerJob
from ..aererror import AerError
from .cluster.utils import split
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For organization we should rename cluster to jobs and put all the job files in there (so what is currently in cluster + aerjob)

from .cluster.aerjobset import AerJobSet


# Logger
logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -135,6 +140,9 @@ def run(self,
``backend_options`` taking precedence. This kwarg is deprecated
and direct kwarg's should be used for options to pass them to
``run_options``.

Raises:
ValueError: if run is not implemented
"""
# DEPRECATED
if backend_options is not None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove backend_options

Suggested change
if backend_options is not None:

Expand All @@ -146,28 +154,68 @@ def run(self,
DeprecationWarning,
stacklevel=3)

if isinstance(circuits, (QasmQobj, PulseQobj)):
warnings.warn('Using a qobj for run() is deprecated and will be '
'removed in a future release.',
PendingDeprecationWarning,
stacklevel=2)
qobj = circuits
executor = None
if backend_options and "executor" in backend_options:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

backend options has been deprecated long enough that we can remove it in 0.9 release so this isn't needed.

Suggested change
if backend_options and "executor" in backend_options:

executor = backend_options["executor"]
del backend_options["executor"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really want to support doing this in backend_options even though it's deprecated.


if "executor" in run_options:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run options are supposed to be the same as the options that can be set with set_options. We should probably add a default executor to the simulator configuration that is the ThreadPoolExecutor that can then be overriden with a custom (ie dask) one using set_options.

executor = run_options["executor"]
del run_options["executor"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of doing this why not just add executor to the default options? That will also enable users to something like:

sim_backend = AerSimulator(executor=dask_magic())
sim_backend.run([qc]*10**5)
sim.backend.run([new_qc]*10**4)

and have it all run on dask.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first, I added the executor option as your code but DASK client variable is needed to be a local variable for the serialization. So I moved the executor option as a run option.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand this comment on why it can't be a a config option? Is the probably re-using an executor between calls to run? If so can executors be copied to avoid this?

Copy link
Contributor Author

@hitomitak hitomitak Aug 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

class AerBackend(Backend, ABC):
    def __init__():
        self.executor = None:
   /* ... */
    def run():
        self.executor = getattr(self._options, 'executor')
class AerBackend(Backend, ABC):
  /*...*/ 
    def run():
        executor = None
        executor = self._options_configuration["executor"]

These codes occur a cloudpickle error because the executor is not a local variable. The same error is mentioned in this website

So I changed the executor option as the run option because executor is passed every calling run function. Are there any good idea to realize the executor options as a config option?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found that the following code can be executed. I don't know why this code is ok but I can move executor option to default options.

class AerBackend(Backend, ABC):
    def __init__():
        self.executor = None:
   /* ... */
    def run():
         executor = None
        if self._executor:
           executor = self._executor
        else:
           executor = getattr(self._options, 'executor')
           delattr(self._options, 'executor')
       /* ....*/     
       aer_job_set = AerJobSet(self, job_id, self._run, experiments, executor)
       aer_job_set.submit()
       self._executor = executor
       return aer_job_set


if executor:
if isinstance(circuits, (QasmQobj, PulseQobj)):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is missing the deprecation warning the single job path raises

experiments = split(circuits)
elif isinstance(circuits, (QuantumCircuit, Schedule)):
experiments = [assemble(circuits, self)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we some control about batch size for splitting circuits. Splitting to 1-circuit per job may be quite inefficient if there are lots of cirucits?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also if we can control this we could use the executor code for default execution but with 1 batch and the default ThreadPoolExecutor

elif (
isinstance(circuits, list) and
all(isinstance(circ, QuantumCircuit) for circ in circuits) or
isinstance(circuits, list) and
all(isinstance(circ, Schedule) for circ in circuits)
):
experiments = [assemble(circ, self) for circ in circuits]
else:
raise ValueError(
"run() is not implemented for this "
"type of experiment ({})".format(str(type(circuits))))

for experiment in experiments:
self._add_options_to_qobj(experiment,
backend_options=backend_options,
**run_options)
# Optional validation
if validate:
self._validate(experiment)

job_id = str(uuid.uuid4())
aer_job_set = AerJobSet(self, job_id, self._run, experiments, executor)
aer_job_set.submit()
return aer_job_set

else:
qobj = assemble(circuits, self)
if isinstance(circuits, (QasmQobj, PulseQobj)):
warnings.warn('Using a qobj for run() is deprecated and will be '
'removed in a future release.',
PendingDeprecationWarning,
stacklevel=2)
qobj = circuits
else:
qobj = assemble(circuits, self)

# Add backend options to the Job qobj
self._add_options_to_qobj(
qobj, backend_options=backend_options, **run_options)
# Add backend options to the Job qobj
self._add_options_to_qobj(
qobj, backend_options=backend_options, **run_options)

# Optional validation
if validate:
self._validate(qobj)
# Optional validation
if validate:
self._validate(qobj)

# Submit job
job_id = str(uuid.uuid4())
aer_job = AerJob(self, job_id, self._run, qobj)
aer_job.submit()
return aer_job
# Submit job
job_id = str(uuid.uuid4())
aer_job = AerJob(self, job_id, self._run, qobj)
aer_job.submit()
return aer_job

def configuration(self):
"""Return the simulator backend configuration.
Expand Down Expand Up @@ -236,7 +284,7 @@ def status(self):
pending_jobs=0,
status_msg='')

def _run_job(self, job_id, qobj, backend_options, noise_model, validate):
def _run_job(self, job_id, qobj, backend_options=None, noise_model=None, validate=False):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method can be removed since it has been deprecated for long enough

"""Run a qobj job"""
warnings.warn(
'The `_run_job` method has been deprecated. Use `_run` instead.',
Expand All @@ -256,6 +304,9 @@ def _run_job(self, job_id, qobj, backend_options, noise_model, validate):
noise_model=noise_model)
return self._run(qobj, job_id)

def _dummy_job(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this here for?

return

def _run(self, qobj, job_id=''):
"""Run a job"""
# Start timer
Expand Down Expand Up @@ -347,6 +398,7 @@ def set_option(self, key, value):
setattr(self._options, key, getattr(self._default_options(), key))

def set_options(self, **fields):
"""Set the simulator options"""
for key, value in fields.items():
self.set_option(key, value)

Expand Down
31 changes: 31 additions & 0 deletions qiskit/providers/aer/backends/cluster/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# -*- coding: utf-8 -*-

# This code is part of Qiskit.
#
# (C) Copyright IBM 2018, 2019.
#
# This code is licensed under the Apache License, Version 2.0. You may
# obtain a copy of this license in the LICENSE.txt file in the root directory
# of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
#
# Any modifications or derivative works of this code must retain this
# copyright notice, and modified files need to carry a notice indicating
# that they have been altered from the originals.
"""
===========================================================
Cluster Backend (:mod:`qiskit.providers.aer.backends.cluster`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole sphinx docstring should be removed. there is no reason for this module to appear in API docs and it is refering to a class from old PR that doesn't exist anymore

Suggested change
Cluster Backend (:mod:`qiskit.providers.aer.backends.cluster`)

===========================================================

.. currentmodule:: qiskit.providers.aer.backends.cluster

High level mechanism for handling cluster jobs.

Classes
==========================
.. autosummary::
:toctree: ../stubs/

ClusterBackend
"""

from .aerjobset import AerJobSet
Loading