RFC: Proposal for reworked basis sytem

[akirakyle]

I've opened a lot of issues and done a lot of prototyping for various ways to improve basis handling here and in dependents. Rather than commenting on each issue separately, I wanted to collect them here and outline my proposal for what a better system looks like and how to get us there. This is partly since there's a lot of subtle interdependence between the issues and fixes for them.

Let's start with the abstract type interface which has been discussed in #26. Based on the experiments in #34 indicating the performance pitfalls of compilation overhead due to combinatorial explosion in type specialization (also see relevant Julia docs), I now think we should not have any type parameters here. If they are necessary in dependent packages, they can reintroduce them easily. So we would have

abstract type Basis end
abstract type StateVector end
abstract type AbstractKet <: StateVector end
abstract type AbstractBra <: StateVector end
abstract type AbstractOperator end
const AbstractQObjType = Union{<:StateVector,<:AbstractOperator} # For convenience
# Note no AbstractSuperOperator, I will address this below

For all the subtypes of Basis defined here, I think we should move to a method interface (see relevant Julia style guide section) which looks like

function basis end # current behavior throwing IncompatibleBases when basis_l(op:T) != basis_r(op:T) for T<:AbstractOperator
function basis_l end # defined for subtypes of AbstractOperator
function basis_r end # defined for subtypes of AbstractOperator
function bases end # defined for subtypes of CompositeBasis and SumBasis and gets the Vector of bases
function spinnumber end # defined for subtypes of SpinBasis
function cutoff end # defined for subtypes of FockBasis
function offset end # defined for subtypes of FockBasis

In addition all subtypes of Basis will be required to implement Base.:(==) and Base.size where the latter must return a tuple with tensor product decomposition of the Hilbert space that the basis defines so that Base.length(b::Basis) = prod(size(b)) gives the total dimension of the Hilbert space (this is currently the .shape field).

Since removing the basis type parameter from AbstractQObjType is meant to go along with moving basis compatibility checks to be entirely dynamic, I think that interface can be improved to look like:

• multiplicable(a::T,b::T) where {T<:AbstractQObjType} checks that they can be multiplied and multiplicable(a,a) checks that an operator is square.
• addible(a::T,b::T) where {T<:AbstractQObjType} checks that they can be added or that the objects can be compared for equality.
• check_* versions of the above with a @compatiblebases macro which disables them.

Finally to address the issue of superoperators and operator bases such as the pauli operator basis (#35, #36). I think I've unfortunately been fairly confused myself about the best way to handle all of this for awhile, but I think I'm hopefully thinking more clearly about it now. I think it's best to view superoperators as simply operators, while the input and output operators they map between are viewed as states corresponding to the vec'd operators.

Concretely this means that the SuperOperator type in QuantumOpticsBase should subtype AbstractOperator and we should introduce a new KetBraBasis <: Basis here. Then we can have that basis_{l,r}(sop::SuperOperator) isa KetBraBasis. More generally all the different types of superoperator representations are related by the different operator bases that they map between. So we can introduce appropriate subtypes of Basis to disambiguate left and right bases of SuperOperator, ChoiState, KrausOperators, PauliTransferMatrix, ChiMatrix. I think this will help a lot with code reuse between the operators*.jl, superoperators.jl and pauli.jl files since there's currently a lot of re-implemented functionality for dense and sparse representations between all those files. I think this also fits much better with Julia's type system. Some operations are the same between operators and superoperators and thus can be shared such as their basis checks. Other are different, such a multiplying two ChoiStates should probably represent the composition of their maps rather than the multiplication of their density matrices even though both are entirely valid things to do with those objects. Method specialization makes this easy to do.

Thoughts @Krastanov @amilsted @apkille?

[amilsted]

I think this looks generally sane to me. Though "QObj" is maybe a little too QuTiP/pythonesque, no? Not sure what else to use though :)

I really like removing the basis type parameters from the operators and states. IIRC we rarely dispatch on them in functions that take operators/states anyway. Perhaps we distinguish composite from single-site bases in some operator/state functions, but that can still be done using a "holy traits" pattern (or even just a simple isa branch in this case).

The superoperator approach also seems about right to me. That said, I'd want to be able to use the usual operator types (like LazySum and LazyTensor) for superoperators too. Is your proposal to have SuperOperator be a wrapper around an arbitrary AbstractOperator? Or for the usual operator types to be treated as superoperators if they are defined on a KetBraBasis? That would be another basis-dependent dispatch, which again could use the traits pattern?

[akirakyle]

Thanks for the comments @amilsted!

Though "QObj" is maybe a little too QuTiP/pythonesque, no?

Yep, happy to rename this to something better. Maybe just being verbose AbstractQuantumObjectType is fine, also perfectly fine doing without this convince const.

Perhaps we distinguish composite from single-site bases in some operator/state functions, but that can still be done using a "holy traits" pattern (or even just a simple isa branch in this case).

check_ptrace_arguments is already an example of this in QuantumOpticsBase and I'm imagining similar helper functions depending on what's necessary. I'm still not sure the best way to handle #27, but another possibility is to have every subtype of Basis implement nsubsystems and Base.getindex. This should be a sufficient interface for accessing composite systems which would be bit more flexible than the binary isa CompositeSystem. For example, sometimes you actually want a function to ensure there's only a given number of nsubsytems such as when creating Choi states.

Is your proposal to have SuperOperator be a wrapper around an arbitrary AbstractOperator? Or for the usual operator types to be treated as superoperators if they are defined on a KetBraBasis?

It's a little confusing since there's many representations of superoperators, while SuperOperator in QuantumOpticsBase is the "Liouville" representation between two vec'd operators (hence "KetBra" basis, where vec'ing just flips the bra into a ket). I usually take "superoperator" to mean the representation independent object as opposed to the "Liouville" representation. Since the "Liouville" superoperator representation shares all the same behavior as a DataOperator with respect to rules around addition/multiplication/tensoring/tracing, I think it makes sense to eventually make SuperOperator just a DataOperator defined on a left and right KetBraBasis, especially since so much of the dense and sparse functionality is duplicated between them currently.

To keep compatibility and since it's convenient to have an explicit type, I would make SuperOperator just a wrapper type for DataOperator with KetBraBasis, but have it instead follow all the same algebraic rules we currently have, such as * for application to a DataOperator (without this wrapper users would have to call something like vec on an operator before applying a DataOperator with KetBraBasis to it, otherwise it would throw IncompatibleBases). For other representations (such as kraus, choi, pauli), they would just directly subtype AbstractOperator. As they have different underlying representations of their data, manipulations like addition/multiplication/tensoring/tracing require different implementation on the underlying data, so there isn't any ability to share code between them.

[amilsted]

To keep compatibility and since it's convenient to have an explicit type, I would make SuperOperator just a wrapper type for DataOperator with KetBraBasis

Could it not wrap a more generic AbstractOperator on a KetBraBasis?

[amilsted]

I'm still not sure the best way to handle #27, but another possibility is to have every subtype of Basis implement nsubsystems and Base.getindex.

I think for #27 it would already help if the mul and add checks would see a 1-element CompositeBasis as compatible with the correct non-composite basis. Although maybe then we are in a situation where we need a promote_basis(a::CompositeBasis, b::AbstractBasis) = <check a ~ b and then return a>?

Btw, should we have an AbstractCompositeBasis in the type hierarchy? I am imagining maybe a LabeledCompositeBasis that adds site label strings, or similar.

[akirakyle]

Could it not wrap a more generic AbstractOperator on a KetBraBasis?

I'm not sure I follow? I think the point is that SuperOperator mirrors Operator with .data fields that are matrices and have dense <:Matrix and spares <:SparseMatrixCSC. I think these can be unified with a KetBraBasis. Likewise other subtypes of AbstractOperator with different representations such as such as LazyOperators can make wrappers for instances that have KetBraBasis to define more "natural" meanings for operations such as * for superoperator application.

[akirakyle]

I think for #27 it would already help if the mul and add checks would see a 1-element CompositeBasis as compatible with the correct non-composite basis

I think this might be easiest to handle by just appropriately defining Base.:(==)(b1::CompositeBasis, b2:Basis) and Base.:(==)(b1::CompositeBasis, b2:Basis) since it would then generally apply to other equality checks.

Btw, should we have an AbstractCompositeBasis in the type hierarchy? I am imagining maybe a LabeledCompositeBasis that adds site label strings, or similar.

Perhaps, but it might be easier to just add support for something like that directly to CompositeBasis? I'll reply to you in #43 for how I imagine that would be related to the idea there.