pvlib for Julia #2337
Replies: 9 comments 17 replies
-
|
Hi Stefan,
Fortune favors the bold! I have heard good things about Julia. 🙂
My new saying is: “Python is a great language. This includes great for
hanging oneself or one’s whole dev team.”
A few comments about design decisions in pvlib that I find problematic:
- The ModelChain wrapper does not seem to be easily/obviously extended with
new models and the “configuration discovery” ends up being more opaque than
convenient (IMHO). I’m very curious if the ModelingToolkit.jl would provide
a better solution here.
- It seems to have an “algorithm first” mindset over an “interface first”
mindset. Given the size and scope, both anre equally important IMHO. If I
want to drop in my own flavor of single-diode model, then it’s hard to know
what interface I must adopt to ensure integration with the ModelChain is
highly likely to be successful without a wandering into a bigger,
interface-breaking refactor. (I learned that there is a DC-model plug-in
mechanism in pvlib, but it just seems to get buried in the presentation.)
- Pandas goes too deep (again, IMHO), into the lowest levels of almost all
functions. Combining this with Python’s duck typing can make development
painful and error prone at worst and inelegant at best. Also, what if I
want to use polars instead? A time-stamped data provider interface would be
great. (To be fair, this is more of an issue in the greater scientific
Python community, but I wonder if Julia is more elegant/flexible here.)
- Typing. Gradual typing is nice, but for all that attention to typing
detail one doesn’t see any performance boost in Python. It’s kinda like
putting lipstick on a pig.
Another thing to consider: How might using Julia suffer from having a
smaller “surrounding ecosystem” than Python, such as web-server,
web-application, etc. packages. For example, can one write a lambda
function in AWS using Julia? If not, then can one instead compile and
deploy a Julia-based binary to AWS Lambda? (Maybe Julia has grown
significantly and is not so deficient here?)
Also, I’m curious how this effort might be taken on without
fracturing/diluting the open-software development side of the solar PV
modeling community. I know a major problem I have is finding time to
contribute to pvlib while I still have to pay a mortgage and raise kids.
For all of my misgivings about using Python in the large project that pvlib
has become, I still greatly appreciate the time commitment by the
maintainers, as well as Python’s popularity working to their advantage with
regards to open source contribution.
Happy to discuss further in this thread, and I may even have a real life
project or two to develop as test use cases. (I am very concerned about my
bandwidth, however.)
Mark Campanelli
LinkedIn <https://www.linkedin.com/in/markcampanelli/>
Intelligent Measurement Systems LLC
<https://intelligentmeasurementsystems.com>
Try PVfit <https://pvfit.app> today!
…On Fri, Dec 20, 2024 at 06:13 Stefan de Lange ***@***.***> wrote:
Hi!
I have been an active user of pvlib for a couple of years and really love
the project. I have used it both commercially and for personal projects.
Recently, I have moved most of my projects from Python to Julia because of
the modelling, scientific computing and autodifferentiation (AD)
ecosystems. Julia has some shortcomings of course, but for the most part I
don't regret the move.
Except for one bit, which is that pvlib is not supported in Julia. I can
get around that by calling python from Julia, but having a native Julia
solution would be much better. Julia and photovoltaic simulations seem like
a great match, so I made this post on the Julia discourse
<https://discourse.julialang.org/t/pvlib-for-julia/118952> a while back.
It got a little bit of traction and some discussions, and also @kevinsa5
<https://github.com/kevinsa5> responded. I have had a small discussion
with him, and he suggested opening a discussion here.
As to concrete plans: I would like to start a 'pvlib for Julia' from
scratch, written in Julia, but making use of the best ideas of pvlib as
much as possible. Pvlib has had a long history and I don't have to / want
to reinvent the wheel. However, I also don't want to have any cross-calling
from Julia to Python, that is not the goal of this project.
The core features I have in mind right now will be:
- Modularity. Everything will be built on top of ModelingToolkit.jl
<https://github.com/SciML/ModelingToolkit.jl>. ModelingToolkit is an
acausal modelling framework, such as Modelica. However, there is a much
larger emphasis on numerical algorithms and performance. The benefit of
this approach is that once components are built we don't have to write any
electrical equations by hand: they will be built symbolically, simplified
and then passed to (symbolic-) numerical algorithms for solving. This
enables very high flexibility when defining and configuring systems. For
example, I always found working with AC/DC optimizers in pvlib a bit of a
pain in the ass, this will now be possible.
- Native autodifferentiation support. This will enable gradient-based
optimization, sensitivity analysis, model-based machine learning etc. There
is a massive ongoing effort in the Julia community to harmonize
autodifferentiation across all packages, see JuliaDiff
<https://github.com/JuliaDiff/>. In Julia you write your code first,
and then you add autodifferentiation. This is distinctively different from
Python, where you first choose a package (JAX, PyTorch, ...) and then you
write your code in that framework. Hence I think it would be a very big
task to add autodifferentiation support to pvlib-python, while in Julia it
would be feasible. This is partly inspired by the paper dPV: An
End-to-End Differentiable Solar-Cell Simulator
<https://arxiv.org/abs/2105.06305> which implemented these ideas at
the cell level in JAX.
- Data analysis. Next to a simulation package it would be nice to have
a data analysis package with more advanced features based on recent
research efforts. This is inspired by Solar Data Tools
<https://solar-data-tools.readthedocs.io/en/main/>, which uses a
signal decomposition framework to support shading and soiling loss
estimations from unlabeled PV data.
- Speed. Julia is fast, very fast, and much faster than Python. That
means we no longer need any C backbones such as numpy. Everything can be
natively written in Julia, even the heavy numerical algorithms, and it will
be fast.
As I am just one person and can only work in my free time it doesn't make
much sense that I work on this in isolation. It would be much more
effective if it could be hosted under the umbrella of pvlib and lead by
someone with more experience than me. Hence why I wanted to start this
discussion here. I understand this is not a small effort, but now seems the
right time to start.
I am very keen to hear your thoughts and opinions!
ps. As to the current state, I am already able to simulate IV curves with
a basic single diode model defined in ModellingToolkit. Check the discourse
post linked above if you're curious.
—
Reply to this email directly, view it on GitHub
<#2337>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAISX4ZX6PNNOGG5WLH7LUT2GQJWTAVCNFSM6AAAAABT7DTMBSVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXG4YTSOJVG4>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks, great points!
Not necessarily. This is mostly a software design problem, having the right abstraction level and clearly defined interfaces is required to make it easy to extend the components of the ModelChain. This will be a matter of defining the right abstract types, their interfaces, and associated tests.
Completely agree with this. Pandas API is very unintuitive to me and working with indexes is a PITA. Actually, this was the main reason I decided to trash my last pvlib project. I even made an issue about it a while back, though in hindsight I didn't word it that tactfully: #1926
That's a valid concern, but for now I would say the focus should be on the modelling aspect, that's where Julia shines.
My hope is that by making PV simulations more accessible in Julia, the Julia community which mostly consists of scientists and engineers will want to start contributing as well. With proper software design we can make it easier to contribute, and perhaps we can reignite interest in some old pvlib contributors as well. |
Beta Was this translation helpful? Give feedback.
-
|
Splitting development effort is my main concern as well. I think the other maintainers would agree that pvlib-python already suffers from too little developer attention. I know I would want to get sucked into a fun new julia version of pvlib, and I am reluctant to sign on to taking that time away from pvlib-python. Especially when I already committed to maintaining solarfactors but have largely failed to deliver on that promise. |
Beta Was this translation helpful? Give feedback.
-
|
Okay let me clear that up then, I am not requesting any development effort from pvlib core developers. But I understand your concern, is there a possible compromise? To me there is a lot of value in having a |
Beta Was this translation helpful? Give feedback.
-
|
Interesting discussion and ideas. I think there are two distinct aspects to the proposal:
Is there a modelling framework in Python that could be used to fulfill a subset of the wish list? |
Beta Was this translation helpful? Give feedback.
-
|
There are some comparisons made with alternatives here: https://docs.sciml.ai/ModelingToolkit/stable/comparison/ I think one of the strengths of Modelingtoolkit.jl is that it integrates perfectly with the wider SciML/Julia ecosystem. Python packages generally live more in isolation. It would anyway be a very large effort to rewrite much of core pvlib functionality in a structural /symbolic modelling language, without much benefit. |
Beta Was this translation helpful? Give feedback.
-
|
Another point worth considering: I believe julia's compiler is nearing readiness for producing "general purpose" shared library binaries, along the lines of what you could produce with C++ or Rust. Once that functionality on the julia tooling side is ready, it means a julia port of pvlib could be "easily" wrapped and used in other programming languages. IIRC, we have heard from some companies that they like pvlib, but can't integrate python into their in-house software, so they either don't use pvlib at all, or have to reimplement their pvlib-based prototypes in whatever language their own software uses. Providing a "language-agnostic" version of pvlib by compiling to shared libraries could make it accessible to a substantial segment of users that are currently prevented from using it because of programming language barriers. Of course, it would take quite a lot of work to get to that point... |
Beta Was this translation helpful? Give feedback.
-
|
I had a long comment redacted from a few days ago but I was hesitant to publish it. It was exactly that point of shared libraries/ABI/bindings/foreign function interfaces or whatever you wanna call it. When I first saw Julia introductions I didn't have a clue that compiled shared libraries were a planned thing. In any case, I support whatever language that uses Apache Arrow (or other alternatives, if any) as the underlying data engine, and bindings are provided for other languages. I recommend having a look at it, as it supports 12 languages if I haven't forgotten how to count. https://arrow.apache.org/docs/index.html I dream with Python being the sandbox to try and configure simulations and then that heavy-duty computations are done in a resources-efficient language. All my support goes to shared libs. |
Beta Was this translation helpful? Give feedback.
-
|
Julia has always had See https://www.youtube.com/watch?v=MKdobiCKSu0 for an introduction to this topic. A lot of progress has been made since then. |
Beta Was this translation helpful? Give feedback.
-
|
@ViralBShah Thanks for the video reference :). Any work being done about a formal foreign function interface (FFI) to more readily generate bindings in a variety of languages (Python, Rust,...) to a AOT compiled and statically-linked Julia library (possibly even without the Julia runtime)? Also, I am always cautious about the inevitable costs of ignoring/hiding types and interfaces for shorter term convenience, which one could view both Julia's dynamic dispatch and Python's duck typing as being examples. (Pardon my very new understanding of Julia at this point.) Thus, it seems that if one is targeting the shared library use case in Julia, then it may greatly behoove one to do things such as defining and using some well scoped generic types as function arguments to essentially well-define and bound the interface and make compilation possible and efficient. Does that sound about right? |
Beta Was this translation helpful? Give feedback.
-
|
In Julia you don't have to use type annotations everywhere, generic functions that look similar to Python can often be inferred very well by the compiler given a set of input types. If your whole program graph infers well given some set of types at the entrypoint functions, then it's amenable to static compilation. For numerical libraries, this can often be achieved without too much effort because it's all numbers. I'm co-author of a data visualization library and there it's much harder to impossible to achieve perfect inference. In fact, much of the upper layer close to the user code is deliberately full of dynamic dispatch for maximum flexibility. The lower layers that are concerned with rendering are type stable where it matters to achieve high performance. As an example, if I want to render all objects in a scene, I might iterate over a small In the future, when static compilation of Julia code becomes more commonplace, new norms will emerge how libraries are written. Certain easy to make mistakes that make libraries not statically compilable will be weeded out across the ecosystem because people will want to compile more and more code. That doesn't mean that this has to hurt the dynamic Julia story. I think it will probably just mean untangling the parts that gain from type instabilities from those that don't need it. Currently, those are mixed together in many places just because it didn't really matter. Other than that, you don't really need a different code architecture from what's currently being done for the static compilation usecase. |
Beta Was this translation helpful? Give feedback.
-
|
I think you folks might be onto something here :). Using Julia instead of, say, Rust (or god forbid C/C++) for a binary-compatible PV "core computational library" might be a good way forward. This approach would stick with a scientific-computing friendly language (like Python), but with the added benefit of performant typing and compilation. In the longer run, Would this work for users that are otherwise currently blocked by Python when they move beyond prototype to production? Let's ask around! Does anyone have a sense as to what the bindings generation process looks like, say if we wanted to make a "tracer bullet" where some existing Python code in |
Beta Was this translation helpful? Give feedback.
-
|
I don't like to be a wet blanket, but here is a counter-argument to writing a core computation library in Julia in close cooperation with pvlib. I am wondering why DoE wouldn't say: If users cannot take pvlib-python to production, then why don't we refocus our resources back into SAM, which is (presumably) performant, has a lower-level SDK, and has Python bindings? |
Beta Was this translation helpful? Give feedback.
-
|
I think the focus of SAM is different. I got this slide from PVPMC 2023 - Nate Blair, NREL. NB: I have used the graphical version of SAM, but I have never use PySAM.
|
Beta Was this translation helpful? Give feedback.
-
Would you mind listing these here or linking to another discussion if you already did this elsewhere? |
Beta Was this translation helpful? Give feedback.
-
|
For me the negatives of switching were:
|
Beta Was this translation helpful? Give feedback.
-
|
@langestefan Ok, so when Julia encounters a new type argument to a “polymorphically” defined function, it must somehow determine if that type has a compatible interface before the function can be compiled. Do you know of a good reference that explains this process (to a newbie)? |
Beta Was this translation helpful? Give feedback.
-
|
Not entirely sure what you mean. Are you asking about multiple dispatch? |
Beta Was this translation helpful? Give feedback.
-
|
Your question answered my question :). I'm such a Julia newbie that I didn't even know it was called "multiple dispatch". I quickly found this: https://discourse.julialang.org/t/understanding-multiple-dispatch/76601 A question that comes to me is if multiple dispatch can be extended to the aforementioned "general purpose" shared library binaries? (I would be very surprised if it could be done simply.) That is, does one have to somehow nail down the types when generating said binaries? |
Beta Was this translation helpful? Give feedback.
-
|
For the new What you can do is have type generic numerical code and then specify an entrypoint each for Float64, Float32, etc. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks at @jkrumbiegel. This anticipated much of my question to @ViralBShah above. I’m wondering if you would have anything more to add to that thread? |
Beta Was this translation helpful? Give feedback.
-
|
The main question here isn't about the potential benefits of using Julia and all the other good stuff. Any group of enthusiasts can get together and implement this at any time. The main question appears to be whether this should be under the existing pvlib umbrella. So perhaps we should be discussing the pros and cons of that aspect a little more.
In my opinion pvlib is doing a great job of providing well-documented reference implementations of models and algorithms and having them be efficient seems secondary. If a company needs a more efficient implementation in python or another language they can invest a little money and get that. I suspect they would only need to implement a small subset of pvlib anyway, and they can still benefit greatly from having the pvlib documentation and can use the "inefficient" version for testing. |
Beta Was this translation helpful? Give feedback.
-
Agreed, we can always start a separate organization with anyone who is interested. Considering pros and cons is more difficult, I am not in the pvlib org so I can't comment from that perspective. The name pvlib is known in the solar modelling community, so if the project were to be hosted here that would be a clear advantage. Hosting a
The efficiency argument is just one part, I named more reasons in my original post. The bare minimum we get out of this is a package which would make complex photovoltaic simulations possible in pure Julia, that is something that currently doesn't exist and so it has value in its own right. The consequences and advantages of building a pvlib from scratch in pure Julia are much broader. The goal is absolutely not to simply build 'a more efficient pvlib'. I am also not claiming that pvlib is slow, not at all. However, I do want to make one comment on this. The popularity of a class of machine learning methods called 'physics informed neural networks' (PINNs) is exploding due to being able to generalize extremely well. It's essentially data-driven physics. These methods minimize a loss function iteratively, and thus would run millions of simulations in succession in order to train a neural network. Having an efficient differentiable simulator is crucial here. That is absolutely something we could support in a
I do not consider |
Beta Was this translation helpful? Give feedback.
-
To me this is a compelling argument in favor. I worry a little about keeping pvlib-python and pvlib-julia in sync. |
Beta Was this translation helpful? Give feedback.
-
|
Monte-Carlo sampling is another area where computational efficiency can make a real difference. Furthermore, Julia seems much more friendly “out of the box” to concurrent computation. Putting on my “self-interest” hat here: As a small business owner I cannot greatly afford the time (or to hire additional software engineers) to translate pvlib-python into, say, Rust so that it runs fast enough for revenue-generating use cases. I don’t mean to demean SAM in the least, but SAM seems to be a bit of a slower moving “C++ beast”, and C++ is just not a “natural” language for physics and engineering. Nor is it anywhere as foolproof and safe as Rust. However, if I were to be able to donate my time to PVSim.jl and get low-cost, thoroughly-tested, production-ready, and callable-from-Rust (or Python!) binaries directly out of it as “compensation”, then I’d see that as a pretty good return on investment. |
Beta Was this translation helpful? Give feedback.
-
|
It's very easy to insert a ModelingToolkit.jl model into a Monte-Carlo sampling framework such as Turing.jl: https://turinglang.org/docs/tutorials/10-bayesian-differential-equations/ I have also used the much more efficient Bayesian message passing framework RxInfer.jl but it is a bit more tricky to use. |
Beta Was this translation helpful? Give feedback.
-
Hi!
I have been an active user of pvlib for a couple of years and really love the project. I have used it both commercially and for personal projects.
Recently, I have moved most of my projects from Python to Julia because of the modelling, scientific computing and autodifferentiation (AD) ecosystems. Julia has some shortcomings of course, but for the most part I don't regret the move.
Except for one bit, which is that pvlib is not supported in Julia. I can get around that by calling python from Julia, but having a native Julia solution would be much better. Julia and photovoltaic simulations seem like a great match, so I made this post on the Julia discourse a while back. It got a little bit of traction and some discussions, and also @kevinsa5 responded. I have had a small discussion with him, and he suggested opening a discussion here.
As to concrete plans: I would like to start a 'pvlib for Julia' from scratch, written in Julia, but making use of the best ideas of pvlib as much as possible. Pvlib has had a long history and I don't have to / want to reinvent the wheel. However, I also don't want to have any cross-calling from Julia to Python, that is not the goal of this project.
The core features I have in mind right now will be:
As I am just one person and can only work in my free time it doesn't make much sense that I work on this in isolation. It would be much more effective if it could be hosted under the umbrella of pvlib and lead by someone with more experience than me. Hence why I wanted to start this discussion here. I understand this is not a small effort, but now seems the right time to start.
I am very keen to hear your thoughts and opinions!
ps. As to the current state, I am already able to simulate IV curves with a basic single diode model defined in ModellingToolkit. Check the discourse post linked above if you're curious.
Beta Was this translation helpful? Give feedback.
All reactions