What kind of analysis is necessary for deciding on an adaptive design scheme?

[sgbaird]

@ancarnevali @anthony-wang @Kaaiian @chc273 @AndrewFalkowski @ardunn @Ryan-Rhys @mkhorton @SurgeArrester @CompRhys @ppdebreuck @blokhin @amorehead @ml-evs @janosh (name order randomly sorted 😉)

Based on recent discussions, I'd be interested to hear your thoughts on this. I think the DiSCoVeR algorithm (this work) is a relatively novel approach to adaptive design; however, it hasn't actually been put "through the ringer" to see how it compares to other adaptive design schemes, except for some basic comparisons against one of sklearn-s novelty algorithms, LocalOutlierFactor (see the 2nd panel plot in https://mat-discover.readthedocs.io/en/latest/figures.html#adaptive-design-comparison), for which the results will probably change once I figure out how to extract the unscaled densities.

What kind of analysis/comparisons to specific techniques would you need to see in order to evaluate whether the DiSCoVeR algorithm would be "trustworthy" enough or not to implement in your own workflow (e.g. expensive DFT simulations, wetlab experiments). I use the word trustworthy, because to me it seems like a leap of faith to choose an algorithm and potentially spend hundreds or thousands of hours with the hope that said algorithm is maximizing (the output of) the time you spend setting up and waiting for simulations and wetlab experiments. Biased answers are very welcome.

[blokhin]

What kind of analysis/comparisons to specific techniques would you need to see in order to evaluate whether the DiSCoVeR algorithm would be "trustworthy" enough or not to implement in your own workflow

please could you re-phrase? Generally I'd suggest sticking to the standard prediction quality metrics: MAE, R2, etc.

with the hope that said algorithm is maximizing the time you spend setting up and waiting for simulations and wetlab experiments

minimizing?

[sgbaird]

Thanks for joining in @blokhin!

please could you re-phrase? Generally, I'd suggest sticking to the standard prediction quality metrics: MAE, R2, etc.

Certainly. To elaborate:

• Is there a particular type of validation study, benchmark, or database that would convince you to use one adaptive design algorithm over another? (This can be specific to your interests)
• For example, would you be more persuaded by a comparison using a list of predefined candidates with real calculations/measurements or a fictitious validation model?
• Does the trust come primarily from a real validation study? (i.e. successful, published results from using the algorithm)
• Are there particular techniques that you'd want to see included in the adaptive design (e.g. Bayesian optimization) and any specific details (e.g. Gaussian process with ARD using mat2vec featurization and expected improvement acquisition function)?
• Any comments on what would make you skeptical of claimed success/superiority of one adaptive design workflow over another?

minimizing?

Woops 😅 I was thinking "maximizing output". Definitely minimizing the amount of time spent. Good catch

[chc273]

While I love the DiSCoVeR algorithm and idea, I am generally not convinced on the use of only composition to guide materials design. I understand it generally works for certain properties, for example elasticity where good materials are typically refractory elements with C/B/N, and also experiments only use composition as guidance. I argue the success are due to the fact we are not exploring large enough space beyond known materials. Examples such as graphite vs diamond makes me think if there is better alternative. For example, on top of composition, can we add more contrains like density range of the materials?

Also eventually before we make any property predictions, we will need to assess the synthesizabilty first, e.g. https://www.nature.com/articles/d41586-019-00676-y

Those are my concerns when dealing with new models/algorithms

[janosh]

I was going to reply along the same lines. ML is difficult enough to trust as it is even if you give it all the available data. With composition only as input, it feels like going out on a limb. Of course sometimes composition is all you have but then I'd prob shy away from the problem.

[sgbaird]

@chc273 and @janosh, thanks for your comments! If mat_discover had an option to swap out CrabNet (composition-based regression model) with a state-of-the-art structural model, and to swap the compositional novelty distance with a crystallographic novelty distance (#33) what concerns might be left? (@chc273 addressing your other points in separate replies)

[sgbaird]

@chc273 think you bring up a great point about composition-only design. One other topic is the materials design of alloys; the options are kind of limited when it comes to composition (i.e. the featurization stays the same, which is good for the transferability of models but bad because the algorithm doesn't have that info about it being an alloy, fractional prevalence of phases, etc.). With the right characterization (or assumptions), I think there are some really interesting paths that could be taken with structural models in alloy design spaces.

I argue the success are due to the fact we are not exploring large enough space beyond known materials.

Could you clarify this? Success of composition-based models (or did you mean lack of success)? Not large enough meaning mostly living in "Materials Project space"? (which is a great space, granted)

[sgbaird]

@chc273 Interesting point about density range. Experimentally (and computationally), density is something that would have to be measured after the synthesis/calculation, respectively. I could see that taking on a couple of forms:

• Calculate theoretical density using DFT and use as a pre-synthesis screening step
• Predict theoretical density using ML and use as a pre-synthesis screening step
• Measure experimental density as a post-synthesis screening step: if the density or some other easily measurable property falls outside of the desired range, don't continue with additional expensive characterization steps (e.g. post arc-melting, but before doing XRD and property measurements)

Do those seem in line with what you were thinking, or did I miss something?

[sgbaird]

@chc273 and others that can chime in,

Also eventually before we make any property predictions, we will need to assess the synthesizabilty first, e.g. https://www.nature.com/articles/d41586-019-00676-y

Definitely agreed about synthesizability; and the reference made for a nice read. Are there particular synthesizability screening routes that would improve or decrease your trust in the claim of success/superiority of an adaptive design validation? Any specific details? For example:

• SMACT (i.e. generate candidates that meet charge neutrality and electronegativity screening criteria)
• DFT-calculated e_above_hull or decomposition energy screening (e.g. via VASP)
• ML-predicted e_above_hull or decomposition energy screening (e.g. via TestStabilityML)
• Crystal-likeness score as a surrogate for synthesizability (e.g. Synthesizability-PU-CGCNN)
• structures produced via GANs (e.g. https://github.com/kaist-amsg/Composition-Conditioned-Crystal-GAN, though not sure if anyone other than the author has been able to get the code running yet 😅)
• Whether the compound has been experimentally realized before (e.g. ICSD tag, see also Literature references associated with chemical formulas (or compounds))
• solid solution substitution synthesis suggestions (e.g. piro [1] [2])

[ardunn]

@ancarnevali @anthony-wang @Kaaiian @chc273 @AndrewFalkowski @ardunn @Ryan-Rhys @mkhorton @SurgeArrester @CompRhys @ppdebreuck @blokhin @amorehead @ml-evs @janosh (name order randomly sorted 😉)

Based on recent discussions, I'd be interested to hear your thoughts on this. I think the DiSCoVeR algorithm (this work) is a relatively novel approach to adaptive design; however, it hasn't actually been put "through the ringer" to see how it compares to other adaptive design schemes, except for some basic comparisons against one of sklearn-s novelty algorithms, LocalOutlierFactor (see the 2nd panel plot in https://mat-discover.readthedocs.io/en/latest/figures.html#adaptive-design-comparison), for which the results will probably change once I figure out how to extract the unscaled densities.

What kind of analysis/comparisons to specific techniques would you need to see in order to evaluate whether the DiSCoVeR algorithm would be "trustworthy" enough or not to implement in your own workflow (e.g. expensive DFT simulations, wetlab experiments). I use the word trustworthy, because to me it seems like a leap of faith to choose an algorithm and potentially spend hundreds or thousands of hours with the hope that said algorithm is maximizing (the output of) the time you spend setting up and waiting for simulations and wetlab experiments. Biased answers are very welcome.

Probably the most useful indicator would be taking a previous screening/ML study and using their results (all the way to synthesis) to evaluate your algorithm using only the info that study had a priori. I.e., can your algorithm improve on their exhaustive search with no extra a priori knowledge?

[ardunn]

For reference, I'm taking this idea from a previous publication of ours https://iopscience.iop.org/article/10.1088/2515-7639/ab0c3d where we did this (sort of) for two problems (solar water splitting perovskites and superhard materials)

[ardunn]

Though the core question really is whether something like Mat-Discover can find new materials outside all expected_ spaces. Testing this is a much harder and complex question; the only way I can think of doing it earnestly is actually exploring an entirely novel space of compounds and evaluating your results.

[sgbaird]

@ramseyissa @truptimohanty