Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Magnetic reflectivity py #131

Merged
merged 69 commits into from
Sep 24, 2021
Merged
Show file tree
Hide file tree
Changes from 61 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
89b4a84
Suggested fix for theta_offset problem in PolarizedNeutronProbe
bmaranville Jun 8, 2021
922af79
trying to make python version of magnetic amplitude calculation
bmaranville Jul 8, 2021
f0c1bb9
working python magnetic_amplitude
bmaranville Jul 8, 2021
9ce32df
add oversample method that sets oversampling
bmaranville Jul 8, 2021
7a81a61
Merge branch 'theta_offset_fix_PNP' into magnetic_reflectivity_py
bmaranville Jul 8, 2021
4dae670
use numba AOT compilation to reduce startup time
bmaranville Jul 9, 2021
fdc21e1
update feedback if module not found
bmaranville Jul 9, 2021
5265cce
minor cleanup
bmaranville Jul 9, 2021
f4cc290
use jit rather than aot for magnetic calculation
Jul 9, 2021
efba989
magnetic numba code tweaks
Jul 9, 2021
1659b24
magnetic numba code tweaks
Jul 9, 2021
4bbd50a
numba doesn't recognize c16 as a numpy dtype, ironically
bmaranville Jul 21, 2021
50a0b76
update test to be compatible with newer matplotlib and magnetic_ampli…
bmaranville Jul 21, 2021
a7ca9bb
use type instead of string for dtype, making numba happy
Jul 22, 2021
d325c99
restructure magnetic calculation to simplify memory handling
pkienzle Jul 30, 2021
08d79ee
adding numba convolve_gaussian
bmaranville Aug 13, 2021
301e8c4
adding numba convolve_gaussian
bmaranville Aug 13, 2021
ac6cbcd
don't enable numba parallelization by default - use multiprocessing s…
bmaranville Aug 13, 2021
e877ccc
remove unused method and assert statement from convolve that is faili…
bmaranville Aug 13, 2021
e0f4da8
remove all asserts from numba convolve
bmaranville Aug 13, 2021
608ad75
handling of iteration to match c
bmaranville Aug 13, 2021
14a1d69
adding numba reflectivity_amplitude
bmaranville Aug 13, 2021
1f93230
fix pointer logic
bmaranville Aug 14, 2021
05d4bfa
fixed pointer arithmetic
bmaranville Aug 14, 2021
80a3ae4
adding numba version of contract_profile
bmaranville Sep 3, 2021
183e393
use numba version of contract_profile instead of reflmodule
bmaranville Sep 3, 2021
8850cf9
use numba rebin library
bmaranville Sep 7, 2021
130d2ae
add numba rebin
bmaranville Sep 7, 2021
8abbae7
remove signatures - let numba autodetect types
bmaranville Sep 7, 2021
755f32e
remove coercion to int
bmaranville Sep 7, 2021
ab15fc6
don't use assert for non-constant checks
bmaranville Sep 7, 2021
796ef31
remove some unneeded asserts
bmaranville Sep 7, 2021
a39b010
remove parts related to building C-extension
bmaranville Sep 7, 2021
52b73e0
no need to build and distribute binary wheels
bmaranville Sep 7, 2021
0cf0fe5
don't need to rebuild extension on run
bmaranville Sep 7, 2021
4e9db44
wheel is now pure python
bmaranville Sep 7, 2021
a9ae07d
use numba align_magnetic in test
bmaranville Sep 7, 2021
4070de3
get lengths from array inputs to align_magnetic instead of passing in
bmaranville Sep 7, 2021
f8e0c44
add rebin_counts_2D
bmaranville Sep 7, 2021
1fd8430
use numba rebin_counts_2D
bmaranville Sep 7, 2021
a26e926
fix rebin2d port
bmaranville Sep 7, 2021
68e74d9
update print statements - not used
bmaranville Sep 7, 2021
9d41eff
adding numba convolve_sampled
bmaranville Sep 7, 2021
eeef4d1
use numba convolve_sampled
bmaranville Sep 7, 2021
d2c850c
don't build docs for reflmodule - it has been removed
bmaranville Sep 7, 2021
72caab3
adding init for lib folder
bmaranville Sep 7, 2021
e5084c2
add lib folder to packages
bmaranville Sep 7, 2021
9436b3f
revert to reflmodule as import (now numba)
bmaranville Sep 8, 2021
de1053a
remove inline numba definitions, move to lib_numba
bmaranville Sep 8, 2021
4c4630e
update test with reflmodule move
bmaranville Sep 8, 2021
531e1e2
calculate length from arrays, instead of passing in
bmaranville Sep 8, 2021
f1ce7ea
lib to lib_numba
bmaranville Sep 8, 2021
8f9dfd6
adding new numba libs, and moving existing from lib
bmaranville Sep 8, 2021
1c23a4f
fix signature change
bmaranville Sep 8, 2021
b8a9499
pep8 formatting
bmaranville Sep 21, 2021
3c00215
revert mistaken sign changes
bmaranville Sep 21, 2021
f9d2a86
use numba calculate_u1_u3
bmaranville Sep 21, 2021
e1a1399
fix calling of calculate_u1_u3 (rhoM is modified in-place)
bmaranville Sep 21, 2021
7a1d70f
pep8
bmaranville Sep 21, 2021
44f23f6
pep8
bmaranville Sep 21, 2021
6ec0977
moving reflmodule.py to refllib.py
bmaranville Sep 21, 2021
fb10625
use refllib
bmaranville Sep 21, 2021
129ef9d
use refllib.py instead of reflmodule
bmaranville Sep 21, 2021
81ea454
remove leading underscores from refllib methods
bmaranville Sep 21, 2021
7299d92
build documentation for refllib
bmaranville Sep 21, 2021
498b6e0
remove underscores from function names
bmaranville Sep 21, 2021
d765385
missed one underscored refllib function
bmaranville Sep 21, 2021
269047b
use python3.7 as base version
bmaranville Sep 21, 2021
f808fac
Merge branch 'master' into magnetic_reflectivity_py
bmaranville Sep 21, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 3 additions & 23 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,8 @@ jobs:
strategy:
matrix:
config:
- { os: ubuntu-latest, py: 3.8, doc: 1 }
- { os: windows-latest, py: 3.6, whl: 1 }
- { os: windows-latest, py: 3.7, whl: 1 }
- { os: ubuntu-latest, py: 3.8, doc: 1, whl: 1 }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is 3.8 now required? Numpy is sitting at 3.7 as the oldest version, so ideally we would match that (though not important).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I was using 3.8 as the target for the dataclasses branch, but I don't remember why. I don't think it's important - 3.7 is fine.

- { os: windows-latest, py: 3.8, exe: 1, whl: 1 }
- { os: windows-latest, py: 3.9, whl: 1 }
- { os: macos-latest, py: 3.7, whl: 1 }
- { os: macos-latest, py: 3.8, whl: 1 }
- { os: macos-latest, py: 3.9, whl: 1 }
# all using to stable abi

steps:
Expand All @@ -37,7 +31,6 @@ jobs:
python -m pip install --upgrade pip
python -m pip install wheel setuptools
python -m pip install numpy scipy matplotlib bumps periodictable scikit-learn pytest pytest-cov numba
python setup.py build_ext --inplace
python setup.py build
mkdir release

Expand Down Expand Up @@ -96,14 +89,7 @@ jobs:
ls * -l
echo "WINDOWS_INSTALLER=$(ls release/*.zip)" >> $GITHUB_ENV
echo "SRC_DIST=$(ls dist/*.tar.gz)" >> $GITHUB_ENV
echo "WINDOWS_36_WHL=$(ls dist/*cp36*win*.whl)" >> $GITHUB_ENV
echo "WINDOWS_37_WHL=$(ls dist/*cp37*win*.whl)" >> $GITHUB_ENV
echo "WINDOWS_38_WHL=$(ls dist/*cp38*win*.whl)" >> $GITHUB_ENV
echo "WINDOWS_39_WHL=$(ls dist/*cp39*win*.whl)" >> $GITHUB_ENV
echo "MACOS_37_WHL=$(ls dist/*cp37*macos*.whl)" >> $GITHUB_ENV
echo "MACOS_38_WHL=$(ls dist/*cp38*macos*.whl)" >> $GITHUB_ENV
echo "MACOS_39_WHL=$(ls dist/*cp39*macos*.whl)" >> $GITHUB_ENV

echo "PY3_WHL=$(ls dist/*.whl)" >> $GITHUB_ENV

- name: Update current release
if: startsWith(github.ref, 'refs/tags')
Expand All @@ -112,13 +98,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
files: |
${{ env.WINDOWS_INSTALLER }}
${{ env.WINDOWS_36_WHL }}
${{ env.WINDOWS_36_WHL }}
${{ env.WINDOWS_36_WHL }}
${{ env.WINDOWS_36_WHL }}
${{ env.MACOS_37_WHL }}
${{ env.MACOS_38_WHL }}
${{ env.MACOS_39_WHL }}
${{ env.PY3_WHL }}

- name: publish distribution to Test PyPI
uses: pypa/gh-action-pypi-publish@master
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/unstable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ jobs:
python -m pip install wheel setuptools
python -m pip install numpy scipy matplotlib periodictable scikit-learn pytest pytest-cov numba
pip install git+https://github.com/bumps/bumps.git
python setup.py build_ext --inplace
python setup.py build
mkdir unstable

Expand All @@ -46,7 +45,7 @@ jobs:

- name: Build binary wheel
run: |
python setup.py bdist_wheel --py-limited-api=cp32
python setup.py bdist_wheel

- name: Build source distribution
run: |
Expand Down
1 change: 0 additions & 1 deletion doc/genmods.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@
('probe', 'Instrument probe'),
('profile', 'Model profile'),
('reflectivity', 'Reflectivity'),
('reflmodule', 'Low level reflectivity calculations'),
bmaranville marked this conversation as resolved.
Show resolved Hide resolved
('resolution', 'Resolution'),
('snsdata', 'SNS Data'),
('staj', 'Staj File'),
Expand Down
2 changes: 1 addition & 1 deletion extra/build_win_installer.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
# Look in https://www.python.org/ftp/python for the latest python version.
$PY_VERSION = "3.8.1"
$PTH_FILE = "python38._pth"
$WHEEL_TAG = "cp32-abi3-win_amd64"
$WHEEL_TAG = "py3-none-any"
$INSTALL_TAG = "exe" # PAK: renamed from "cp38-embedded-amd64" to make docs clearer
$PACKAGE = "refl1d"
$APP_NAME = "Refl1D"
Expand Down
2 changes: 1 addition & 1 deletion extra/build_win_installer_unstable.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
# Look in https://www.python.org/ftp/python for the latest python version.
$PY_VERSION = "3.8.1"
$PTH_FILE = "python38._pth"
$WHEEL_TAG = "cp32-abi3-win_amd64"
$WHEEL_TAG = "py3-none-any"
$INSTALL_TAG = "exe" # PAK: renamed from "cp38-embedded-amd64" to make docs clearer
$PACKAGE = "refl1d"
$APP_NAME = "Refl1D"
Expand Down
5 changes: 4 additions & 1 deletion refl1d/experiment.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,12 +110,15 @@ def update(self):

def residuals(self):
if 'residuals' not in self._cache:
# Trigger reflectivity calculation even if there is no data to
# compare against so that we can profile simulation code, and
# so that simulation smoke tests are run more thoroughly.
QR = self.reflectivity()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like an important change, outside the refactor with numba... I hope this is in the other branches too

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not super important. In the past I've forced the call directly within the model file, but this didn't work when profiling.

if ((self.probe.polarized
and all(x is None or x.R is None for x in self.probe.xs))
or (not self.probe.polarized and self.probe.R is None)):
resid = np.zeros(0)
else:
QR = self.reflectivity()
if self.probe.polarized:
resid = np.hstack([(xs.R - QRi[1])/xs.dR
for xs, QRi in zip(self.probe.xs, QR)
Expand Down
Empty file added refl1d/lib_numba/__init__.py
Empty file.
265 changes: 265 additions & 0 deletions refl1d/lib_numba/contract_profile.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,265 @@
import numba
import math

Z_EPS = 1e-6

ALIGN_MAGNETIC_SIG = 'i4(f8[:], f8[:], f8[:], f8[:], f8[:], f8[:], f8[:], f8[:], f8[:,:])'


# @numba.njit(ALIGN_MAGNETIC_SIG, parallel=False, cache=True)
@numba.njit(cache=True)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why no function signature?

If it is because the inputs are not of the correct time and we are relying on conversion during call, then I would prefer to trigger an error here and force the caller to produce the correct types.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for some of these there was explicit polymorphism encoded in the wrapper, and it seemed like a reasonable policy in numba to address that by not specifying a signature. I did some testing and was not able to see any performance difference between signature/no signature. For this function I suspect the usage of the variables is pretty unambiguous to numba.

def align_magnetic(d, sigma, rho, irho, dM, sigmaM, rhoM, thetaM, output_flat):
# ignoring thickness d on the first and last layers
# ignoring interface width sigma on the last layer
nlayers = len(d)
nlayersM = len(dM)
noutput = nlayers + nlayersM

# making sure there are at least two layers
if nlayers < 2 or nlayersM < 2:
raise ValueError("only works with more than one layer")

output = output_flat
magnetic = 0 # current magnetic layer index
nuclear = 0 # current nuclear layer index
z = 0.0 # current interface depth
next_z = 0.0 # next nuclear interface
next_zM = 0.0 # next magnetic interface
# int active = 3# active interfaces, active&0x1 for nuclear, active&0x2 for magnetic

k = 0 # current output layer index
while True:
# repeat over all nuclear/magnetic layers
if (k == noutput):
return -1 # exceeds capacity of output

# printf("%d: %d %d %g %g %g\n", k, nuclear, magnetic, z, next_z, next_zM)
# printf("%g %g %g %g\n", rho[nuclear], irho[nuclear], rhoM[magnetic], thetaM[magnetic])
# printf("%g %g %g %g\n", d[nuclear], sigma[nuclear], dM[magnetic], sigmaM[magnetic])

# Set the scattering strength using the current parameters
output[k][2] = rho[nuclear]
output[k][3] = irho[nuclear]
output[k][4] = rhoM[magnetic]
output[k][5] = thetaM[magnetic]

# Check if we are at the last layer for both nuclear and magnetic
# If so set thickness and interface width to zero. We are doing a
# center of the loop exit in order to make sure that the final layer
# is added.

if (magnetic == nlayersM-1 and nuclear == nlayers-1):
output[k][0] = 0.
output[k][1] = 0.
k += 1
break

# Determine if we are adding the nuclear or the magnetic interface next,
# or possibly both. The order of the conditions is important.
#
# Note: the final value for next_z/next_zM is not defined. Rather than
# checking if we are on the last layer we simply add the value of the
# last thickness to z, which may be 0, nan, inf, or anything else. This
# doesn't affect the algorithm since we don't look at next_z when we are
# on the final nuclear layer or next_zM when we are on the final magnetic
# layer.
#
# Note: averaging nearly aligned interfaces can lead to negative thickness
# Consider nuc = [1-a, 0, 1] and mag = [1+a, 1, 1] for 2a < Z_EPS.
# On the first step we set next_z to 1-a, next_zM to 1+a and z to the
# average of 1-a and 1+a, which is 1. On the second step next_z is
# still 1-a, so the thickness next_z - z = -a. Since a is tiny we can just
# pretend that -a == zero by setting thickness to fmax(next_z - z, 0.0).

if (nuclear == nlayers-1):
# No more nuclear layers... play out the remaining magnetic layers.
output[k][0] = max(next_zM - z, 0.0)
output[k][1] = sigmaM[magnetic]
magnetic += 1
next_zM += dM[magnetic]
elif (magnetic == nlayersM-1):
# No more magnetic layers... play out the remaining nuclear layers.
output[k][0] = max(next_z - z, 0.0)
output[k][1] = sigma[nuclear]
nuclear += 1
next_z += d[nuclear]
elif (math.fabs(next_z - next_zM) < Z_EPS and math.fabs(sigma[nuclear]-sigmaM[magnetic]) < Z_EPS):
# Matching nuclear/magnetic boundary, with almost identical interfaces.
# Increment both nuclear and magnetic layers.
output[k][0] = max(0.5*(next_z + next_zM) - z, 0.0)
output[k][1] = 0.5*(sigma[nuclear] + sigmaM[magnetic])
nuclear += 1
next_z += d[nuclear]
magnetic += 1
next_zM += dM[magnetic]
elif (next_zM < next_z):
# Magnetic boundary comes before nuclear boundary, so increment magnetic.
output[k][0] = max(next_zM - z, 0.0)
output[k][1] = sigmaM[magnetic]
magnetic += 1
next_zM += dM[magnetic]
else:
# Nuclear boundary comes before magnetic boundary
# OR nuclear and magnetic boundaries match but interfaces are different.
# so increment nuclear.
output[k][0] = max(next_z - z, 0.0)
output[k][1] = sigma[nuclear]
nuclear += 1
next_z += d[nuclear]

z += output[k][0]
k += 1

return k


CONTRACT_MAG_SIG = 'i4(f8[:], f8[:], f8[:], f8[:], f8[:], f8[:], f8)'


# @numba.njit(CONTRACT_MAG_SIG, parallel=False, cache=True)
@numba.njit(cache=True)
def contract_mag(d, sigma, rho, irho, rhoM, thetaM, dA):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this code is broken and the magnetic folk always run with dA=0. I've added a ticket.

n = len(d)
i = newi = 1 # /* Skip the substrate */
while (i < n):

# /* Get ready for the next layer */
# /* Accumulation of the first row happens in the inner loop */
dz = weighted_dz = 0
rhoarea = irhoarea = rhoMarea = thetaMarea = 0.0
rholo = rhohi = rho[i]
irholo = irhohi = irho[i]
maglo = maghi = rhoM[i] * math.cos(thetaM[i] * math.pi / 180.0)

# /* Accumulate slices into layer */
while True:
# /* Accumulate next slice */
dz += d[i]
rhoarea += d[i] * rho[i]
irhoarea += d[i] * irho[i]

# /* Weight the magnetic signal by the in -plane contribution
# * when accumulating rhoM and thetaM. */
weight = math.cos(thetaM[i]*math.pi/180.)
mag = rhoM[i]*weight
rhoMarea += d[i]*rhoM[i]*weight
thetaMarea += d[i]*thetaM[i]*weight
weighted_dz += d[i]*weight

# /* If no more slices or sigma != 0, break immediately */
i += 1
if (i == n or sigma[i-1] != 0.):
break

# /* If next slice exceeds limit then break */
if (rho[i] < rholo):
rholo = rho[i]
if (rho[i] > rhohi):
rhohi = rho[i]
if ((rhohi-rholo)*(dz+d[i]) > dA):
break

if (irho[i] < irholo):
irholo = irho[i]
if (irho[i] > irhohi):
irhohi = irho[i]
if ((irhohi-irholo)*(dz+d[i]) > dA):
break

if (mag < maglo):
maglo = mag
if (mag > maghi):
maghi = mag
if ((maghi-maglo)*(dz+d[i]) > dA):
break

# /* Save the layer */
assert(newi < n)
d[newi] = dz
if (i == n):
# /* Last layer uses surface values */
rho[newi] = rho[n-1]
irho[newi] = irho[n-1]
rhoM[newi] = rhoM[n-1]
thetaM[newi] = thetaM[n-1]
# /* No interface for final layer */
else:
# /* Middle layers uses average values */
rho[newi] = rhoarea / dz
irho[newi] = irhoarea / dz
rhoM[newi] = rhoMarea / weighted_dz
thetaM[newi] = thetaMarea / weighted_dz
sigma[newi] = sigma[i-1]
# /* First layer uses substrate values */
newi += 1

return newi


CONTRACT_BY_AREA_SIG = 'i4(f8[:], f8[:], f8[:], f8[:], f8)'


# @numba.njit(CONTRACT_BY_AREA_SIG, parallel=False, cache=True)
@numba.njit(cache=True)
def contract_by_area(d, sigma, rho, irho, dA):
n = len(d)
i = newi = 1 # /* Skip the substrate */
while (i < n):

# /* Get ready for the next layer */
# /* Accumulation of the first row happens in the inner loop */
dz = rhoarea = irhoarea = 0.0
rholo = rhohi = rho[i]
irholo = irhohi = irho[i]

# /* Accumulate slices into layer */
while True:
# /* Accumulate next slice */
dz += d[i]
rhoarea += d[i]*rho[i]
irhoarea += d[i]*irho[i]

# /* If no more slices or sigma != 0, break immediately */
i += 1
if (i == n or sigma[i-1] != 0.):
break

# /* If next slice won't fit, break */
if (rho[i] < rholo):
rholo = rho[i]
if (rho[i] > rhohi):
rhohi = rho[i]
if ((rhohi-rholo)*(dz+d[i]) > dA):
break

if (irho[i] < irholo):
irholo = irho[i]
if (irho[i] > irhohi):
irhohi = irho[i]
if ((irhohi-irholo)*(dz+d[i]) > dA):
break

# /* dz is only going to be zero if there is a forced break due to
# * sigma, or if we are accumulating a substrate. In either case,
# * we want to accumulate the zero length layer
# */
# /* if (dz == 0) continue; */

# /* Save the layer */
assert(newi < n)
d[newi] = dz
if (i == n):
# /* printf("contract: adding final sld at %d\n",newi); */
# /* Last layer uses surface values */
rho[newi] = rho[n-1]
irho[newi] = irho[n-1]
# /* No interface for final layer */
else:
# /* Middle layers uses average values */
rho[newi] = rhoarea / dz
irho[newi] = irhoarea / dz
sigma[newi] = sigma[i-1]
# /* First layer uses substrate values */
newi += 1

return newi
Loading