Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle npz files #865

Open
wants to merge 147 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
147 commits
Select commit Hold shift + click to select a range
c74a30e
Format windows ci file
minhqdao Aug 2, 2024
39df36b
Install fypp from msys2 package, clean up file
minhqdao Aug 2, 2024
ca8fc47
Only run on push
minhqdao Aug 2, 2024
abcd256
Clean up CI, run all on push only
minhqdao Aug 2, 2024
cf1cfe1
Bump to macos and intel-classic version
minhqdao Aug 2, 2024
557e65e
Revert to macos-12
minhqdao Aug 2, 2024
cefec39
Fix typos
minhqdao Aug 3, 2024
6eb9f3d
Reorder modules
minhqdao Aug 3, 2024
f556f18
Rename argument to avoid name clash
minhqdao Aug 3, 2024
b65f164
Remove whitespace
minhqdao Aug 3, 2024
bc3c551
Not use generics but a submodule instead to hopefully make all compil…
minhqdao Aug 3, 2024
9ae364c
Only allocate once
minhqdao Aug 3, 2024
21eee02
Commenting out a bit, it should build
minhqdao Aug 3, 2024
9aec6d0
Comment out even more
minhqdao Aug 3, 2024
30e20bf
Comment out the entire thing
minhqdao Aug 3, 2024
f59956b
Revert "Comment out the entire thing"
minhqdao Aug 3, 2024
9dfee6f
Revert "Comment out even more"
minhqdao Aug 3, 2024
d36ef05
Revert "Commenting out a bit, it should build"
minhqdao Aug 3, 2024
1353893
Revert "Only allocate once"
minhqdao Aug 3, 2024
be3603b
Revert "Not use generics but a submodule instead to hopefully make al…
minhqdao Aug 3, 2024
ad534b8
Just comment out that call
minhqdao Aug 3, 2024
7ad213b
Comment out the generics
minhqdao Aug 3, 2024
5ea9739
Reenable type-bound method
minhqdao Aug 4, 2024
1c19820
Remove msys, re-enable both push and pull_request and comment out met…
minhqdao Aug 4, 2024
070fc19
Remove on pull_request otherwise it runs twice
minhqdao Aug 4, 2024
9deca6c
Just inline the process
minhqdao Aug 4, 2024
4195544
Revert some changes
minhqdao Aug 4, 2024
597721c
Add unzip command
minhqdao Aug 6, 2024
60b116e
Allocate cmdmsg
minhqdao Aug 6, 2024
d4e6618
Add tests for non-zip and empty zip files
minhqdao Aug 6, 2024
5f6ccb8
Remove unnecessary code
minhqdao Aug 6, 2024
22f5f37
Add list files in zip and add tests
minhqdao Aug 6, 2024
8d3fee2
Fix list_files_in_zip and add test for zip file containing an empty file
minhqdao Aug 7, 2024
0682b82
Add test for zip file containing file empty.txt
minhqdao Aug 7, 2024
44946cc
Use textfile with sth in it
minhqdao Aug 7, 2024
540e8d5
Add simple npz file
minhqdao Aug 7, 2024
00d3d75
Add unzip msys dependency
minhqdao Aug 7, 2024
b9d19fc
Try identify error
minhqdao Aug 7, 2024
fd33fbf
Remove /
minhqdao Aug 7, 2024
58ab58e
Remove prints
minhqdao Aug 7, 2024
66d959d
Try adding shell
minhqdao Aug 7, 2024
08abff8
Remove again because it default exists
minhqdao Aug 7, 2024
aee5712
Try adding some script
minhqdao Aug 7, 2024
acab595
Use shell script
minhqdao Aug 7, 2024
b0e42cb
Try cpp variant
minhqdao Aug 7, 2024
95e979d
Remove list_files
minhqdao Aug 8, 2024
c703175
Use fypp
minhqdao Aug 8, 2024
b2f2b95
Revert "Use fypp"
minhqdao Aug 8, 2024
37be446
Add compiler options
minhqdao Aug 8, 2024
b2489ac
Add test for zip file that contains two files
minhqdao Aug 8, 2024
a966602
Apply compiler properties to indiviual files only to hopefully reduce…
minhqdao Aug 9, 2024
2a301f4
Extract methods to stdlib_filesystem.f90
minhqdao Aug 9, 2024
292e1af
Extract list_files into stdlib_filesystem
minhqdao Aug 9, 2024
1dbb325
Start adding tests for stdlib_filesystem
minhqdao Aug 9, 2024
762cbc4
Add tests for run
minhqdao Aug 9, 2024
8a08ec2
Rename to list_dir_contents and add tests
minhqdao Aug 9, 2024
acd89f7
Use test_failed
minhqdao Aug 10, 2024
e822047
Improve naming
minhqdao Aug 10, 2024
0d9b7c1
Only have single level of directories
minhqdao Aug 10, 2024
4777c75
Fix procedure name
minhqdao Aug 10, 2024
34a3ae6
Create temp file if it not exists
minhqdao Aug 10, 2024
94061fd
Fix error handling and define api for array loading
minhqdao Aug 10, 2024
bb6108f
Rename npy tests to np
minhqdao Aug 10, 2024
2c95d85
Only use custom error bc it's currently gibberish, not use error_stop…
minhqdao Aug 11, 2024
9c2e691
Add more tests
minhqdao Aug 11, 2024
625f59b
Add forgotten file
minhqdao Aug 11, 2024
2cd0144
Try redirected import to satisfy compiler
minhqdao Aug 11, 2024
98bd3d4
Use preprocessor to set the correct path to the test files
minhqdao Aug 11, 2024
7533cb2
Add first test, configure cpp and fix some errors
minhqdao Aug 12, 2024
cb1fa2f
Uncomment code and actually test it
minhqdao Aug 12, 2024
9acd33e
Add test that contains values
minhqdao Aug 12, 2024
b446715
Add int example
minhqdao Aug 12, 2024
cb63bab
Add example including complex numbers
minhqdao Aug 12, 2024
53d2036
Use complex64 to satisfy MinGW32
minhqdao Aug 13, 2024
31d54b8
Use test for npz file containing two arrays
minhqdao Aug 13, 2024
dbf5e7b
Add test for a compressed npz file
minhqdao Aug 13, 2024
46acdb9
Inject tmp folder so tests don't use the same one
minhqdao Aug 13, 2024
a91f083
Add zip compression with tests
minhqdao Aug 13, 2024
5fcd7d9
Add zip as msys2 package
minhqdao Aug 13, 2024
cc633e0
Implement save_npz
minhqdao Aug 13, 2024
1133a59
Use redirected import to satisfy intel once more
minhqdao Aug 13, 2024
ad6e96e
Improve tests by checking error, add a test for npz_write
minhqdao Aug 14, 2024
b6f2f69
Implement add_array
minhqdao Aug 14, 2024
1aacaa9
Extract arr_size to satisfy Intel
minhqdao Aug 14, 2024
aece66a
Make array name optional such as in np
minhqdao Aug 14, 2024
d5fa8ab
Finish first proper test for add_array
minhqdao Aug 14, 2024
6cbf989
Remove redundant interface
minhqdao Aug 14, 2024
fdae839
Not reallocate array
minhqdao Aug 14, 2024
7354010
Finish seconds test
minhqdao Aug 14, 2024
0854521
Add more tests for add_arr
minhqdao Aug 14, 2024
ecbb349
Add option to either compress zip file or not
minhqdao Aug 14, 2024
1403359
Finalize all tests
minhqdao Aug 15, 2024
9c56d6d
Fix rank
minhqdao Aug 15, 2024
6cf0960
Try fixing errors
minhqdao Aug 15, 2024
20ab3aa
Fix paths, print expected values
minhqdao Aug 15, 2024
c803c48
Add prints
minhqdao Aug 15, 2024
bd3a93e
Try allocate differently, run zip quietly
minhqdao Aug 15, 2024
53cf786
Fix typo
minhqdao Aug 15, 2024
c3d370a
Run unzip quietly
minhqdao Aug 15, 2024
e0057fc
Add some more prints
minhqdao Aug 15, 2024
7b1e1dc
Where is the comma
minhqdao Aug 15, 2024
4803954
Remove prints, initialize stat to 0
minhqdao Aug 15, 2024
6e6fa4b
Print stuff
minhqdao Aug 15, 2024
5035d21
Properly read value
minhqdao Aug 15, 2024
3beff30
Print
minhqdao Aug 15, 2024
36b7ad7
Use other array notation
minhqdao Aug 15, 2024
0c0c4a1
Maybe it needs allocation
minhqdao Aug 15, 2024
c5134f2
Use another notation
minhqdao Aug 15, 2024
bdbd9c9
Let's use a constant literal
minhqdao Aug 15, 2024
ded2c3c
Use proper allocates in add_array
minhqdao Aug 15, 2024
a8d1181
Print in add_array
minhqdao Aug 15, 2024
473f193
Print properly
minhqdao Aug 16, 2024
feb93f5
Print the most important part
minhqdao Aug 16, 2024
ac91b2f
Print before and after adding wrapper to array
minhqdao Aug 16, 2024
fd426c7
Invert addition and allocation
minhqdao Aug 16, 2024
666bd51
Use tmp_array for allocation instead
minhqdao Aug 16, 2024
d538078
Add examples
minhqdao Aug 16, 2024
96a49a2
Relocate add_array to stdlib_array bc that's where it belongs
minhqdao Aug 16, 2024
5b3506d
Set working directory directly in cmake
minhqdao Aug 16, 2024
751d495
Print pwd
minhqdao Aug 17, 2024
9e9c76d
Fix unzip and check for file existence
minhqdao Aug 17, 2024
eb4db69
Create temp dir
minhqdao Aug 17, 2024
69d4798
Move example npz file to io
minhqdao Aug 17, 2024
acea2a3
Revert ci changes
minhqdao Aug 21, 2024
d0cbbbc
Revert CI changes all the way
minhqdao Aug 21, 2024
b2ac7de
Indent correctly
minhqdao Aug 21, 2024
e91d25f
Try venv
minhqdao Aug 21, 2024
7f599e3
Use -m
minhqdao Aug 21, 2024
cfcfe96
Use correct pipe symbol
minhqdao Aug 21, 2024
cb2bbf3
Source
minhqdao Aug 21, 2024
aeacc54
Do in one go
minhqdao Aug 21, 2024
4c39f57
Align with current state of master
minhqdao Aug 21, 2024
c873d1b
Install unzip and zip dependencies
minhqdao Aug 22, 2024
ca38bef
Revert "Revert to macos-12"
minhqdao Aug 22, 2024
239a670
Revert "Bump to macos and intel-classic version"
minhqdao Aug 22, 2024
dec06a4
Revert "Clean up CI, run all on push only"
minhqdao Aug 22, 2024
9b17ae1
Merge branch 'main' into load-npz
minhqdao Aug 22, 2024
619b655
Remove file that was added by accident
minhqdao Aug 22, 2024
78e7b9f
Actually use values in example instead of just printing them
minhqdao Aug 22, 2024
d41ee29
Add get_values for better user experience
minhqdao Aug 22, 2024
5d7789a
Simplify example
minhqdao Aug 22, 2024
a22f401
Rename types and add some documentation
minhqdao Aug 22, 2024
92aca11
Add some docs
minhqdao Aug 22, 2024
99bbef0
Change stdlib_filesystem to stdlib_io_filesystem
minhqdao Aug 22, 2024
6aa19aa
Add documentation for stdlib_array
minhqdao Aug 23, 2024
7b76756
Add documentation
minhqdao Aug 23, 2024
b4e7c3a
Change stat to iostat
minhqdao Aug 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci_windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ jobs:
mingw-w64-${{ matrix.arch }}-python-fypp
mingw-w64-${{ matrix.arch }}-cmake
mingw-w64-${{ matrix.arch }}-ninja
unzip
zip

- run: >-
PATH=$PATH:/mingw64/bin/ cmake
Expand Down
123 changes: 121 additions & 2 deletions doc/specs/stdlib_array.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,58 @@ title: array

Module for index manipulation and array handling tasks.

## Procedures and methods provided
## Derived types provided
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this PR. Could you open a PR specific for the additions related to stdlib_array, please? It will make the review of this PR easier. Let me know if you need help.


### `array_wrapper_type`

A derived type that wraps a polymorphic `array_type` and helps with its allocation. By loading an npz file with `load_npz`, a list of array wrappers will be obtained. On the other hand, a list of array wrappers can be saved to an npz file using `save_npz`. Use `add_array` to add an array to a list of array wrappers and call `get_values` on the array wrapper to obtain the values of the underlying array.

#### Status

Experimental

#### Example

```fortran
program npz_example
use stdlib_array, only: array_wrapper_type, add_array
use stdlib_io_np, only: save_npz, load_npz
implicit none

type(array_wrapper_type), allocatable :: input_arrays(:), output_arrays(:)
real :: x(3, 2) = 1
integer :: y(2, 3) = 2
real, allocatable :: x_out(:,:)
integer, allocatable :: y_out(:,:)

call add_array(input_arrays, x)
call add_array(input_arrays, y)

call save_npz('example_save.npz', input_arrays)

call load_npz('example_save.npz', output_arrays)

if (size(input_arrays) /= 2) then
print *, 'Error: Output array has unexpected size.'; stop
end if

call output_arrays(1)%get_values(x_out)
call output_arrays(2)%get_values(y_out)

print *, x_out
print *, y_out
end
```

### `array_type`

An abstract type that can be extended according to the type and rank of the stored array. It is usually not necessary to interact with this type directly. It is used to store multiple arrays of different types and ranks in a single array.

#### Status

Experimental

## Procedures and methods provided

### `trueloc`

Expand Down Expand Up @@ -49,7 +99,6 @@ Returns an array of default integer size, with a maximum length of `size(array)`
{!example/array/example_trueloc.f90!}
```


### `falseloc`

#### Status
Expand Down Expand Up @@ -85,3 +134,73 @@ Returns an array of default integer size, with a maximum length of `size(array)`
```fortran
{!example/array/example_falseloc.f90!}
```

### `add_array`

#### Status

Experimental

#### Description

Add an array of defined type and rank to a list of array wrappers.

#### Syntax

`call ` [[stdlib_array(module):add_array(interface)]] `(arrays, array[, stat, msg, name])`

#### Class

Pure subroutine.

#### Arguments

`arrays`: List of array wrappers of type `array_wrapper_type` to add `array` to. This argument is `intent(inout)`.

`array`: Array with defined type and rank to be added to the list of array wrappers. This argument is `intent(in)`.

`stat`: Status variable of type `integer`. This argument is `optional` and `intent(out)`. The operation is successful if `stat` is `0`.

`msg`: Error message. This argument is `optional` and `intent(out)`.

`name`: Name of the array. This argument is `optional` and `intent(in)`. If not provided, the name will be set to the default value.

#### Examples

```fortran
{!example/io/example_save_npz.f90!}
```

### `get_values`

#### Status

Experimental

#### Description

Get the values of the array within the array wrapper.

#### Syntax

`call ` [[stdlib_array(module):array_wrapper_type(type)]] `%` [[array_wrapper_type(type):get_values(bound)]] `(wrapper, values[, stat, msg])`

#### Class

Pure subroutine.

#### Arguments

`wrapper`: Array wrapper of type `array_wrapper_type` to get the values from. This argument is `intent(in)`.

`values`: Array of the same type and rank as the array within the array wrapper. This argument is `intent(out)`.

`stat`: Status variable of type `integer`. This argument is `optional` and `intent(out)`. The operation is successful if `stat` is `0`.

`msg`: Error message. This argument is `optional` and `intent(out)`.

#### Examples

```fortran
{!example/io/example_load_npz.f90!}
```
185 changes: 181 additions & 4 deletions doc/specs/stdlib_io.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,11 +136,11 @@ Loads an `array` from a npy formatted binary file.

### Syntax

`call ` [[stdlib_io_npy(module):load_npy(interface)]] `(filename, array[, iostat][, iomsg])`
`call ` [[stdlib_io_np(module):load_npy(interface)]] `(filename, array[, iostat][, iomsg])`

### Arguments

`filename`: Shall be a character expression containing the file name from which to load the `array`.
`filename`: Shall be a character expression containing the file name from which to load the `array`.
This argument is `intent(in)`.

`array`: Shall be an allocatable array of any rank of type `real`, `complex` or `integer`.
Expand All @@ -164,7 +164,6 @@ Returns an allocated `array` with the content of `filename` in case of success.
{!example/io/example_loadnpy.f90!}
```


## `save_npy`

### Status
Expand All @@ -177,7 +176,7 @@ Saves an `array` into a npy formatted binary file.

### Syntax

`call ` [[stdlib_io_npy(module):save_npy(interface)]] `(filename, array[, iostat][, iomsg])`
`call ` [[stdlib_io_np(module):save_npy(interface)]] `(filename, array[, iostat][, iomsg])`

### Arguments

Expand Down Expand Up @@ -205,6 +204,70 @@ Provides a npy file called `filename` that contains the rank-2 `array`.
{!example/io/example_savenpy.f90!}
```

## `load_npz`

### Status

Experimental

### Description

Populates an array of `array_wrapper_type` with the contents of an npz file.

### Syntax

`call ` [[stdlib_io_np(module):load_npz(interface)]] `(filename, arrays[, iostat][, iomsg][, tmp_dir])`

### Arguments

`filename`: Shall be a character expression containing the name of the npz file to load from. The argument is `intent(in)`.

`arrays`: Shall be an allocatable array of type `array_wrapper_type` to load the content of the npz file to. This argument is `intent(out)`.

`iostat`: Default integer, contains status of loading to file, zero in case of success. This argument is `optional` and `intent(out)`.

`iomsg`: Deferred length character value, contains error message in case `iostat` is non-zero. It is an optional argument, error message will be dropped if not present. This argument is `intent(out)`.

`tmp_dir`: Shall be a character expression containing the name of the temporary directory to extract the npz file to. The argument is `optional` and `intent(in)`.

### Example

```fortran
{!example/io/example_load_npz.f90!}
```

## `save_npz`

### Status

Experimental

### Description

Saves an array of `array_wrapper_type` into a npz file.

### Syntax

`call ` [[stdlib_io_np(module):save_npz(interface)]] `(filename, arrays[, iostat][, iomsg][, compressed])`

### Arguments

`filename`: Shall be a character expression containing the name of the file that contains the arrays. This argument is `intent(in)`.

`arrays`: Shall be arrays of type `array_wrapper_type` that are meant to be saved to disk. This argument is `intent(in)`.

`iostat`: Default integer, contains status of saving to file, zero in case of success. This argument is `optional` and `intent(out)`.

`iomsg`: Deferred length character value, contains error message in case `iostat` is non-zero. It is an optional argument, error message will be dropped if not present. This argument is `intent(out)`.

`compressed`: Shall be a logical expression that determines if the npz file should be compressed. The argument is `optional` and `intent(in)`. The default value is `.false.`.

### Example

```fortran
{!example/io/example_save_npz.f90!}
```

## `getline`

### Status
Expand Down Expand Up @@ -260,3 +323,117 @@ Provides formats for all kinds as defined in the `stdlib_kinds` module.
```fortran
{!example/io/example_fmt_constants.f90!}
```

## `zip`

### Status

Experimental

### Description

Compresses a file or directory into a zip archive.

### Syntax

`call ` [[stdlib_io_zip(module):zip(subroutine)]] ` (output_file, files[, stat][, msg][, compressed])`

### Arguments

`output_file`: Character expression representing the name of the zip file to create. This argument is `intent(in)`.

`files`: Array of `string_type` representing the names of the files to compress. This argument is `intent(in)`.

`stat`: Default integer, contains status of reading from unit, zero in case of success. This argument is `optional` and `intent(out)`.

`msg`: Deferred length character value, contains error message in case `iostat` is non-zero. It is an `optional` argument, error message will be dropped if not present. This argument is `intent(out)`.

`compressed`: Logical expression that determines if the zip file should be compressed. The argument is `optional` and `intent(in)`. The default value is `.true.`.

## `unzip`

### Status

Experimental

### Description

Extracts a zip archive into a directory.

### Syntax

`call ` [[stdlib_io_zip(module):unzip(subroutine)]] ` (filename, outputdir[, stat][, msg])`

### Arguments

`filename`: Character expression representing the name of the zip file to extract. This argument is `intent(in)`.

`outputdir`: Character expression representing the name of the directory to extract the zip file to. This argument is `intent(in)`.

`stat`: Default integer, contains status of reading from unit, zero in case of success. This argument is `optional` and `intent(out)`.

`msg`: Deferred length character value, contains error message in case `iostat` is non-zero. It is an `optional` argument, error message will be dropped if not present. This argument is `intent(out)`.

## `exists`

### Status

Experimental

### Description

Whether a file or directory exists at the given location in the filesystem.

### Syntax

`is_existing = ` [[stdlib_io_filesystem(module):exists(function)]] ` (filename)`

### Arguments

`filename`: Character expression representing the name of the file or directory to check for existence. This argument is `intent(in)`.

## `list_dir`

### Status

Experimental

### Description

Lists the contents of a directory.

### Syntax

`call ` [[stdlib_io_filesystem(module):list_dir(subroutine)]] ` (dir, files[, iostat][, iomsg])`

### Arguments

`dir`: Character expression representing the name of the directory to list. This argument is `intent(in)`.

`files`: Array of `string_type` representing the names of the files in the directory. This argument is `intent(out)`.

`iostat`: Default integer, contains status of reading from unit, zero in case of success. This argument is `optional` and `intent(out)`.

`iomsg`: Deferred length character value, contains error message in case `iostat` is non-zero. It is an `optional` argument, error message will be dropped if not present. This argument is `intent(out)`.

## `run`

### Status

Experimental

### Description

Runs a command in the shell.

### Syntax

`call ` [[stdlib_io_filesystem(module):run(subroutine)]] ` (command[, iostat][, iomsg])`

### Arguments

`command`: Character expression representing the command to run. This argument is `intent(in)`.

`iostat`: Default integer, contains status of reading from unit, zero in case of success. This argument is `optional` and `intent(out)`.

`iomsg`: Deferred length character value, contains error message in case `iostat` is non-zero. It is an `optional` argument, error message will be dropped if not present. This argument is `intent(out)`.
2 changes: 2 additions & 0 deletions example/io/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,5 @@ ADD_EXAMPLE(loadtxt)
ADD_EXAMPLE(open)
ADD_EXAMPLE(savenpy)
ADD_EXAMPLE(savetxt)
ADD_EXAMPLE(load_npz)
ADD_EXAMPLE(save_npz)
Binary file added example/io/example_load.npz
Binary file not shown.
18 changes: 18 additions & 0 deletions example/io/example_load_npz.f90
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
program example_load_npz
use stdlib_array, only: array_wrapper_type
use stdlib_kinds, only: int32, sp
use stdlib_io_np, only: load_npz
implicit none

type(array_wrapper_type), allocatable :: arrays(:)
real(sp), allocatable :: array_1(:,:)
integer(int32), allocatable :: array_2(:,:)

call load_npz('example_load.npz', arrays)

call arrays(1)%get_values(array_1)
call arrays(2)%get_values(array_2)

print *, array_1
print *, array_2
end
Loading
Loading