Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MATLAB] Update MATLAB CI workflows to use MATLAB R2023b #37809

Open
kevingurney opened this issue Sep 20, 2023 · 3 comments
Open

[MATLAB] Update MATLAB CI workflows to use MATLAB R2023b #37809

kevingurney opened this issue Sep 20, 2023 · 3 comments
Labels
Component: Continuous Integration Component: MATLAB Type: enhancement

Comments

@kevingurney
Copy link
Member

Describe the enhancement requested

The MATLAB CI workflows fail on Windows when using R2023b.

This is less than ideal since the latest publicly available MATLAB version is R2023b.

We should investigate why this failure is occurring on Windows so that we can build with R2023b in CI.

Component(s)

Continuous Integration, MATLAB
@github-project-automation github-project-automation bot moved this to Backlog in Arrow MATLAB Sep 20, 2023
This was referenced Sep 20, 2023
Merged
Closed
kevingurney added a commit that referenced this issue Sep 20, 2023
@kevingurney
GH-37805: [CI][MATLAB] Hard-code release to R2023a for `matlab-ac…
e068b7f 
…tions/setup-matlab` action in MATLAB CI workflows (#37808)

### Rationale for this change

Due to a recent change to default to `R2023b` by default for the `[matlab-actions/setup-matlab`](https://github.com/matlab-actions/setup-matlab) action in GitHub Actions, the MATLAB CI workflows are failing.

Example failure logs:

https://github.com/apache/arrow/actions/runs/6250586979/job/16970596069?pr=37773#step:9:70

Our preferred solution to address this in the short term is to explicitly specify the [`release` parameter](https://github.com/matlab-actions/setup-matlab#set-up-matlab) to the `setup-matlab` action to be `R2023a`.

In the long term, we can work on figuring out why the build is failing on Windows with `R2023b`.

For reference - it appears that the `FindMatlab` CMake module only recently added R2023b to its list of recognized versions:

https://gitlab.kitware.com/cmake/cmake/-/merge_requests/8804

### What changes are included in this PR?

1. Hard-coded MATLAB `release` to `R2023a` for the [`matlab-actions/setup-matlab`](https://github.com/matlab-actions/setup-matlab) action in the MATLAB CI workflows.

### Are these changes tested?

Yes.

1. [MATLAB CI workflows are passing on all platforms in `mathworks/arrow`](https://github.com/mathworks/arrow/actions/runs/6251345588).

### Are there any user-facing changes?

No.

This change only impacts the MATLAB CI workflows which were defaulting to `R2023a` by default in the past.

### Future Directions

1. #37809
* Closes: #37805

Authored-by: Kevin Gurney <[email protected]>
Signed-off-by: Kevin Gurney <[email protected]>
JerAguilon pushed a commit to JerAguilon/arrow that referenced this issue Oct 23, 2023
@kevingurney
apacheGH-37805: [CI][MATLAB] Hard-code release to R2023a for `mat…
8d07717 
…lab-actions/setup-matlab` action in MATLAB CI workflows (apache#37808)

### Rationale for this change

Due to a recent change to default to `R2023b` by default for the `[matlab-actions/setup-matlab`](https://github.com/matlab-actions/setup-matlab) action in GitHub Actions, the MATLAB CI workflows are failing.

Example failure logs:

https://github.com/apache/arrow/actions/runs/6250586979/job/16970596069?pr=37773#step:9:70

Our preferred solution to address this in the short term is to explicitly specify the [`release` parameter](https://github.com/matlab-actions/setup-matlab#set-up-matlab) to the `setup-matlab` action to be `R2023a`.

In the long term, we can work on figuring out why the build is failing on Windows with `R2023b`.

For reference - it appears that the `FindMatlab` CMake module only recently added R2023b to its list of recognized versions:

https://gitlab.kitware.com/cmake/cmake/-/merge_requests/8804

### What changes are included in this PR?

1. Hard-coded MATLAB `release` to `R2023a` for the [`matlab-actions/setup-matlab`](https://github.com/matlab-actions/setup-matlab) action in the MATLAB CI workflows.

### Are these changes tested?

Yes.

1. [MATLAB CI workflows are passing on all platforms in `mathworks/arrow`](https://github.com/mathworks/arrow/actions/runs/6251345588).

### Are there any user-facing changes?

No.

This change only impacts the MATLAB CI workflows which were defaulting to `R2023a` by default in the past.

### Future Directions

1. apache#37809
* Closes: apache#37805

Authored-by: Kevin Gurney <[email protected]>
Signed-off-by: Kevin Gurney <[email protected]>
loicalleyne pushed a commit to loicalleyne/arrow that referenced this issue Nov 13, 2023
@kevingurney @loicalleyne
apacheGH-37805: [CI][MATLAB] Hard-code release to R2023a for `mat…
1c04196 
…lab-actions/setup-matlab` action in MATLAB CI workflows (apache#37808)

### Rationale for this change

Due to a recent change to default to `R2023b` by default for the `[matlab-actions/setup-matlab`](https://github.com/matlab-actions/setup-matlab) action in GitHub Actions, the MATLAB CI workflows are failing.

Example failure logs:

https://github.com/apache/arrow/actions/runs/6250586979/job/16970596069?pr=37773#step:9:70

Our preferred solution to address this in the short term is to explicitly specify the [`release` parameter](https://github.com/matlab-actions/setup-matlab#set-up-matlab) to the `setup-matlab` action to be `R2023a`.

In the long term, we can work on figuring out why the build is failing on Windows with `R2023b`.

For reference - it appears that the `FindMatlab` CMake module only recently added R2023b to its list of recognized versions:

https://gitlab.kitware.com/cmake/cmake/-/merge_requests/8804

### What changes are included in this PR?

1. Hard-coded MATLAB `release` to `R2023a` for the [`matlab-actions/setup-matlab`](https://github.com/matlab-actions/setup-matlab) action in the MATLAB CI workflows.

### Are these changes tested?

Yes.

1. [MATLAB CI workflows are passing on all platforms in `mathworks/arrow`](https://github.com/mathworks/arrow/actions/runs/6251345588).

### Are there any user-facing changes?

No.

This change only impacts the MATLAB CI workflows which were defaulting to `R2023a` by default in the past.

### Future Directions

1. apache#37809
* Closes: apache#37805

Authored-by: Kevin Gurney <[email protected]>
Signed-off-by: Kevin Gurney <[email protected]>
@kevingurney
Copy link
Member Author

@sgilmore10 and I have been investigating this issue more.

We managed to get a successful Windows build using MATLAB R2023b:

https://github.com/mathworks/arrow/actions/runs/6949608081/job/18908073567#step:9:526

However, we had to remove Ninja and the call to vcvarsall.bat from the Windows CI workflow. We all called cmake directly from the GitHub Actions workflow file, rather than calling matlab_build.sh using bash -c.

We noticed that the MATLAB version with Ninja and vcvarsall.bat is reported as "unknown":

https://github.com/apache/arrow/actions/runs/6250586979/job/16972019524?pr=37773#step:9:34

However, without Ninja and vcvarsall.bat, the MATLAB version is reported as "23.2":

https://github.com/mathworks/arrow/actions/runs/6949608081/job/18908073567#step:9:550

@kou
Copy link
Member

kou commented Nov 22, 2023

Interesting. One of downsides by removing Ninja is that we can't use ccache. It will increase build time.

Anyway, could you try with -DMATLAB_FIND_DEBUG=ON again to show more debug messages?

@kevingurney
Copy link
Member Author

Thanks for the suggestion of using -DMATLAB_FIND_DEBUG=ON!

We actually figured out a way to continue using Ninja (and the rest of the existing CI workflow code).

By setting Matlab_ROOT_DIR and MATLAB_ADDITIONAL_VERSIONS explicitly, we were able to get the expected MATLAB version to be detected.

Example CMake code: 0114138#diff-282bdaf7afd3f3d7974e3ab41857a65e3eea6a566107ea7d209ba3fec72e2e77R22
Successful CI Run with Ninja: https://github.com/mathworks/arrow/actions/runs/6950057226/job/18909470493
Correct MATLAB version being detected: https://github.com/mathworks/arrow/actions/runs/6950057226/job/18909470493#step:9:34

This seems like a fairly good solution that should also hopefully work when future versions of MATLAB are released (i.e. we won't need to constantly update the minimum required CMake version in order to get access to the latest FindMatlab code).

Setting the Matlab_ROOT_DIR explicitly may also be more reliable in general because FindMatlab uses a variety of heuristics to locate MATLAB and associated libraries (which generally, seem to work quite well). However, we have seen a few sporadic issues in CI that might be related to the heuristics not working 100% of the time (e.g. mathworks/libmexclass#58).

dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@kevingurney @dgreiss
apacheGH-37805: [CI][MATLAB] Hard-code release to R2023a for `mat…
8f8260c 
…lab-actions/setup-matlab` action in MATLAB CI workflows (apache#37808)

### Rationale for this change

Due to a recent change to default to `R2023b` by default for the `[matlab-actions/setup-matlab`](https://github.com/matlab-actions/setup-matlab) action in GitHub Actions, the MATLAB CI workflows are failing.

Example failure logs:

https://github.com/apache/arrow/actions/runs/6250586979/job/16970596069?pr=37773#step:9:70

Our preferred solution to address this in the short term is to explicitly specify the [`release` parameter](https://github.com/matlab-actions/setup-matlab#set-up-matlab) to the `setup-matlab` action to be `R2023a`.

In the long term, we can work on figuring out why the build is failing on Windows with `R2023b`.

For reference - it appears that the `FindMatlab` CMake module only recently added R2023b to its list of recognized versions:

https://gitlab.kitware.com/cmake/cmake/-/merge_requests/8804

### What changes are included in this PR?

1. Hard-coded MATLAB `release` to `R2023a` for the [`matlab-actions/setup-matlab`](https://github.com/matlab-actions/setup-matlab) action in the MATLAB CI workflows.

### Are these changes tested?

Yes.

1. [MATLAB CI workflows are passing on all platforms in `mathworks/arrow`](https://github.com/mathworks/arrow/actions/runs/6251345588).

### Are there any user-facing changes?

No.

This change only impacts the MATLAB CI workflows which were defaulting to `R2023a` by default in the past.

### Future Directions

1. apache#37809
* Closes: apache#37805

Authored-by: Kevin Gurney <[email protected]>
Signed-off-by: Kevin Gurney <[email protected]>
@kevingurney kevingurney mentioned this issue Apr 25, 2024
Closed
@sgilmore10 sgilmore10 mentioned this issue May 2, 2024
Merged
sgilmore10 added a commit that referenced this issue May 3, 2024
@sgilmore10 @kou
GH-41385: [CI][MATLAB][Packaging] Add support for MATLAB R2024a in …
c8cf61c 
…CI and crossbow packaging workflows (#41504)

### Rationale for this change

MATLAB `R2024a` is the latest available version of MATLAB as of April 2024.

We are currently building against MATLAB `R2023a` in CI and for the crossbow packaging workflow.

We should update the version of MATLAB we support to the latest available version.

We previously created an issue to use `R2023b` (#37809). However, `R2024a` has become publicly available since then.

### What changes are included in this PR?

1. Changed the `release` argument supplied to `matlab-actions/setup-matlab@ v2` to `R2024a` from `R2023a` in `.github/workflows/matlab.yml` and `dev/tasks/matlab/github.yml`.

2. Updated the script used to package the MLTBX file (`arrow/matlab/tools/packageMatlabInterface.m`) to specify the version of MATLAB currently running as the only compatible version of MATLAB for the interface. 

3.  Updated display tests to react to a change in how class names are formatted when an objected is displayed in the Command Window.

### Are these changes tested?

Yes. Existing tests used.

### Are there any user-facing changes?

There are no changes to the MATLAB Arrow Interface. However, the MATLAB release used to build, test, and package the interface has been upgraded.

### Future Work

1. We may want to consider building and packaging against multiple versions of MATLAB in parallel, rather than just the latest. This would require some more thought on how many releases back to support over time, however.

2. #41435

* GitHub Issue: #41385

Lead-authored-by: Sarah Gilmore <[email protected]>
Co-authored-by: Sarah Gilmore <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sarah Gilmore <[email protected]>
vibhatha pushed a commit to vibhatha/arrow that referenced this issue May 25, 2024
@sgilmore10 @kou @vibhatha
apacheGH-41385: [CI][MATLAB][Packaging] Add support for MATLAB `R2024…
ee21cf4 
…a` in CI and crossbow packaging workflows (apache#41504)

### Rationale for this change

MATLAB `R2024a` is the latest available version of MATLAB as of April 2024.

We are currently building against MATLAB `R2023a` in CI and for the crossbow packaging workflow.

We should update the version of MATLAB we support to the latest available version.

We previously created an issue to use `R2023b` (apache#37809). However, `R2024a` has become publicly available since then.

### What changes are included in this PR?

1. Changed the `release` argument supplied to `matlab-actions/setup-matlab@ v2` to `R2024a` from `R2023a` in `.github/workflows/matlab.yml` and `dev/tasks/matlab/github.yml`.

2. Updated the script used to package the MLTBX file (`arrow/matlab/tools/packageMatlabInterface.m`) to specify the version of MATLAB currently running as the only compatible version of MATLAB for the interface. 

3.  Updated display tests to react to a change in how class names are formatted when an objected is displayed in the Command Window.

### Are these changes tested?

Yes. Existing tests used.

### Are there any user-facing changes?

There are no changes to the MATLAB Arrow Interface. However, the MATLAB release used to build, test, and package the interface has been upgraded.

### Future Work

1. We may want to consider building and packaging against multiple versions of MATLAB in parallel, rather than just the latest. This would require some more thought on how many releases back to support over time, however.

2. apache#41435

* GitHub Issue: apache#41385

Lead-authored-by: Sarah Gilmore <[email protected]>
Co-authored-by: Sarah Gilmore <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sarah Gilmore <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Continuous Integration Component: MATLAB Type: enhancement
Projects
Arrow MATLAB
Status: Backlog
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kou @kevingurney