release Chapel track

[lucaferranti]

fixes #1

[lucaferranti]

@ErikSchierboom this is finally happening (only one year delay 😅 ).

The PR is in draft because I would first like to double-check with you everything is good and in place. A few notes

• I looked into CodeMirror thing, there's no lezer parser for Chapel, but writing one does not seem too difficult (I actually have working experience with compilers and parsers, which is pretty handy here ^.^). I'll give myself one week to look into this, otherwise let just leave this as post-release improvement.

• Could you double-check everything is good? Anything missing or that needs to be done before release?

• I will actually be giving a talk at ChapelCon (Chapel conference, held once a year online, free) about the Chapel track on exercism, its development and roadmap.

[ErikSchierboom]

@lucaferranti Things look good so far. My only concern is that the test runner is quite slow. Running the tests took about 17 seconds for Hello World, which is just shy of our 20 second cutoff. With infrastructure perf varying, this likely means that student will see timeouts when using the online editor. Is there anything to be done to speedup things in the test runner? Maybe like pre-compiling things or something (which some tracks' test runners benefit a lot from)?

I'm not sure if I had already suggested it, but if not, I would suggest to open a topic on the forum to ask if people want to help test the track.

[lucaferranti]

@ErikSchierboom thanks! I can try to take a look at the test runner to see if there's some optimization capability, but in general Chapel compilation overhead can be quite noticeable and I think most of the overhead comes from compiling the solution file. Given the exercises relatively limited complexity, I would not expect a huge difference in compilation between e.g. hello-world and forth.

I'll do some benchmarks and give it a thought if I can find some optimization opportunities, but I would not bet on this, any chance we can bump the cutoff from 20 sec to e.g. 30?

Thanks for the tip about the forum! I was also thinking to recruit some beta-testers (and maybe mentors volunteers) from the Chapel team itself

[ErikSchierboom]

I'll do some benchmarks and give it a thought if I can find some optimization opportunities, but I would not bet on this, any chance we can bump the cutoff from 20 sec to e.g. 30?

Not really, this is an Exercism-wide cutoff.

I was also thinking to recruit some beta-testers (and maybe mentors volunteers) from the Chapel team itself
Great!

[lucaferranti]

yeah e.g. I tried submitting forth on the app and it times out... :/

[lucaferranti]

ok good news, it might be I was wrong and we can do something

❯ time mason test
Skipping registry update since no dependency found in manifest file.

----------------------------------------------------------------------
Ran 49 tests in 5.83217 seconds

OK (passed = 49 )

________________________________________________________
Executed in    6.02 secs    fish           external
   usr time    5.60 secs    0.00 millis    5.60 secs
   sys time    0.39 secs    1.19 millis    0.39 secs

on my laptop (64GB RAM and gen 12 intel i7, which is prob more powerful than exercism runner, granted) mason test runs in 5 seconds for the forth test. We have two options

1. Chapel is not actually the bottleneck and we can do something
2. exercism runner is actually 4x slower than my machine

If it's the second one, then it's a bit more complicated, but if it's the first one we can prob do something. On the other hand, the bash postprocessing taking 15 seconds sounds a bit strange 🤔

@ErikSchierboom any details (or pointer to the docs) to see what exactly has to be run within the 20 sec limits?

[ErikSchierboom]

Another option: the docker image is slow (sometimes the Docker image for languages is sub-optimal).
The best way to start testing this is to run the test runner: https://github.com/exercism/chapel-test-runner/
Then you can run individual files: https://github.com/exercism/chapel-test-runner/?tab=readme-ov-file#run-the-test-runner-on-an-exercise-using-docker

The Exercism machine will definitely be less beefy, and also has to deal with it processing many things. But running the above Docker image should help diagnose things (you can even add in logging).

[lucaferranti]

@ErikSchierboom finding now some more time to look closer.

When running the docker script I get an authentication error

❯ ./bin/run-in-docker.sh hello-world ../chapel/exercises/practice/hello-world/.meta/ tmpOutput
ERROR: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied

do you think you could help me a bit with this?

[lucaferranti]

(btw, I'll give tomorrow a talk about this at ChapelCon)

[ErikSchierboom]

@lucaferranti Check https://stackoverflow.com/a/48957722/2071395

[ErikSchierboom]

Good luck with the talk!

[lucaferranti]

lol I was one sudo away from making hte script work xD

[lucaferranti]

@ErikSchierboom I don't quite understand how I'm supposed to use the run-in-docker.sh

I am in chapel-test-runner

I have copy-pasted the forth/ folder into the chapel-test-runner and replaces src/forth.chpl with .meta/reference.chpl (the model solution) so that I can run mason test while being in forth (and it works).

However, if I run

sudo ./bin/run-in-docker.sh forth ./forth/ ./forth

then the generated results.json only has

{
  "version": 1,
  "status": "fail",
  "message": "Mason could not find your configuration file (Mason.toml)"
}

which is not quite what I would expect.

when running exercism/chapel-test-runner in the docker image? where is the test run?

[lucaferranti]

mmmm, I tried to add an

pwd
ls

before

test_output=$(mason test  2>&1)

and it seems correct

/solution
Mason.lock
Mason.toml
results.json
src
target
test

at least it is in a folder with the Mason.toml, so mason test should work

[ErikSchierboom]

I have problems getting the tests of the test runner to run.

When I run: ./bin/run-tests-in-docker.sh, I get:

sed: preserving permissions for '/opt/test-runner/tests/syntax-error/sedCy6exO': Permission denied

Any idea?

[lucaferranti]

Mmm ./bin/run-tests-in-docker.sh works for me locally, the error message does not strike to me as Chapel related

[lucaferranti]

maybe you had some file open in some editor and that was preserving sed from modifying it?

[ErikSchierboom]

I've just updated the tests of the test runner: exercism/chapel-test-runner#29

What was your plan right now? Do you need me for anything?

[mppf]

I'm surprised to see forth.chpl and its test runner taking close to 20 seconds to build. Like Luca, I see it taking closer to 5 seconds (with the just-released Chapel 2.2).

While we are working on improving the compiler's performance, I don't think we should wait for that to launch the Exercism track. So, looking for a workaround.

I built the Chapel test runner from https://github.com/exercism/chapel-test-runner/blob/main/Dockerfile (using apptainer) and it seems to be around 500 MB. It looks like it is using the Chapel Docker image which is based on Debian 11.

I found https://exercism.org/docs/building/tooling/best-practices#h-performance which recommends trying a few different base OSes and that Alpine frequently results in a small image.

It's unclear to me, will reducing the Docker file size help with the 20 seconds limit? Or is the 20 seconds counting something after the Docker image is already loaded up?

That document also suggests trying network internal rather than the default of none.

Lastly, I noticed this section https://exercism.org/docs/building/tooling/docker#h-configuration :

Some tools require (slight) deviations from the default configuration. If so, these are configured in the tools.json file in the Tooling Invoker repository.

Clicking through to the linked tools.json, I can see that Kotlin, Haskell, and F# test runners use 30 second timeouts instead of the usual 20. Would it be possible to get Chapel to also have a 30 second timeout?

[DanilaFe]

Another data point is that forth compilation takes about 9 seconds in the GitHub Codespace for Chapel. This is getting closer towards the 20 second figure, but is still below half of the upper bound. In my experience Codespaces are quite resource-constrained; although the 9 second time points to the importance of the underlying hardware, it also suggests that if Exercism is using comparable hardware, then 9 seconds should be possible. On the other hand, if Exercism's hardware is weaker, then the delays become relatively plausible.

[formlessdao]

Any ETA when people can start learning Chapel on Exercism? I'm a complete beginner to programming. Thanks

[formlessdao]

@ErikSchierboom Hi Erik, can you finalize this please?

[ErikSchierboom]

I think someone needs to look at the test runner to try and speed it up

[ErikSchierboom]

I think someone needs to look at the test runner to try and speed it up

[lucaferranti]

Sorry for the delay with this. Let me try to take a look at this.

@ErikSchierboom do you have comments on the @mppf 's questions?

Clicking through to the linked tools.json, I can see that Kotlin, Haskell, and F# test runners use 30 second timeouts instead of the usual 20. Would it be possible to get Chapel to also have a 30 second timeout?

It's unclear to me, will reducing the Docker file size help with the 20 seconds limit? Or is the 20 seconds counting something after the Docker image is already loaded up?

[ErikSchierboom]

@lucaferranti FYI, I'm no longer a member of the Exercism staff, so I won't reply as quickly as I did before.

Clicking through to the linked tools.json, I can see that Kotlin, Haskell, and F# test runners use 30 second timeouts instead of the usual 20. Would it be possible to get Chapel to also have a 30 second timeout?

Something for @iHiD to decide

It's unclear to me, will reducing the Docker file size help with the 20 seconds limit? Or is the 20 seconds counting something after the Docker image is already loaded up?

Reducing will help with a reduced startup time, and in general smaller Docker images execute more quickly.