Skip to content

2023 Developer Meetings

Jump to bottom
Kate edited this page Oct 16, 2023 · 40 revisions

Next meeting: 23/10 14:00 CEST

2023-10-16

Present: Kate (@kit-ty-kate), Raja (@rjbou)

2023-09-11

Present: Kate (@kit-ty-kate), Raja (@rjbou), Riku (@rikusilvola)

  • Generation of Windows binaries

    • Generating static binaries would've required reimplementing C++ code in C or OCaml. This effort was deemed not worth it.
    • Support for MinGW achieved with flexdll (Kate to make a PR)
    • With MSVC we’ll need to expect the redistributable (installer could check)

  • Installation experience for Windows

    • Consider using opam-wix to create the installer
    • For next alpha we will simply provide the binaries without installer

PR:

  • #5659
    • good to go
    • rjbou take a look
  • #5636
    • to be discussed with dra27
    • handle escape backslash for non path variables
    • look at the opam guard
  • #5607 relies on #5657
  • #5652
    • updated: add test & change display from warn to error
  • #5634
    • needs louis review

Issues:

  • #5661
    • already known issue
  • #5660
    • should be fixed in alpha2
  • #5658
    • Proposal to have level sub pinning, or path exclusion
  • #5656
    • rely on repo lock file ? if still present, recompute cache
    • related to x & y
  • #5653
    • maybe working-dir issue

2023-09-04

Present: Kate (@kit-ty-kate), Raja (@rjbou), Riku (@rikusilvola)

No notes

2023-08-29

Present: Kate (@kit-ty-kate), Raja (@rjbou)

PR:

  • #5560 - Reftests: add tests to check url handling behaviours
    • to merge
  • #5636 - Path rewriting for setenv: and build-env:
    • good to review
    • to update with removal of rewriting for ocaml package
  • new tests for filters : #5643 & #5642

Issues:

  • #5646 - Show the full list of pins and repositories for the current switch in opam config report
    • rjbou: ok on the idea, maybe have another command (or --full) to have all setup
  • #5645 - Outputting json format
    • Under discussion, may be implemented by author
  • #5644 - [windows] opam hints to use eval after install
    • After lookup, there is some . process that make shell detection fail

2023-08-22

Present: Kate (@kit-ty-kate), Raja (@rjbou)

  • #5560 - Reftests: add tests to check url handling behaviours
    • split on several PRs
    • logging one: format %a
    • then good to review
  • #5628 - make cold on Windows: inconsistent assumptions over implementation Stdlib__Sys
    • hint fixes it, move it as part of install
    • #5635 - Fix "make cold" on Windows when gcc is available: merged
    • #5600 - make cold on Windows: flexlink error: closed
  • #5636 - Path rewriting for setenv: and build-env:
    • to update for parse/print

2023-08-14

Present: Kate (@kit-ty-kate), Raja (@rjbou)

  • Discussion about #5602, best way to do it in a non invasive way is x- field proposal, with a lint to be sure it is not forgotten
  • #5630 left for after 2.2 release, we need to see how opam depext can link these packages
  • Discussion on windows related issues #5628
  • Discussion on opam publish & github token issue opam-publish#155, can be fixed by allowing stdin on launching git process

2023-08-07

Present: @kit-ty-kate, @AltGr

2023-07-24

Present: Ben (@benmandrew), Kate (@kit-ty-kate), Raja (@rjbou)

  • Last week, alpha2 all out
  • Next one, alpha3/beta:
    • estimated at the end of august
    • blocking:
      • git on windows
      • opam tree: #5613 to review, check recurse & subpath
  • rjbou added a page on wiki Feature requests to have a view on all feature requests issues
  • Ben Andrew works on ci & docker images for opam-repo & ocaml & healthcheck
    • have a proper ci to test for windows for opam-repo
    • ftm working just on base images, after that the full ci
    • goal for windows opam repo: ocaml package + dune working, the rest is on the community

Backlog

  • #5616 - release script update
    • try to have it only on mac
      • rjbou will try to launch on mac only
      • try to have non graphic qemu launch
  • #5620: move to opam publish
  • #5605: good to integrate, propose to make a PR
  • #5606: ++ good to have for the final
  • url check PRs: tbd
  • #5534: to check
  • #5185: good to have, if we have time
  • #5619: to discuss further later
  • #4777: see with david
  • TODO: add a warning if installation cygwin interne & espace
  • Cleaning 2.2.0 project

2023-07-24

Present: Kate (@kit-ty-kate), Raja (@rjbou), Riku (@rikusilvola)

  • Final PR for alpha 2 has been reviewed by Kate and Louis. Raja has tested it and it is merged. Release process will begin this afternoon.
  • Discussions ongoing with Jonah of Diskuv to potentially integrate some of their changes into opam directly for alpha 3. Further details are on OCamlLabs (#opam-dev). The teams will have a video call on Tuesday 25/07 at 5pm (CEST).
  • New channel on OCamlLabs slack created for coordinating the efforts between the compiler team, the opam team and the OCaml CI team (#opam-windows).
  • Work on upstreaming David’s compiler patches from his fork of the opam-repository is planned for next week, which would streamline use of opam 2.2 on Windows.
  • Following the discussions on OCamlLabs Slack (#windows) the plan is now to deliver Windows binaries only for MSVC (32/64bit). Before release these need to be thoroughly tested, and as long as we get them for the RC these could be stress tested also on CI.

2023-07-17

Present: Kate (@kit-ty-kate), Raja (@rjbou), Riku (@rikusilvola)

  • Cygwin PR still pending review (expected this wednesday) - sole expected PR for alpha 2.2
  • Issues identified with alpha
    • #5601
    • #5600
    • Need more time to understand the issue - need more Windows expertise
      • Potentially due to backslash on paths
      • To be raised on OCamlLabs Slack #windows channel
  • Windows binaries
    • Need to deliver opam and opam-putenv binaries
    • 64 bit only ?
    • Raise the question on OCamlLabs Slack #windows channel
  • Opam Future
    • Raja working on categorizing feature requests
    • To be picked up next week

2023-07-10

Present: Raja (@rjbou), Riku (@rikusilvola)

  • Raja and Sabine are continuing work on opam package and opam tool documentation on ocaml.org
  • Opam 2.2 alpha announcement made on blog, changelog, discuss
    • A single issue with building opam on Windows: to be investigated
  • Raja is picking up work again on Cygwin installation PR
    • Needs some UX improvements, an option to provide the path from CLI, etc

2023-07-03

Present: Raja (@rjbou), Riku (@rikusilvola)

  • 2.2 alpha announcement

    • Platform-blog post is ready to be published
    • Discuss post to be written this afternoon
    • OCaml.org changelog to have the full platform-blog content (will be big)
    • Feature request to ocaml.org team to partially hide overly long announcements?

  • Opam web page

    • Opam2web should be “deprecated” but parts of the page will need to remain
    • Package documentation should be only on ocaml.org
    • Opam documentation needs to be reworked into a odoc generateble form
      • How would we generate man pages? How does dune do it?

  • 2.2 alpha2

    • Opam 2.2.0 Github project updated to track alpha-2
    • New 2.2.0~alpha2 milestone to track progress
    • Single mandatory PR for alpha 2 : #5545
    • Raja in touch with Hannes and Reynir regarding security improvements PR which could be included
    • w.r.t release procedure, when we’ll be on RC we can accept only bugfixes and security fixes to master

  • Opam roadmap

    • Ongoing maintenance
    • Opam library interface requires a better understanding of the needs (later in the year) - convergence!
    • Many user requests to choose from, but nothing major
    • Introducing swhid for all packages (at least the latest version?) would improve robustness, not a huge amount of work

2023-06-26

Present: Raja (@rjbou), Riku (@rikusilvola)

  • 2.2 alpha status update

    • Raja has finished finalizing the changelog
    • Final PR merged #5580
    • Release notes are ready
    • Release post in preparation - requires reorganization of the changelog
    • OCaml mingw64 compiles correctly, but ocamlbuild incompatible with Windows and needs an update: issues have been opened
    • Next steps
      1. Tag
      2. GitHub release (draft)
      3. Build & signing binaries, upload them and add release notes -> publish GitHub release
      4. Opam-repository PR creation
      5. Opam-repository PR merge + blog post
    • Release process
      • readme
      • Announce! section to be updated to include ocaml.org changelog

  • Opam dev team public PGP key updated

2023-06-26

Present: Kate (@kit-ty-kate), Marek (@Leonidas-from-XI), Raja (@rjbou), Riku (@rikusilvola), Steve (@gridbugs), Thibaut (@tmattio)

  • Status update. https://github.com/ocaml/opam/projects/2

    • Raja: Louis reviewed the last PRs for opam 2.2. We clarified the release notes. Completed the last PRs. Antonin open a PR to fix a configure error causing build failures on Windows images. Cleaning master change log to populate CHANGES and release notes.
    • Two PRs that are yet to be merged:
      • #5578.
      • #5543. Better cygwin support in core. Need to close it, it was split only for review
    • Announcement on OCaml.org
      • Let’s not include the changelog in the alpha announcement. To be included in the stable release.
    • Release cycle. 3 alpha, 1 beta, 1 rc.
      • Thibaut: why do we anticipate so many alpha?
      • Raja: alpha2 is needed as it contains opam init without cygwin preinstalled
      • Raja: Doesn’t have to be, if there’s no issue, we can do 2 alpha, 1 beta, 1 rc.
    • Release cycle:
      • alphaX is only compatible with opam-repository-mingw. Beta is compatible with upstream ocaml/opam-repository.
      • Should be working with sunset repo (mingw), but we didn’t test it, should test before we announce. Raja can test, but need to have access to a VM. Kate has a VM, can give access to Raja.

  • What is opam 3?

    • Kate: have a library that’s tailored to be used in Dune.
    • Raja: we can have the library (as today) without having an opam 3 ; for me it's more about update on opam file format, upgrading opam repo, have breaking changes in format
    • Steve: from what we’ve done, we have access to all the API we need already
    • Thibaut: but do we have the API we need for all the features? E.g. depext?
    • Steve: we’d need to integrate to know.
    • Marek: also too many dependencies is a problem
    • Thibaut: are we worried about using API that are too low-level and re-implementing things?
    • Steve: we can try to implement things in dune, then see which parts we can upstream

  • Stepping back, what are projects for the upcoming versions of opam?

    • More general version of depext system
    • Revision system
    • Missing filters
    • Custom commands to retrieve packages (support third party dpkg repository)
      • Why not just add a custom repository?

  • Next time, continue the conversation on the roadmap. In particular, let’s try to clarify what the interface between opam library and its clients should look like. Do we need more than a few new API here and there?

2023-06-12

Present: Raja (@rjbou), Riku (@rikusilvola), Thibaut (@tmattio)

Notes

2.2.0 alpha

Raja: Looking good for the release of opam 2.2 - Louis didn’t have time to review but once the PRs are reviewed we should be able to cut the alpha release

Raja: need to start working on the announcement.

Raja: There is some work done on performance, following on work that Kate had been doing #5503.

Raja: Will be joining the Tarides Build System retreat.

Thibaut: what will be missing after the alpha to get the stable version?

Riku: there’s a swimlane and we’ll need to sort the tasks to refine scope, project

Raja: no new features, only bug fixes and enhancements

Thibaut: From David’s initial PR #5220, what did we extract and what did we leave outside of the scope? For instance the depext system?

Raja: depext system was implemented and merged, see #5542

Riku: Kate has been working on improvements that won’t be part of the scope

Raja: the PR is for the shell, this is part of #5541.

Raja: David also had a branch in addition to the PRs, i’ve extracted commits from these also, finish the work to have a complete working opam on windows.

Thibaut: so we didn’t cut the scope?

Raja: we did, we’ve only focused on opam the tool, but didn’t make changes to the opam-repository.

BLOCKERS: internal handing of cygwin #5543 and installation of cygwin #5544

Thibaut: What’s needed to cut the release?

Raja: Only Louis can review the PRs, waiting on reviews and not much else.

Meeting platform

Riku: Looking into the stack for the dev meetings. We’re considering Galène.

Raja: OCamlPro already host a server

Thibaut: we could run a pilot for Galène for the opam dev meetings and if it works well, propose that as the videoconferencing software for other Platform dev meetings.

2023-06-07

Present: Raja (@rjbou), Riku (@rikusilvola)

General

  • Weekly opam meetings
    • These meetings are public but attended by few
    • Schedule and meeting link to be shared on the wiki.
    • Any change to the meeting time will be announced on the wiki.

2.2.0

  • 2.2.0~alpha milestone cleaned up; now only containing relevant PRs: four pending review and one ready for merge.
    • #5543 - Better cygwin support in core
      • WAITING FOR REVIEW
    • #5544 - init: detect local cygwin installation
      • WAITING FOR REVIEW
    • #5545 - init: install cygwin internally
      • WAITING FOR REVIEW
    • #5541 - Some windows shell updates
      • WAITING FOR REVIEW
    • #5571 - Fix URL in warning 62
      • READY TO MERGE
  • once alpha is out, the post alpha swimlane will be cleaned up as well. Some items will go in beta and others will be moved to 2.3.0

2023-05-08

Present: Kate (@kit-ty-kate), Raja (@rjbou)

Backlog

  • #5538 - Fix opam installing packages without checking their checksum when the local cache is corrupted in some cases
    • Main critical security issue
    • @kate to debug output (ci failing) and good to merge
  • #4823 - Use OCaml code to copy/move/remove directories instead of unix commands
    • good to go, to test in real condition this week
  • #5417 - Fully revertible environment updates
    • good to merge
  • #5220 - opam init for Windows shells
    • some item still to be done in this PR
    • @rjbou to open a new PR with a subset of this PR
  • Windows support - cygwin
  • Variable resolution with backslashes into slashes, for windows
    • to look at, confirm
  • #5456 - init: change shell setup defaut to true
    • to test & good to merge
  • #5522 - configure: Ensure a complementary (32bit on 64bit platforms and 64bit on 32bit platforms) C compiler is installed on Windows
    • to review after alpha
  • #5534 - Unset OPAM_SWITCH_PREFIX when using make cold
    • good to go, @kate to update message (not touching just make cold)
  • #5455 - Depext filter enhancement: resolve package variables
    • after alhpa, to look to another solution maybe
  • #5438 - Stop using mv and switch to robocopy on Windows
  • #5535 - The conflicts field's filter does not support package variables
    • @rjbou to look at if there is time before release
  • #5532 - Ensure all make targets are run serially
    • good to go, @kate to update
  • CI: cache generation broken with #5511 (configure --with-vendored-deps)
  • #5527 - Do not silently disable mccs if a C++ compiler is not present
    • good to have, @kate to update
  • #5526 - Move the .ocamlinit script out of the root directory
    • good to merge
  • #5512 - doc: ? evaluates to true if defined
    • good to merge, @kate to rebase
  • #5510 - Add the OPAMSOLVERTOLERANCE environment variable to allow users to fix solver timeouts for good
    • mccs PRs merged
    • needs an mccs release
    • good to go
  • #5503 - Fix performance regression in opam install/remove/upgrade/reinstall
    • god to have ; @rjbou to review, if there is time

2023-05-01

No meeting

2023-04-24

Present: Kate (@kit-ty-kate), Raja (@rjbou)

Issues

  • #5513 - Output of opam pin list should indicate whether a local pin is using the working state of the directory
    • feature wish
  • #5517 - opam is not installing 'with-test' dependencies unless invoked with '--with-test' flag
    • resolved with extended package variables, at install set on cli with-test to true
    • Does it work on config ? no
    • Anyway, possible to set OPAMTEST
    • redirect to plugins opam-build & opam-test
  • #5519 - Feature request: add an option for opam lock to exclude packages from the locked file
    • no objection
  • #5520 - opam lock does not exit with an error code after displaying an error
    • bug
  • #5521 - opam install --locked behavior may not be the one expected by users
    • no objection

Backlog

  • Windows ocaml package compiling, not dune
  • #5438 - Stop using mv and switch to robocopy on Windows
    • for move, check robocopy doc: /mov /L /COPYALL /E ; or move tool
  • #5514 - Update to latest msvs-detect
    • good o go
  • #5518 - Speedup OpamVersionCompare by 15%
    • on the idea, ok

2023-04-17

Present: Kate (@kit-ty-kate), Raja (@rjbou)

Issues

  • #5509 - Warn and document about url fields in source repositories
    • Need to see where to add the lint, as it is only for pinned packages

Backlog

  • #5507 - Fix linting on opam-crowbar.opam
    • Update message & good to go
  • #5510 - Add the OPAMSOLVERTOLERANCE environment variable to allow users to fix solver timeouts for good
  • #5508 - Remove preprocessing of backwards-compatibility code
    • good to go
  • #5511 - Stop the configure script from downloading and vendoring dependencies by default
    • good to go
  • #5512 - doc: ? evaluates to true if defined
    • tiny change, good on the idea
  • packages & windows: wait & see
    • 3 options: some packaging ingeneering

2023-04-11

Present: Kate (@kit-ty-kate), Raja (@rjbou)

Issues

  • #5506 - opam 2.1.3 wants to recompile a lot of packages upon removal of an unused package
  • #5505 - opam install should ignore certain environment variables
    • add sandboxed environment, whitelisted env variables, with a command to add whitelisted ones
    • wrap-*-commands until implemented
  • #5504 - resolution of the request timed out" even with OPAMSOLVERTIMEOUT=1200
    • known issue, cf #5353
    • proposal to have a z3 binary to download, need to adapt release scripts

Backlog

  • #5503 - Fix performance regression in opam install/remove/upgrade/reinstall
    • @rjbou to review
    • look at variable resolution removal, its origin
    • note for future: look at universe computation for Query, and split them into several functions if needed
  • #5501 - Run opam init for a fresh install
    • ok for opam init, but ask carefully
  • #5500 - explicitly connect /dev/tty. Allows running via sh
  • cygwin
    • split branch into several PRs : one for cygwin only, one for update, one for opam-cygwin
    • discussions about the need to have opam managing a cygwin install

2023-04-03

Present: Kate (@kit-ty-kate), Raja (@rjbou)

Issues:

  • #5495 - vendoring: Inaccurate error message when an archive fails to download

Backlog:

  • #5455 - Depext filter enhancement: resolve package variables
    • @rjbou can we have those information (depext filter that depends on installed opam package) as a solver entry
    • @kit-ty-kate to check
  • #5496 - Add package stanza on all rules that depend on opamMain.exe.exe
    • good to go
  • #5497 - Check for the presence of swhid_core in the configure script
    • good to go
  • #5499 : depexts: reword message
    • good to go
  • #5498 - Replace usage of CPPO by dune features and small helper
    • it won't complexify
    • good to go once opam file changed
  • #5494 - src_ext: Remove unused/untested Makefile targets
    • postpone to 2.3
  • opam lib & printing
    • when using opam lib, it can be needed to pass the printer to opam lib, for specific display handling
    • functorise all opam lib -> not really wanted
    • functorise only OpamConsole, and permit to instantiate it with specific printer: how to handle that ?

2023-03-27

Present: Hannes (@hannesm), Kate (@kit-ty-kate), Raja (@rjbou), Reynir (@reynir), Riku (@rikusilvola)

Security review

We reviewed what have been done from security review: a reftest added addressing for all points, some of them already addressed.

  • On tests, we need to add extra-sources check
  • We need to look at some tests on 2.1 branch
  • Some of PRs will be backported into 2.1

Issues

  • #5493 - opam tree . isn't supported
    • in 2.2, post alpha
  • #5491 - Invariant syntax in opam config report doesn't match the expected syntax by --invariant
    • in 2.2, post alpha

Backlog

  • #5492 - Fix parsing of OPAMFETCH (support quotes / proper POSIX shell syntax)
    • @rjbou: can we not use astring?
    • @kit-ty-kate: proposal to replace OpamStd.String by astring
    • we need to check performance before that
  • #5487 - No documentation on loading state & their lock leads to read-only access to the opam 2.2 state fails when opam-state 2.1.* is used
    • @rjbou to look at lock kind for is_newer_than_self
  • #5455: Depext filter enhancement: resolve package variables
    • @kit-ty-kate why not use package variables, or global variables, in order to not have resolving variables after solver outputted a solution

2023-03-20

Present: Kate (@kit-ty-kate), Raja (@rjbou)

Backlog

  • #5480 - config report: add invariant and compiler packages
    • merged
  • #5453 - depexts: add more precise depext testing family
    • merged
  • #5386 - Add package selection for opam admin add-constraint
    • merged
  • #5468 - Lock: remove url section from generated lock file
    • merged
  • #5476 - Support atomic environment updates in OpamFile
    • all discussion are more about ##5417 (witch lock, write, etc.)
    • @rjbou to check for change dir, rebase & good to go
  • #5417 - Fully revertible environment updates
    • @rjbou: writing last env files at each prompt is conceavable, it's not consuming, and they are reused
  • #5486 - No opam init hook eval when .envrc from direnv
    • Check if it is global or local config
    • to be discussed with jonahbeckford
  • #5456 - init: change shell setup defaut to true
  • #5455 - Depext filter enhancement: resolve package variables
    • to review & test
  • hash/extra-files : @rjbou' todo
  • @kit-ty-kate: want to add time test for solver timeouts
    • proposal to use timeout) library
    • addition of specific reftest that uses it, and a dedicated github action job

2023-03-14

Present: Kate (@kit-ty-kate), Raja (@rjbou), Riku (@rikusilvola)

Issues

  • Issue #5479 - opam is utterly slow on basic errors
    • already fixed on master with #5297, some discussions on what can be done more
  • Issue #5478 - Include invariant packages in opam config report and error messages
    • Pr opened for config report #5480, to open for error messages @rjbou
  • Issue #5472 - opam pin foo ./vcs-local-path : rsync error on non versioned files
    • We need optimise that copy, tokeep only opam files, postponed to after 2.2
  • Issue #5469 - Pinning local package warns of rsync permission error
    • Check rsync option to minimise error output (and link it to opam verbosity ?)
  • Issue #5425 - opam lock . creates lock files for opam files that aren't in the current directory
    • Change default to generate near opam file, add a --lock-file-location option, keep same behaviour for older cli by using that option internally

Backlog

  • PR #5036 - init scripts: revert environment before settings new one, on session loading
    • bump to 2.3,
  • PR #5456 - init: change shell setup defaut to true
    • keep in 2.2, to updated with notes, assigned to @kit-ty-kate
  • PR #4933 - repo add/set-url: add --no-action
    • bump to 2.3
  • PR #5468 - Lock: remove url section from generated lock file
    • good to go
  • PR #5480 - config report: add invariant and compiler packages
    • some typo, and good to go
  • PR #4903 - More accurate error message on not found (package) variable with opam var
    • bump to 2.3, until review suggestions are applied
  • PR #4904 - On variables that shadow options
    • bump to 2.3
  • PR #5377 & #5286 - Reftests: handle cygwin paths & add test for build-env that overwrite opam vars
    • bump to 2.3
  • PR #5385 & #5386 - Add test for opam admin & add package selection for opam admin add-constraint
    • Fix test output for #5385
    • keep #5386 on 2.2, and no more depends on #5386
  • PR #5476 - Support atomic environment updates in OpamFile
    • PR ok, we are not sure about --read-only removal in hook scripts
    • Proposal: add hook specific option to permit read-only except for environment file ; it resolves switch lock issue in the same time as there is no clash possible on environment files name (temporary files)
    • warn users on release note for hook update (option not available on cli < 2.2)
  • PR #5417 - Fully revertible environment updates
    • not blocking for 2.2~alpha, we need to have for final release
  • PR #5438 - Stop using mv and switch to robocopy on Windows
  • PR #5220 - opam init for Windows shells
    • assigned to rjbou, once depending PRs merged (#5438 & #5476)
  • PR #5457 - windows: opam root redirection when path contains spaces
    • very good to have for 2.2, but not blocking
  • PR #5453 - depexts: add more precise depext testing family
    • to update with new syntax and good to go
  • PR #5455 - Depext filter enhancement: resolve package variables
    • to review & test
  • add --avoid-checksum-algorithm option
    • have it at parsing level for all files
    • add a warning on ignored checksum algorithm found
  • extra-files & extra-sources: parsing error if .. encountered in path
  • make extra-files field mandatory: ftm opam get on its own when extra-files are found but not specified, or erroneously in opam file.

Next meeting: 2023-03-20 13:00 UTC

2023-03-06

Present: David (@dra27), Kate (@kit-ty-kate), Raja (@rjbou)

  • Issue #5448 - space saving in opam switches
    • Suggestions already in the issue for future work (2.3+)
  • Issue #5450 - macOS sandboxing issue
    • Already has a PR (which was merged!)
  • Issue #5454 - shell scripting shims when installing on Windows
    • @dra27 can't remember which bits of previous OPAM 1.3 Windows shims are merged and which aren't. The aim was to warn when shell scripts get installed (as they're not that usable). Will return to it.
  • Issue #5460 - utop issue opened in the wrong repo
    • Transfer manually over to ocaml-community/utop
  • Issue #5461 - opam env output even with no switch selected
    • Seems reasonable, and maybe even comes with the Windows changes (Cygwin bin directory needs to be added globally)
  • Issue #5463 - fewer interactive prompts when installing
    • safe-yes looks like a possibility (sitting between no and yes).
    • Can simply check the solution on opam install to ensure no downgrades as suggested
    • @rjbou: need to worry about falling back to ask for questions which haven't been upgraded.
    • @dra27: that said, falling back to "ask" means that those would be OK
  • Issue #5462
    • Long discussion on options, some live-coding, but no notes - to be updated with summary

Backlog

  • @rjbou: Test file completely up-to-date with test cases for all the security issues discussed last week
  • PR [#5452] - it's slightly strange to do, but a private variant type side-steps all the accessor questions
  • @dra27: Confused at the commits from PR #4926 still present in PR #5436, but with no conflict?
  • @rjbou: on this rare occasion, the commits are identical to the original PR, so GitHub actually offers merge without conflict (including on master_changes). Will rebase to remove them.
  • @dra27: PR #5286 - anything to discuss on the review of this?
  • @rjbou: It's not on the critical path of other fixes anymore, so not as urgent to deal with.
  • @dra27 to review PR #5457 urgently (progress blocking on decisions).
  • PR #5453 (depext testing) needs reviewing to unblock PR #5455
  • @dra27: do the changes in PR #5208 include outputting invariant packages in opam config report
  • @rjbou: not at the moment, no!
  • @dra27: especially with the Windows compiler on their way (both from opam-repository-mingw and upstream) and also with bytecode-only architectures now becoming more common, opam config report should include the list of compiler packages. The ideal would be to include the compiler packages selected by the invariant along with their dependencies, but not post dependencies (i.e. ocaml, ocaml-base-compiler, ocaml-option-*, etc. but not base-*) but displaying all of the packages captured by the invariant would be fine too.
  • @kit-ty-kate: that should also be on the error screen when a package fails
  • @dra27: this can either be added to the PR, or we can track that as a request separately

Next meeting: 2023-03-14 13:00 UTC

2023-02-27

Present: David (@dra27), Hannes (@hannesm), Kate (@kit-ty-kate), Raja (@rjbou), Reynir (@reynir)

opam security review

This meeting was entirely focused on opam security, following a review of opam by Robur as part of ongoing work to deploy Conex repository signing.

  • Security of CI systems w.r.t. to binaries being distributed. Major concern here as to whether any systems could be distributing binaries created on a system where opam's download cache should not be trusted
    • ocaml-ci / opam-repo-ci both neither distribute binaries and are in fact using opam's sandbox, so the platform's systems aren't affected
    • Action: publicise that the download cache is untrusted?
  • Checksums: linting check, and consistent handling of multiple checksum definitions for the algorithm to an issue
  • Requiring checksums: changing --require-checksums to be the default option for the default and 2.2 CLI
  • Could we have opam lint require a tarball - so a lint warning/error. Stick with a warning
  • @dra27: could suggest the conversion of a tarball with no checksum to a git branch as appropriate?
  • @rjbou: just need to be careful to differentiate release tarballs
  • Action Issue to track updating the linting rules for tarball sources with no checksums + repository update
  • Local caching: at the moment a weak hash algorithm can be used to perform a gradual denial of service on the download cache as the first hashing algorithm is used for the actual file, and the others symlink it. Open issue for storing files in the cache with the strongest checksum (or even computing it by default, e.g. for files which just specify md5, etc.)
  • Trusting the local cache: should measure the performance impact of verifying the files on each load, but should be OK (especially when considering future work integrating ocaml-tar, etc., where the checksum could be being trivially computed as the file is processed)
  • Add unified error messages for mis-matching checksums (and possibly doing the check only once - the fear is of code smell meaning that checksums aren't necessarily being recorded elsewhere). This likely won't be possible for 2.2, but the primary aim would be to restructure the code so that extraction from the download cache always goes through exactly the same code path.
  • extra-files checksums - switch the default to a hard-error with --require-checksums
  • Switch from warning to error for missing checksums on extra-files. Lint needs to have a hard error for this - then opam-repo-ci keeps the repository in that state. opam lint should have full checking the files directory and opam file are completely in sync
  • opam lint should warn if either extra-files, extra-source or url conflict on basenames.
  • Further discussion on the extra-files section - final conclusion was that it could be completely removed from opam-repository with no impact to Conex.
  • Open an issue proposing the removing of extra-files
  • Add --require-checksums to opam source for consistency with opam install
  • opam lint step to warn on just md5, then sweep opam-repository to ensure it passes
  • Would also be good to have a feature that allows specifying which hash algorithms are ignored - disabled ones would act as if they're not there at all

Most of these issues should be possible to add to the opam 2.2 slate. Suggest meeting on 27 March to review where things are with them.

Next meeting: 2023-03-06 13:00 UTC

2023-02-21

Present: David (@dra27), Kate (@kit-ty-kate), Raja (@rjbou), Riku (@rikusilvola)

  • Issue #5439 - configure issue spotted dealing with PR #5437
    • To be dealt with "at some point"!
  • Issue #5440 - corrupt root following out-of-disk condition
    • Can be mitigated as part of PR #5417 adding atomic updates to files?
    • Also discussed last week: whichever PR is completed first can include the atomic updates part of OpamFile!
  • Issue #5443 - possible issue with parallel download code on Windows?
  • Issue #5445 - no-op when local vcs directory doesn't have an opam file
  • Issue #5446 - change default behaviour for opam command
    • @dra27 There are of plans to have a workflow like this, but there woll never be the opam command
  • @dra27: wind minimum version of OCaml forward to 4.08?
    • Previously looked at July 2021, with 4.08 for the state, repository and client libraries; 4.03 for core, format and installer
    • Impact now looks near-zero, looking at revdeps in opam-repository
    • Could possibly also be done in parallel with a PR to do it?
    • @dra27: which distros are still on less than 4.08?
      • FreeBSD - 4.07
      • Debian old-stable - 4.05
      • Ubuntu 18.04 - 4.05

Backlog

  • PR #5305 - awaiting final review
  • @rjbou finishing off compiler package invariants (PR #5308
  • @rjbou - added issue for opam admin add-constraint for specific packages (Issue #5386) to 2.2 milestone. Queued on admin reftest (PR #5285) which is queued on the cygpath chain (PR #5286).
  • PR #5380 - reducing the number of repository hashes in the testsuite
    • @kit-ty-kate: why do we need this?
    • @rjbou: ideally we'd have none, using simulated repositories for the reproduction cases
    • @kit-ty-kate: but we'll always need these, won't we? reftests only take 2.5 mins from cold locally
    • @rjbou: seeing 15 mins even initialised, surely? Concerned that this will only grow if we don't prune them back
    • (@dra27: we should apparently get @rjbou a new laptop...)
    • @rjbou: but what's the issue with changing the hash for a test when the output doesn't change?
    • @kit-ty-kate: say with opam list where these things can change. So the absence of some lines can also be part of the test.
    • @rjbou (and @dra27): 💡! This particular PR has now become obsolete (one of the commits is being used in another PR; the other is the kind of list commit @kit-ty-kate is concerned about)
    • @rjbou: for the future, when we're particularly concerned about the output, let's add a comment with the repo commit

Next meeting: 2023-02-27 12:00 UTC

2023-02-06

Present: David (@dra27), Raja (@rjbou)

  • Issue #5432 opam pin failing in snapcraft
    • Quick comment back to make progress - it really doesn't appear to be opam issuing the exit code
  • PR #5436 - sys-pkg-manager-cmd field replacing global variable
    • Discussion on implementation - referred back to notes from 6 Jan
    • @dra27: that field then gives the place for Cygwin setup as well.
  • Active branches:
    • @rjbou: :installed variable for depexts: implementation in progress
    • @rjbou: redirection of opam roots when the default path has a space: implementation in progress
    • @dra27: having a look at delaying the Cygwin installation to later in opam init (ultimate aim would be to install it just as the first switch is created).
  • PR #5374 is blocking #5356
    • The latter (fixing case-preserving environment updates for Windows) needs custom comparison functions for some of the List functions which is part of #5374.

Next meeting: 2023-02-13 13:00 UTC

2023-01-30

Present: David (@dra27), Raja (@rjbou), Kate (@kit-ty-kate)

  • Issue #5426 - producing a lockfile without installing the package
    • @rjbou: definitely easy to have it that the package itself isn't installed, but the switch needs the dependencies installed
    • @dra27: is this more that it's a outputting the solver results?
    • @rjbou: this is possible, but the current lock implementation provides the assurance that the lockfile works (because it was installed in the switch)
    • @kit-ty-kate: what if that assurance has come from, say, CI, i.e. is verified externally?
    • @rjbou: this could be done, yes - it's a new locking mechanism, though
    • @dra27: I think if we do this it would be an alternate way of locking, rather than replacing the existing switch-based mechanism
    • Conclusion: it's a good feature; but it's a different locking mechanism from the current one. No issue if the dependencies are already installed in the switch.
    • Action: @dra27 write back on the issue
  • PR #4859 - Software Heritage PR; in particular the recent comment from @rdicosmo
    • Standardisation effort for SWHID
    • Something to be looking at post 2.2.0 when it's actually deployed
    • At present, the packages are uploaded (as part of @zapaschcanon's work). They are already crawling opam-repository (this was the backdrop for the previous opam-repository PRs). The opam side is then what we're adding in 2.2.0.
    • Key point (previously missed by @dra27) is that the archives are already being constantly archived, the infrastructure to add SWHIDs to opam files will then allow opam users to retrieve them (i.e. we don't actually need to do anything at the moment to keep the archive alive)
  • .ocamlinit issue from OPAM 1.x (Unbound module Topdirs error from OPAM-generated .ocamlinit)
    • While explaining that the issue should be affecting OCaml 5.0 as well, @dra27 realised the PR in trunk OCaml that's triggered this
    • Suggest bouncing this back to ocaml/ocaml in order to allow possible changes before 5.1 is branched (@kit-ty-kate opened ocaml/ocaml#11979).

Next meeting: 2023-02-06 13:00 UTC

2023-01-23

Present: David (@dra27), Raja (@rjbou), Kate (@kit-ty-kate)

General triage of issues for clearing the 2.2.0 project

  • Issue #4989 and PR #5069 - ready to merge, but needs an issue to add a warning for when synopsis is empty.
  • PR #5039 - bump to 2.3 as an improvement to OPAM_LAST_ENV.
  • Issue #5421 - not sure this needs to be addressed for 2.2, given that upstream OCaml doesn't officially support MSYS2. Bumped to 2.3.
  • PR #5381 - resolution for next steps on 14 Dec. Bumped to 2.3.
  • Issue #5339 - worth pushing djs55/ocaml-sha#58 along; possibly also worth putting in a src_ext patch so that make cold works for OpenBSD. Yes - this should be patched. Upstream PR merged during the meeting! So just a src_ext version bump needed.
  • PR #5260 - silencing errors on env_hooks
    • @kit-ty-kate: not at all sure that this is a good idea
    • @rjbou: there could be reasonable use-cases for suppressing errors, but in the absence of one, not objecting to leaving the PR unmerged.
    • @dra27: uneasy that hiding errors here would result in a more cross user at not receiving error messages, than a cross user at receiving too many!
    • Action: @dra27 to close PR
  • Issue #5163
    • @dra27: easy to implement (certainly for Windows). I (still) think we should do this for 2.2.0
    • Agreed!
  • Issue #4891 - have we done everything we intend for 2.2?
    • Indeed - looking at adding --force (or changing the default and adding --keep or some such) to fix this properly in 2.3
  • Issue #4361, Issue #4406 and PR #4414 - bump to 2.3.
  • Issue #4608 and PR #4931 - bump to 2.3
  • Issue #4373 - have we done everything we intend for 2.2?
    • Indeed
  • Issue #4272 and PR #4996 - bump to 2.3.
    • @kit-ty-kate - we should sort out the internal representation of versions as part of this as well (they're stored as strings and then converted every time they're compared)
  • Issue #5012 and PR #5014 - bump to 2.3.

General discussion

  • @dra27: will try to convert TODO items (especially for Windows) which don't yet have issues into notes on the project board
  • @kit-ty-kate: would like to ensure that opam 2.2.0 has the ability to warn the user of the need to run opam init --reinit (because users don't)
    • Suggestion here would be to write the precise (i.e. 2.2.0~alpha) version as a new field in the root config as part of the root upgrade
    • Future point releases inspect this and warn that opam init --reinit is required
  • @rjbou: will need to check the workflows involving multiple root versions, etc.
  • @dra27: should be just the client doing this; so sounds like it should be fine, and a good stop-gap until something like #4414 reliably warns when scripts/hooks are out-of-date.

Next meeting: 2023-01-30 19:00 UTC

2023-01-17

Present: David (@dra27), Raja (@rjbou), Kate (@kit-ty-kate)

  • Issue #5415 - opam list --external returning too many depexts
    • issue appears to be using --required-by when --resolve would be better.
    • Issue updated and should be closeable.
  • Issue #5411 - more graceful behaviour on failed opam update
    • Related to this is the issue @kit-ty-kate reported from Discord yesterday (corrupted opam switch state following a crash during a switch upgrade).
    • Discussion around that - writing files atomically might mitigate that a bit further (only so far, as there are times when the directory needs to be written atomically).
    • For the issue itself, it should be possible to differentiate from a network error.
  • PR #5418 - fixing opam switch export --freeze w.r.t. extra-files with missing checksums
    • issue spotted by @kit-ty-kate working on opam-repo-ci. Idea is to have a guaranteed stage which doesn't need networking, so this issue got spotted looking at switch export --freeze as a way exporting this.
    • @rjbou to review.
  • PR #5414 - bug in opam list --coinstallable
    • Some bug fixes are bigger than others!
    • The testcase is quite long (in terms of lines), but doesn't take long to execute and maps to a real-world usage of this.
  • PR #5412 - follow-up from broken CI check in #5405
    • What should we do to prevent it in future?
    • @rjbou: prefer to be running tests only when needed?
    • @dra27: although here it meant we skipped? Given that they run in parallel, it wouldn't slow CI runs down to be running this one always (obviously it uses more energy)?
    • @rjbou: don't mind, if it's not clear-cut exactly what the criteria are for running it.
    • Action: @dra27 follow-up PR for hygiene work-flow
  • PR #5417 - robustly revertible environment updates
    • Related to #5039
    • @kit-ty-kate pointed out potential security hole using ``/tmp` - written back to the PR as a task
  • PR #5407 - timings for dose
    • Good to go!
  • Issue #5409 - question for backports for 2.1.5.
    • @dra27: #5315 is a bug-fix, so clearly fine. #5342[#5342] is arguably a bug-fix too.
    • Back-port list updated.
  • @dra27 working this week on Windows depext support
  • @rjbou looking at depext configuration following on from discussion two weeks ago
  • @dra27: story needed for line ending issues in repository... two possible solution; @dra27's internal monologue on it will be resolved by the time we cut the alpha!
  • @dra27: OK to switch these meetings to Zoom?
    • No problem; they'll switch - we may start publishing the link once @dra27's checked the meeting settings!
  • @dra27: Adopting the OCaml Code of Conduct
    • No objections - @dra27 to open a PR

Next meeting: 2023-01-17 13:00 UTC

2023-01-06

Present: David (@dra27), Raja (@rjbou)

  • PR #5405 - python 3 issue on macOS GHA runners
    • Suggestion from @rjbou to use opam option instead
  • Issue #5401 - enable shell hooks by default
    • Could make the comment in the profile file very clear about how to remove the lines
    • Could also be even more explicit about the update that has been made
  • Issue #5408
    • Calling opam repo set-url twice should work - switch to git and then switch back
    • We should have a better error message - or even do a complete reset of the remote - in addition to the other fixes
  • PR #5396 - calling Gc.compact while evaluating the action graph
    • @dra27 keen on this solution; @rjbou suggests switching the reference to a unit lazy to achieve the one-shot and the API should be documented that it does a compaction.
    • @dra27 to review.
  • PR #5348 - MSYS2 PR
    • It's just waiting on the variable being renamed apropos meeting from 19 Dec, then it's good to go
    • @rjbou to update and merge
  • PR #5352 - bug fix for expansions in build-env
    • @dra27: can this be back-ported to 2.1 and 2.0?
    • OK for those two - open tracking issues.
  • PR #5350 - crash when updating the environment with empty string values
    • @dra27: similarly (fixes #4840 since 2.1.1 and [https://github.com/ocaml/opam/pull/4326] since 2.0.8)
  • @rjbou's 2.2 backlog (or, rather, @dra27's reviewing backlog!):
    • #5305 - bug fix for opam root upgrading
    • #4926 - actual root upgrade for 2.2.0~alpha
    • #4859 - software heritage can then be merged (needs the root upgrade)
    • #5208 - independent of these; simply needs review
  • Further discussion on the depexts:
    • @rjbou: why are the sys-pkg-manager-cmd-* settings going in as variables rather than config fields?
    • @dra27: partly so that it could be used as a variable in an opam file (🤷‍♀️) and possibly partly because we don't know all the package managers (i.e. we're adding sys-pkg-manager-cmd-* as a variable pattern, rather than a field)?
    • @rjbou: why don't we do this as a setting with a list? Could instead be opam option 'sys-pkg-manager-cmd+=[msys2]["/t/l/m"]'
    • Agreed this third iteration looks better (less pollution of the global variable list, and still configurable for Diskuv). Merge MSYS2 PR as-is for now, implement this as part of sorting out the Cygwin depext mode (and ping @jonahbeckford when that happens). At least this is churning before the release!

Next meeting: 2023-01-17 13:00 UTC

Clone this wiki locally