OCaml Changelog

We are pleased to announce a first release candidate for the long-awaited opam 2.0.0.

A lot of polishing has been done since the last beta, including tweaks to the built-in solver, allowing in-source package definitions to be gathered in an opam/ directory, and much more.

With all of the 2.0.0 features getting pretty solid, we are now focusing on bringing all the guides up-to-date¹, updating the tools and infrastructure, making sure there are no usability issues with the new workflows, and being future-proof so that further updates break as little as possible.

You are invited to read the beta5 announcement for details on the 2.0.0 features. Installation instructions haven't changed:

1. From binaries: run

   sh <(curl -sL https://opam.ocaml.org/install.sh)
   

   or download manually from the Github "Releases" page to your PATH.

2. From source, using opam:

   opam update; opam install opam-devel
   

   (then copy the opam binary to your PATH as explained)

3. From source, manually: see the instructions in the README.

Thanks a lot for testing out the RC and reporting any issues you may find. See what we need tested for more detail.

¹ You can at the moment rely on the manpages, the Manual, and of course the API, but other pages might be outdated.

After a few more months brewing, we are pleased to announce a new beta release of opam. With this new milestone, opam is reaching feature-freeze, with an expected 2.0.0 by the beginning of next year.

This version brings many new features, stability fixes, and big improvements to the local development workflows.

What's new

The features presented in past announcements: local switches, in-source package definition handling, extended dependencies are of course all present. But now, all the glue to make them interact nicely together is here to provide new smooth workflows. For example, the following command, if run from the source tree of a given project, creates a local switch where it will restore a precise installation, including explicit versions of all packages and pinnings:

opam switch create ./ --locked

this leverages the presence of opam.locked or <name>.opam.locked files, which are valid package definitions that contain additional details of the build environment, and can be generated with the opam-lock plugin (the lock command may be merged into opam once finalised).

But this new beta also provides a large amount of quality of life improvements, and other features. A big one, for example, is the integration of a built-in solver (derived from mccs and glpk). This means that the opam binary works out-of-the box, without requiring the external aspcud solver, and on all platforms. It is also faster.

Another big change is that detection of architecture and OS details is now done in opam, and can be used to select the external dependencies with the new format of the depexts: field, but also to affect dependencies or build flags.

There is much more to it. Please see the changelog, and the updated manual.

How to try it out

Our warm thanks for trying the new beta and reporting any issues you may hit.

1. The easiest is to use our pre-compiled binaries. This script will also make backups if you migrate from 1.x, and has an option to revert back:

   sh <(curl -sL https://opam.ocaml.org/install.sh)
   

   This uses the binaries from https://github.com/ocaml/opam/releases/tag/2.0.0-beta5

2. Another option is to compile from source, using an existing opam installation. Simply run:

   opam update; opam install opam-devel
   

   and follow the instructions (you will need to copy the compiled binary to your PATH).

3. Compiling by hand from the inclusive source archive, or from the git repo. Use ./configure && make lib-ext && make if you have OCaml >= 4.02.3 already available; make cold otherwise.

   If the build fails after updating a git repo from a previous version, try git clean -fdx src/ to remove any stale artefacts.

Note that the repository format is different from that of opam 1.2. Opam 2 will be automatically redirected from the opam-repository to an automatically rewritten 2.0 mirror, and is otherwise able to do the conversion on the fly (both for package definitions when pinning, and for whole repositories). You may not yet contribute packages in 2.0 format to opam-repository, though.

What we need tested

We are interested in all opinions and reports, but here are a few areas where your feedback would be specially useful to us:

• Use 2.0 day-to-day, in particular check any packages you may be maintaining. We would like to ensure there are no regressions due to the rewrite from 1.2 to 2.0.
• Check the quality of the solutions provided by the solver (or conflicts, when applicable).
• Test the different pinning mechanisms (rsync, git, hg, darcs) with your project version control systems. See the --working-dir option.
• Experiment with local switches for your project (and/or opam install DIR). Give us feedback on the workflow. Use opam lock and share development environments.
• If you have any custom repositories, please try the conversion to 2.0 format with opam admin upgrade --mirror on them, and use the generated mirror.
• Start porting your CI systems for larger projects to use opam 2, and give us feedback on any improvements you need for automated scripting (e.g. the --json output).

Add support for 4.06. Use Leo White's short-path for 4.05. Various bug fixes (in locate, in emacs serialization).

The major change Windows support is contributed by David Allsopp.

Other changes are a bunch of fixes:

• compilation on FreeBSD contributed by Malcolm Matalka
• improvement to emacs mode contributed by Olivier Andrieu, Christophe Troestler and Steve Purcell
• improvement to vim mode by Fabian Hemmer and Gregory Nisbet
• fixes to ppx invocation by Keigo Imai
• fixes to Merlin s-expr dialect to bring UTF-8 compatibility with Emacs (WIP)

Bug fix after 3.0.1:

• CMT magic number for 4.05 was wrong
• handle merlin.focus, merlin.ignore, merlin.loc/merlin.relaxed-loc and merlin.syntax-error
• missing include preventing build on some platforms contributed by Bernhard Schommer

Bug fix release after 3.0.0 major release.

• portability fixes by David Allsop in configure script and vim mode (tough Windows support is not ready yet)
• preliminary support for findlib toolchains with FINDLIB_TOOLCHAIN .merlin directive
• make ocamlmerlin.c frontend more portable
• various fixes to the frontend

The major change is a new protocol that moves process management inside Merlin codebase, saving a lot of pain in Emacs and Vim. There are not much new user facing features.

Windows support is not yet available.

In editor configuration is now done with merlin-flags, merlin-extensions and merlin-use in Emacs and :MerlinFlags, :MerlinExtensions and :MerlinPackages in Vim. In previous versions, enabled extensions, flags and packages were retained while now only the last command is remembered.

"M-x merlin-use a", "M-x merlin-use b" should be replaced by "M-x merlin-use a,b". ":MerlinUse a", ":MerlinUse b" should be replaced by ":MerlinUse a b".

The old protocol is still supported, so existing editor modes should not be affected (tested with Atom, Visual Studio and Sublime-text).

Other main changes:

• Support for OCaml 4.05 was added
• Merlin uses a new implementation of short-path by Leo White which addresses performance problems
• Merlin now works with the upstream version of Menhir
• numerous cleanup and refactoring to decrease the amount of changes to upstream typechecker
• emacs-imenu feature was contributed by tddsg. It is similar the "outline" feature in vim for navigating in a buffer.

Thanks to the many contributors (Jochen Bartl, tddsg, Ximin Luo, Jason Staten, Leo White, Leandro Ostera, Jacob Bass, Xavier Guérin, Yotam Barnoy, Jacques Pascal Deplaix, David Allsopp, ...).

Minor release.

Minor release.

Initial release.

UPDATE (2017-02-14): A beta2 is online, which fixes issues and performance of the opam build command. Get the new binaries, or recompile the opam-devel package and replace the previous binary.

We are pleased to announce that the beta release of opam 2.0 is now live! You can try it already, bootstrapping from a working 1.2 opam installation, with:

opam update; opam install opam-devel

With about a thousand patches since the last stable release, we took the time to gather feedback after our last announcement and implemented a couple of additional, most-wanted features:

• An opam build command that, from the root of a source tree containing one or more package definitions, can automatically handle initialisation and building of the sources in a local switch.
• Support for repository signing through the external Conex tool, being developed in parallel.

There are many more features, like the new opam clean and opam admin commands, a new archive caching system, etc., but we'll let you check the full changelog.

We also improved still on the already announced features, including compilers as packages, local switches, per-switch repository configuration, package file tracking, etc.

The updated documentation is at https://opam.ocaml.org/doc/2.0/. If you are developing in opam-related tools, you may also want to browse the new APIs.

Try it out

Please try out the beta, and report any issues or missing features. You can:

• Build it from source in opam, as shown above (opam install opam-devel)
• Use the pre-built binaries.
• Building from the source tarball: download here and build using ./configure && make lib-ext && make if you have OCaml >= 4.01 already available; make cold otherwise
• Or directly from the git tree, following the instructions included in the README. Some files have been moved around, so if your build fails after you updated an existing git clone, try to clean it up (git clean -dx).

Some users have been using the alpha for the past months without problems, but you may want to keep your opam 1.2 installation intact until the release is out. An easy way to do this is with an alias:

alias opam2="OPAMROOT=~/.opam2 path/to/opam-2-binary"

Changes to be aware of

Command-line interface

• opam switch create is now needed to create new switches, and opam switch is now much more expressive
• opam list is also much more expressive, but be aware that the output may have changed if you used it in scripts
• new commands:
  • opam build: setup and build a local source tree
  • opam clean: various cleanup operations (wiping caches, etc.)
  • opam admin: manage software repositories, including upgrading them to opam 2.0 format (replaces the opam-admin tool)
  • opam env, opam exec, opam var: shortcuts for the opam config subcommands
• opam repository add will now setup the new repository for the current switch only, unless you specify --all
• Some flags, like --test, now apply to the packages listed on the command-line only. For example, opam install lwt --test will build and install lwt and all its dependencies, but only build/run the tests of the lwt package. Test-dependencies of its dependencies are also ignored
• The new opam install --soft-request is useful for batch runs, it will maximise the installed packages among the requested ones, but won't fail if all can't be installed

As before, opam is self-documenting, so be sure to check opam COMMAND --help first when in doubt. The bash completion scripts have also been thoroughly improved, and may help navigating the new options.

Metadata

There are both a few changes (extensions, mostly) to the package description format, and more drastic changes to the repository format, mainly related to translating the old compiler definitions into packages.

• opam will automatically update, internally, definitions of pinned packages as well as repositories in the 1.2 format
• however, it is faster to use repositories in the 2.0 format directly. To that end, please use the opam admin upgrade command on your repositories. The --mirror option will create a 2.0 mirror and put in place proper redirections, allowing your original repository to retain the old format

The official opam repository at https://opam.ocaml.org remains in 1.2 format for now, but has a live-updated 2.0 mirror to which you should be automatically redirected. It cannot yet accept package definitions in 2.0 format.

Package format

• Any available: constraints based on the OCaml compiler version should be rewritten into dependencies to the ocaml package
• Separate build: and install: instructions are now required
• It is now preferred to include the old url and descr files (containing the archive URL and package description) in the opam file itself: (see the new synopsis: and description: fields, and the url {} file section)
• Building tests and documentation should now be part of the main build: instructions, using the {test} and {doc} filters. The build-test: and build-doc: fields are still supported.
• It is now possible to use opam variables within dependencies, for example depends: [ "foo" {= version} ], for a dependency to package foo at the same version as the package being defined, or depends: [ "bar" {os = "linux"} ] for a dependency that only applies on Linux.
• The new conflict-class: field allows mutual conflicts among a set of packages to be declared. Useful, for example, when there are many concurrent, incompatible implementations.
• The ocaml-version: field has been deprecated for a long time and is no longer accepted. This should now be a dependency on the ocaml package
• Three types of checksums are now accepted: you should use md5=<hex-value>, sha256=<hex-value> or sha512=<hex-value>. We'll be gradually deprecating md5 in favour of the more secure algorithms; multiple checksums are allowed
• Patches supplied in the patches: field must apply with patch -p1
• The new setenv: field allows packages to export updates to environment variables;
• Custom fields x-foo: can be used for extensions and external tools
• """ delimiters allow unescaped strings
• & has now the customary higher precedence than | in formulas
• Installed files are now automatically tracked meaning that the remove: field is usually no longer required.

The full, up-to-date specification of the format can be browsed in the manual.

Repository format

In the official, default repository, and also when migrating repositories from older format versions, there are:

• A virtual ocaml package, that depends on any implementation of the OCaml compiler. This is what packages should depend on, and the version is the corresponding base OCaml version (e.g. 4.04.0 for the 4.04.0+fp compiler). It also defines various configuration variables, see opam config list ocaml.
• Three mutually-exclusive packages providing actual implementations of the OCaml toolchain:
  • ocaml-base-compiler is the official releases
  • ocaml-variants.<base-version>+<variant-name> contains all the other variants
  • ocaml-system-compiler maps to a compiler installed on the system outside of opam

The layout is otherwise the same, apart from:

• The compilers/ directory is ignored
• A repo file should be present, containing at least the line opam-version: "2.0"
• The indexes for serving over HTTP have been simplified, and urls.txt is no longer needed. See opam admin index --help
• The archives/ directory is no longer used. The cache now uses a different format and is configured through the repo file, defaulting to cache/ on the same server. See opam admin cache --help

Feedback

Thanks for trying out the beta! Please let us have feedback, preferably to the opam tracker; other options include the opam-devel list and #opam IRC channel on Freenode.

Minor release.

opam-lib 1.3

The package for opam-lib version 1.3 has just been released in the official opam repository. There is no release of opam with version 1.3, but this is an intermediate version of the library that retains compatibility of the file formats with 1.2.2.

The purpose of this release is twofold:

• provide some fixes and enhancements over opam-lib 1.2.2. For example, 1.3 has an enhanced lint function
• be a step towards migration to opam-lib 2.0.

This version is compatible with the current stable release of opam (1.2.2), but dependencies have been updated so that you are not (e.g.) stuck on an old version of ocamlgraph.

Therefore, I encourage all maintainers of tools based on opam-lib to migrate to 1.3.

The respective APIs are available in HTML for 1.2 and 1.3.

A note on plugins: when you write opam-related tools, remember that by setting flags: plugin in their definition and installing a binary named opam-toolname, you will enable the users to install package toolname and run your tool with a single opam toolname command.

Architectural changes

If you need to migrate from 1.2 to 1.3, these tips may help:

• there are now 6 different ocamlfind sub-libraries instead of just 4: format contains the handlers for opam types and file formats, has been split out from the core library, while state handles the state of a given opam root and switch and has been split from the client library.

• OpamMisc is gone and moved into the better organised OpamStd, with submodules for String, List, etc.

• OpamGlobals is gone too, and its contents have been moved to:

  • OpamConsole for the printing, logging, and shell interface handling part
  • OpamXxxConfig modules for each of the libraries for handling the global configuration variables. You should call the respective init functions, with the options you want to set, for proper initialisation of the lib options (and handling the OPAMXXX environment variables)

• OpamPath.Repository is now OpamRepositoryPath, and part of the repository sub-library.

opam-lib 2.0 ?

The development version of the opam-lib (2.0~alpha5 as of writing) is already available on opam. The name has been changed to provide a finer granularity, so it can actually be installed concurrently -- but be careful not to confuse the ocamlfind package names (opam-lib.format for 1.3 vs opam-format for 2.0).

The provided packages are:

• opam-file-format: now separated from the opam source tree, this has no dependencies and can be used to parse and print the raw opam syntax.
• opam-core: the basic toolbox used by opam, which actually doesn't include the opam specific part. Includes a tiny extra stdlib, the engine for running a graph of processes in parallel, some system handling functions, etc. Depends on ocamlgraph and re only.
• opam-format: defines opam data types and their file i/o functions. Depends just on the two above.
• opam-solver: opam's interface with the dose3 library and external solvers.
• opam-repository: fetching repositories and package sources from all handled remote types.
• opam-state: handling of the opam states, at the global, repository and switch levels.
• opam-client: the client library, providing the top-level operations (installing packages...), and CLI.
• opam-devel: this packages the development version of the opam tool itself, for bootstrapping. You can install it safely as it doesn't install the new opam in the PATH.

The new API can be also be browsed ; please get in touch if you have trouble migrating.

This release mainly brings support for OCaml 4.04. Internal code was simplified and bugs were fixed in the meantime (cache invalidation, ast traversal, type error recovery, certain cases of completion, ppx working directory, locate, ...).

Bug fix release before major version.

We are pleased to announce a preview release for opam 2.0, with over 700 patches since 1.2.2. Version 2.0~alpha4 has just been released, and is ready to be more widely tested.

This version brings many new features and changes, the most notable one being that OCaml compiler packages are no longer special entities, and are replaced by standard package definition files. This in turn means that opam users have more flexibility in how switches are managed, including for managing non-OCaml environments such as Coq using the same familiar tools.

A Few Highlights

This is just a sample, see the full changelog for more:

• Sandboxed builds: Command wrappers can be configured to, for example, restrict permissions of the build and install processes using Linux namespaces, or run the builds within Docker containers.

• Compilers as packages: This brings many advantages for opam workflows, such as being able to upgrade the compiler in a given switch, better tooling for local compilers, and the possibility to define coq as a compiler or even use opam as a generic shell scripting engine with dependency tracking.

• Local switches: Create switches within your projects for easier management. Simply run opam switch create <directory> <compiler> to get started.

• Inplace build: Use opam to build directly from your source directory. Ensure the package is pinned locally then run opam install --inplace-build.

• Automatic file tracking:: opam now tracks the files installed by packages and is able to cleanly remove them when no existing files were modified. The remove: field is now optional as a result.

• Configuration file: This can be used to direct choices at opam init automatically (e.g. specific repositories, wrappers, variables, fetch commands, or the external solver). This can be used to override all of opam's OCaml-related settings.

• Simpler library: the OCaml API is completely rewritten and should make it much easier to write external tools and plugins. Existing tools will need to be ported.

• Better error mitigation: Through clever ordering of the shell actions and separation of build and install, most build failures can keep your current installation intact, not resulting in removed packages anymore.

Roll out

You are very welcome to try out the alpha, and report any issues. The repository at opam.ocaml.org will remain in 1.2 format (with a 2.0 mirror at opam.ocaml.org/2.0~dev in sync) until after the release is out, which means the extensions can not be used there yet, but you are welcome to test on local or custom repositories, or package pinnings. The reverse translation (2.0 to 1.2) is planned, to keep supporting 1.2 installations after that date.

The documentation for the new version is available at https://opam.ocaml.org/doc/2.0/. This is still work in progress, so please do ask if anything is unclear.

Interface changes

Commands opam switch and opam list have been rehauled for more consistency and flexibility: the former won't implicitly create new switches unless called with the create subcommand, and opam list now allows to combine filters and finely specify the output format. They may not be fully backwards compatible, so please check your scripts.

Most other commands have also seen fixes or improvements. For example, opam doesn't forget about your set of installed packages on the first error, and the new opam install --restore can be used to reinstall your selection after a failed upgrade.

Repository changes

While users of opam 1.2 should feel at home with the changes, the 2.0 repository and package formats are not compatible. Indeed, the move of the compilers to standard packages implies some conversions, and updates to the relationships between packages and their compiler. For example, package constraints like

available: [ ocaml-version >= "4.02" ]

are now written as normal package dependencies:

depends: [ "ocaml" {>= "4.02"} ]

To make the transition easier,

• upgrade of a custom repository is simply a matter of running opam-admin upgrade-format at its root;
• the official repository at opam.ocaml.org already has a 2.0 mirror, to which you will be automatically redirected;
• packages definition are automatically converted when you pin a package.

Note that the ocaml package on the official repository is actually a wrapper that depends on one of ocaml-base-compiler, ocaml-system or ocaml-variants, which contain the different flavours of the actual compiler. It is expected that it may only get picked up when requested by package dependencies.

Package format changes

The opam package definition format is very similar to before, but there are quite a few extensions and some changes:

• it is now mandatory to separate the build: and install: steps (this allows tracking of installed files, better error recovery, and some optional security features);
• the url and description can now optionally be included in the opam file using the section url {} and fields synopsis: and description:;
• it is now possible to have dependencies toggled by globally-defined opam variables (e.g. for a dependency needed on some OS only), or even rely on the package information (e.g. have a dependency at the same version);
• the new setenv: field allows packages to export updates to environment variables;
• custom fields x-foo: can be used for extensions and external tools;
• allow """ delimiters around unescaped strings
• & is now parsed with higher priority than |
• field ocaml-version: can no longer be used
• the remove: field should not be used anymore for simple cases (just removing files)

Let's go then -- how to try it ?

First, be aware that you'll be prompted to update your ~/.opam to 2.0 format before anything else, so if you value it, make a backup. Or just export OPAMROOT to test the alpha on a temporary opam root.

Packages for opam 2.0 are already in the opam repository, so if you have a working opam installation of opam (at least 1.2.1), you can bootstrap as easily as:

opam install opam-devel

This doesn't install the new opam to your PATH within the current opam root for obvious reasons, so you can manually install it as e.g. "opam2" using:

sudo cp $(opam config var "opam-devel:lib")/opam /usr/local/bin/opam2

You can otherwise install as usual:

• Using pre-built binaries (available for OSX and Linux x86, x86_64, armhf) and our install script:

  wget https://raw.github.com/ocaml/opam/2.0-alpha4-devel/shell/opam_installer.sh -O - | sh -s /usr/local/bin

  Equivalently, pick your version and download it to your PATH;

• Building from our inclusive source tarball: download here and build using ./configure && make lib-ext && make && make install if you have OCaml >= 4.01 already available, make cold && make install otherwise;

• Or from source, following the included instructions from the README. Some files have been moved around, so if your build fails after you updated an existing git clone, try to clean it up (git clean -fdx).

This release also contains contributions from: Rudi Grinberg, Fourchaux, Christopher Reichert, David Allsopp, Nick Borden, Mario Rodas, @Twinside, Pierre Chambart, Philipp Haselwarter, Tomasz Kołodziejski and Syohei Yoshida.