package ppxlib
Install
Dune Dependency
Authors
Maintainers
Sources
sha256=2e134cc8d9252e0d3404370eb7eaa28e1fdabb030d4720b4dec4c56ec5e45c51
md5=bb278ff6e819e0e4a4d8a005bb2512a4
Description
A comprehensive toolbox for ppx development. It features:
- a OCaml AST / parser / pretty-printer snapshot,to create a full frontend independent of the version of OCaml;
- a library for library for ppx rewriters in general, and type-driven code generators in particular;
- a feature-full driver for OCaml AST transformers;
- a quotation mechanism allowing to write values representing the OCaml AST in the OCaml syntax;
- a generator of open recursion classes from type definitions.
Published: 10 Jan 2019
README
Ppxlib - Comprehensive ppx tool set
This readme is still under construction.
History
This repository is the merge of several ppx-related repositories, namely:
Future development will happen only in this repository, and the aforementioned ones will only contain synonym definitions to provide backward compatibility. The structure of this repository is as follows:
ast/
contains theppxlib.ast
library, that replacesppx_ast
;src/
contains theppxlib
library, that replacesppx_core
,ppx_driver
, andtype_conv
;metaquot/
contains theppxlib.metaquot
library, that replacesppx_metaquot
;metaquot_lifters/
contains theppxlib.metaquot_lifters
library, that replacesppx_metaquot.lifters
;print-diff/
contains theppxlib.print_diff
library, that replacesppx_driver.print_diff
;runner/
contains theppxlib.runner
library, that replacesppx_driver.runner
;runner_as_ppx/
contains theppxlib.runner_as_ppx
library, that replacesppx_driver.runner_as_ppx
;traverse/
contains theppxlib.traverse
library, that replacesppx_traverse
;traverse_builtins/
contains theppxlib.traverse_builtins/
library, that replacesppx_traverse_builtins
.
Ast
Ppxlib_ast
selects a specific version of the OCaml Abstract Syntax Tree from the ocaml-migrate-parsetree project that is not necessarily the same one as the one being used by the compiler.
It also snapshots the corresponding parser and pretty-printer from the OCaml compiler, to create a full frontend independent of the version of OCaml.
This AST is used in all Jane Street ppx rewriters, and more generally in all ppx rewriters based on Ppxlib. Using a different AST allows to "detach" the ppx code from the compiler libraries, and allow to use ppx rewriters with new compilers before upgrading the ppx code.
Ppxlib
Ppxlib is a standard library for OCaml AST transformers, that uses the AST from Ppxlib_ast
. It features:
various auto-generated AST traversal using an open recursion scheme
helpers for building AST fragments
helpers for matching AST fragments
a framework for dealing with attributes and extension points
spellchecking and other hints on misspelled/misplaced attributes and extension points
checks for unused attributes (they are otherwise silently dropped by the compiler)
Other ASTs
If you want to write code that works with several versions of Ppxlib
using different AST versions, you can use the versionned alternatives for Ast_builder
and Ast_pattern
. For instance:
open Ppxlib
module Ast_builder = Ast_builder_403
module Ast_pattern = Ast_pattern_403
Drivers
A driver is an executable created from a set of OCaml AST transformers linked together with a command line frontend.
The aim is to provide a tool that can be used to:
easily view the pre-processed version of a file, no need to construct a complex command line:
ppx file.ml
will do;use a single executable to run several transformations: no need to fork many times just for pre-processing;
improved errors for misspelled/misplaced attributes and extension points.
Using driver-based rewriters
The recommended way to use rewriters based on Ppxlib.Driver
is through dune. All you need to is add this line to your (library ...)
or (executables ...)
stanza:
(preprocess (pps (rewriter1 rewriter2 ... ppxlib.runner)))
dune will automatically build a static driver including all these rewriters. Note the ppxlib.runner
at the end of the list, it will still work if you don't put but some specific features of ppxlib
won't be available.
If you are not using dune, you can build a custom driver yourself using ocamlfind.
These methods are described in the following sections.
Creating a new Ppx_driver based rewriter
If using dune, you can just use the following jbuild file:
(library
((name my_ppx)
(public_name my_ppx)
(kind ppx_rewriter)
(libraries (ppxlib))
(ppx_runtime_libraries (<runtime dependencies if any>))
(preprocess (pps (ppx_metaquot)))))
(kind ppx_driver)
has two effects:
it links the library with
-linkall
. Since plugins register themselves with the Ppx_driver library by doing a toplevel side effect, you need to be sure they are linked in the static driver to be taken into accound;it instructs dune to produce a special META file that is compatible with the various ways of using ppx rewriters, i.e. for people not using dune.
Building a custom driver using ocamlfind
To build a custom driver using ocamlfind, simply link all the AST transformers together with the ppxlib.runner
package at the end:
ocamlfind ocamlopt -predicates ppx_driver -o ppx -linkpkg \
-package ppx_sexp_conv -package ppx_bin_prot \
-package ppxlib.runner
Normally, ppxlib.driver
-based rewriters should be build with the approriate -linkall
option on individual libraries. If one is missing this option, the code rewriter might not get linked in. If this is the case, a workaround is to pass -linkall
when linking the custom driver.
Building rewriter that you are currently developing
Note: if using dune, you do not need to read this as dune already does all the right things for you. This section is written having ocamlbuild in mind.
When developing a new rewriter you are very likely to prepare a few tests for it. The compilation line above doesn't suit this task very well (because ocamlfind package with your rewriter is not yet installed) and it will be more convenient to specify .cmx[a]
with your rewriter manually.
For example, let's suppose that the standalone rewriter (pp_foo.native
) have this code
let () = Ppxlib.Driver.standalone ()
in pp_foo.ml
and your generator is begin loaded in ppx_foo.ml
. You need a few extra switches to compile standalone rewriter (N.B. order matters)
ocamlfind ... dependecy1_of_ppx_foo.cmx ... ppx_foo.cmx -package ppxlib -o pp_foo.native
or, if you have already created ppx_foo.cmxa
using -linkall
option
ocamlfind ... ppx_foo.cmxa -package ppxlib -o pp_foo.native
And now you can specify that your test suite uses your rewriter and depends on a few extra .cma
's by adding a few lines into your _tags
file
<regression/test*.*>: ppx(./pp_foo.native --as-ppx)
<regression/test*.*>: depends_on_foo
and specifing dependencies in your myocamlbuild.ml
file using
dep ["compile";"depends_on_foo"] ["ppx_foo.cmxa"; "pp_foo.native"]
The driver as a command line tool
It recognizes the following command-line switches:
-loc-filename <string> File name to use in locations
-reserve-namespace <string> Mark the given namespace as reserved
-no-check Disable checks (unsafe)
-apply <names> Apply these transformations in order (comma-separated list)
-dont-apply <names> Exclude these transformations
-no-merge Do not merge context free transformations (better for debugging rewriters)
-as-ppx Run as a -ppx rewriter (must be the first argument)
--as-ppx Same as -as-ppx
-as-pp Shorthand for: -dump-ast -embed-errors
--as-pp Same as -as-pp
-o <filename> Output file (use '-' for stdout)
- Read input from stdin
-dump-ast Dump the marshaled ast to the output file instead of pretty-printing it
--dump-ast Same as -dump-ast
-dparsetree Print the parsetree (same as ocamlc -dparsetree)
-embed-errors Embed errors in the output AST (default: true when -dump-ast, false otherwise)
-null Produce no output, except for errors
-impl <file> Treat the input as a .ml file
--impl <file> Same as -impl
-intf <file> Treat the input as a .mli file
--intf <file> Same as -intf
-debug-attribute-drop Debug attribute dropping
-print-transformations Print linked-in code transformations, in the order they are applied
-print-passes Print the actual passes over the whole AST in the order they are applied
-ite-check No effect (kept for compatibility)
-pp <command> Pipe sources through preprocessor <command> (incompatible with -as-ppx)
-reconcile (WIP) Pretty print the output using a mix of the input source and the generated code
-reconcile-with-comments (WIP) same as -reconcile but uses comments to enclose the generated code
-no-color Don't use colors when printing errors
-diff-cmd Diff command when using code expectations
-pretty Instruct code generators to improve the prettiness of the generated code
-styler Code styler
-help Display this list of options
--help Display this list of options
When passed a file as argument, a ppx driver will pretty-print the code transformed by all its built-in AST transformers. This gives a convenient way of seeing the code generated for a given attribute/extension.
A driver can simply be used as the argument of the -pp
option of the OCaml compiler, or as the argument of the -ppx
option by passing -as-ppx
as first argument:
$ ocamlc -c -pp "ppx -as-pp" file.ml
$ ocamlc -c -ppx "ppx -as-ppx" file.ml
Rewriters as findlib libraries
Note: if using dune, you do not need to read this as dune already does all the right things for you.
In normal operation, Ppxlib.Driver rewriters are packaged as findlib libraries. When using dune everything is simple as preprocessors and normal dependencies are separated. However historically, people have been specifying both preprocessors and normal library dependencies together. Even worse, many build system still don't use a static driver and call out to multiple ppx commands to preprocess a single file, which slow downs compilation a lot.
In order for all these different methods to work properly, you need a peculiar META file. The rules are explained below.
It is recommended to split the findlib package into two:
one for the main library, which almost assume it is just a normal library;
another sub-package one for:
allowing to mix preprocessors and normal dependencies;
the method of calling one executable per rewriter.
In the rest we'll assume we are writing a META file for a ppx_foo
rewriter, that itself uses the ppxlib
and re
libraries, and produces code using ppx_foo.runtime-lib
.
We want the META file to support all of these:
mix normal dependencies and preprocessors, using one executable per rewriter:
ocamlfind ocamlc -package ppx_foo -c toto.ml
mix normal dependencies and preprocessors, using a single ppx driver:
$ ocamlfind ocamlc -package ppx_foo -predicates custom_ppx \ -ppx ./custom-driver.exe -c toto.ml
build a custom driver:
$ ocamlfind ocamlc -linkpkg -package ppx_foo -predicates ppx_driver \ -o custom-driver.exe
build systems properly specifying preprocessors as such, separated from normal dependencies, as dune does
Since preprocessors and normal dependencies are always specified separately in jbuild files, dune just always set the ppx_driver
predicates.
In the end the META file should look like this:
# Standard package, expect it assumes that the "ppx_driver" predicate is set
version = "42.0"
description = "interprets [%foo ...] extensions"
requires(ppx_driver) = "ppxlib re"
archives(ppx_driver,byte) = "ppx_foo.cma"
archives(ppx_driver,native) = "ppx_foo.cmxa"
plugin(ppx_driver,byte) = "ppx_foo.cma"
plugin(ppx_driver,native) = "ppx_foo.cmxs"
# This is what dune uses to find out the runtime dependencies of
# a preprocessor
ppx_runtime_deps = "ppx_foo.runtime-lib"
# This line makes things transparent for people mixing preprocessors
# and normal dependencies
requires(-ppx_driver) = "ppx_foo.deprecated-ppx-method"
package "deprecated-ppx-method" (
description = "glue package for the deprecated method of using ppx"
requires = "ppx_foo.runtime-lib"
ppx(-ppx_driver,-custom_ppx) = "./as-ppx.exe"
)
package "runtime-lib" ( ... )
You can check that this META works for all the 4 methods described above.
Derivers
The Ppxlib.Deriving
module factors out functionality needed by different preprocessors that generate code from type specifications. Example libraries currently depending on Deriving
:
ppx_bin_prot
;ppx_compare
;ppx_fields_conv
;ppx_sexp_conv
;ppx_variants_conv
.
Compatibility with ppx_deriving
Ppxlib.Deriving
-based code generators are meant to be used with Ppxlib.Driver
. However Deriving
allows to export a compatible ppx_deriving
plugin. By default, when not linked as part of a driver, packages using Deriving
will just use ppx_deriving
.
So for instance this will work as expected using ppx_deriving
:
ocamlfind ocamlc -c -package ppx_sexp_conv foo.ml
For end users, the main advantage of using Deriving
-based generators is that it will catch typos and attributes misplacement. For instance:
# type t = int [@@derivin sexp]
Error: Attribute `derivin' was not used
Hint: Did you mean deriving?
# type t = int [@@deriving sxp]
Error: ppxlib_deriving: 'sxp' is not a supported type deriving generator
Hint: Did you mean sexp?
# type t = int [@deriving sexp]
Error: Attribute `deriving' was not used
Hint: `deriving' is available for type declarations, type extensions
and extension constructors but is used here in the context of a core type.
Did you put it at the wrong level?"
Deriving Syntax
This section is only relevant if you are not using ppx_deriving
.
Deriving
interprets the [@@deriving ...]
attributes on type declarations, exception declarations and extension constructor declarations:
type t = A | B [@@deriving sexp, bin_io]
sexp
and bin_io
are called generators. They are functions that generate code given the declaration. These functions are implemented by external libraries such as ppx_sexp_conv
or ppx_bin_prot
. Deriving
itself provides no generator, it does only the dispatch.
Generators can take arguments. This is done using the following syntax:
type t = A | B [@@deriving foo ~arg:42]
For arguments that are just switches, it is common to use the following syntax:
type t = A | B [@@deriving foo ~bar]
Metaquot (and Metaquot_lifters)
Ppxlib_metaquot
is a ppx rewriter allowing you to write values representing the OCaml AST in the OCaml syntax.
For instance:
[%expr x + 1]
is a value of type Ppxlib_ast.Ast.expression
, represention the OCaml expression x + 1
.
Ppxlib_metaquot
is similar to ppx_tools.metaquot, expect that:
it uses the version of the OCaml AST defined by Ppxlib_ast rather than the one from the current compiler
it can be used simultaneously with other rewriters using
Ppxlib.Driver
.
Ppxlib_metaquot_lifters
provides lifting functions for OCaml predefined types (int
, string
, list
, ...).
Traverse (and Traverse_builtins)
Ppxlib_traverse
is a Deriving
plugin generating open recursion classes from type definition. Users can overwrite a specific method of the generated classes in order to specialize the recursion on specific nodes. Ppxlib_traverse
is in particular used to generate the open recursion classes to traverse the OCaml AST.
For instance, this is the kind of code generated (the generated code is between the [@@deriving_inline ...]
and [@@@end]
):
type expression =
| Var of string
| Const of int
| Add of expression * expression
| If of cond * expression * expression
and cond =
| Cond_var of string
| Cond_const of bool
| Cond_and of cond * cond
[@@deriving_inline traverse_map]
class map = object(self)
method virtual int : int -> int
method virtual string : string -> string
method virtual int : int -> int
method expression = function
| Var x -> Var (self#string x)
| Const x -> Const (self#int x)
| Add (x, y) -> Add (self#expression x, self#expression y)
| If (x, y, z) -> If (self#cond x, self#expression y, self#expression z)
method cond = function
| Cond_var x -> Cond_var (self#string x)
| Cond_const x -> Cond_const (self#bool x)
| Cond_and (x, y) -> Cond_and (self#cond x, self#cond y)
[@@end]
Now if you wanted to do a deep-copy of an expression, replacing boolean variable foo
by true
:
let replace_var = object
inherit Ppx_traverse_builtins.map
inherit map as super
method cond = function
| Cond_var "foo" -> Cond_const true
| c -> super#cond c
end
let replace_var expr = replace_var replace_var#expression expr
Ppx_traverse_builtins.map
contains the definition for all the builtin types, such as int
, string
, list
, ...
Classes
Ppx_traverse
can generate the following classes: map
, iter
, fold
, fold_map
, map_with_context
, lift
. [@@deriving traverse]
is an alias to generate all the supported classes.
lift
is a special class that is mostly useful to lift an OCaml constant to the AST that represent this constant. To do so, you can use Ppx_metaquot_lifters
:
type t = { x : int; y : int } [@@deriving traverse_lift]
let expression_of_t ~loc t : Ast.expression =
let lift = object
inherit Ppx_metaquot_lifters.expression_lifters loc
inherit lift
end in
lift#t t
Dependencies (7)
-
stdio
>= "v0.11.0"
-
ppx_derivers
>= "1.0"
-
ocaml-migrate-parsetree
>= "1.0.9" & < "2.0.0"
-
ocaml-compiler-libs
>= "v0.11.0"
-
dune
>= "1.6.0"
-
base
>= "v0.11.0"
-
ocaml
>= "4.04.1" & < "4.08.0"
Dev Dependencies (1)
-
ocamlfind
with-test
Used by (91)
- arrayjit
-
base_quickcheck
< "v0.13.0"
-
bistro
= "0.5.0"
- cconv-ppx
-
comby
< "1.3.0"
-
core
>= "v0.11.1" & < "v0.12.0"
-
elpi
= "1.11.0"
-
embedded_ocaml_templates
< "0.7"
-
fieldslib
= "v0.11.0"
-
frenetic
= "5.0.3"
-
GT
>= "0.5.2"
- genprint
- hack_parallel
-
js_of_ocaml-ppx_deriving_json
>= "3.5.0" & < "3.5.2"
-
lablqml
>= "0.6" & < "0.7"
-
learn-ocaml
>= "0.13.0"
-
learn-ocaml-client
>= "0.13.0"
-
let-if
< "0.3.0"
- logs-ppx
-
melange-json
>= "1.2.0"
-
merlin
>= "4.7-412"
-
mlt_parser
= "v0.12.0"
- neural_nets_lib
-
noCanren
>= "0.3.0~alpha1"
- noise
-
OCanren-ppx
< "0.2.0"
-
obus
= "1.2.0"
- ocamlapi_ppx
- octez-internal-libs
- ojs_base_ppx
-
override
< "0.2.0"
-
parsexp_io
= "v0.11.0"
-
pattern
< "0.1.1"
- ppx-owl-opt
-
ppx_assert
>= "v0.11.0" & < "v0.13.0"
-
ppx_base
>= "v0.11.0" & < "v0.13.0"
-
ppx_bench
>= "v0.11.0" & < "v0.13.0"
-
ppx_bin_prot
>= "v0.11.1" & < "v0.12.1"
-
ppx_blob
= "0.7.2"
- ppx_bsx
-
ppx_compare
>= "v0.11.1" & < "v0.13.0"
-
ppx_compose
>= "0.2.1"
-
ppx_conv_func
>= "v0.11.0" & < "v0.13.0"
-
ppx_csv_conv
>= "v0.11.1" & < "v0.13.0"
-
ppx_custom_printf
>= "v0.11.0" & < "v0.12.1"
-
ppx_deriving_encoding
< "0.2" | = "0.2.3"
-
ppx_deriving_hardcaml
= "v0.12.0"
- ppx_deriving_hash
-
ppx_deriving_jsoo
< "0.2"
-
ppx_deriving_rpc
>= "6.0.0" & < "7.1.0"
- ppx_dream_eml
- ppx_enum
-
ppx_enumerate
>= "v0.11.1" & < "v0.13.0"
-
ppx_expect
>= "v0.11.0" & < "v0.13.0"
-
ppx_factory
< "0.1.0"
-
ppx_fail
>= "v0.11.0" & < "v0.13.0"
-
ppx_fields_conv
>= "v0.11.0" & < "v0.13.0"
-
ppx_fun
>= "0.0.9"
-
ppx_hash
>= "v0.11.1" & < "v0.13.0"
-
ppx_here
>= "v0.11.0" & < "v0.13.0"
-
ppx_import
>= "1.5-3-gbd627d5" & < "1.8.0"
-
ppx_inline_test
>= "v0.11.0" & < "v0.13.0"
-
ppx_jane
>= "v0.11.0" & < "v0.13.0"
-
ppx_js_style
>= "v0.11.0" & < "v0.13.0"
-
ppx_jsobject_conv
>= "0.6.0" & < "0.8.0"
-
ppx_let
>= "v0.11.0" & < "v0.13.0"
-
ppx_module_timer
< "v0.13.0"
- ppx_mysql
-
ppx_optcomp
>= "v0.11.0" & < "v0.13.0"
-
ppx_optional
>= "v0.11.0" & < "v0.13.0"
-
ppx_pipebang
>= "v0.11.0" & < "v0.13.0"
-
ppx_protocol_conv
>= "3.1.0" & < "5.1.0"
- ppx_rapper
-
ppx_relit
>= "0.2.0"
-
ppx_sexp_conv
>= "v0.11.2" & < "v0.13.0"
-
ppx_sexp_message
>= "v0.11.0" & < "v0.13.0"
-
ppx_sexp_value
>= "v0.11.0" & < "v0.13.0"
-
ppx_stable
< "v0.13.0"
-
ppx_string_interpolation
< "1.0.1"
-
ppx_type_conv
>= "v0.11.0"
-
ppx_typerep_conv
>= "v0.11.1" & < "v0.13.0"
-
ppx_variants_conv
>= "v0.11.1" & < "v0.13.0"
-
ppx_xml_conv
>= "v0.11.0" & < "v0.13.0"
-
ppx_yojson
< "1.1.0"
-
qcow
>= "0.11.0"
-
relit_helper
>= "0.2.0"
- res_tailwindcss
- speed
-
spin
>= "0.8.0"
-
toplevel_expect_test
>= "v0.11.0" & < "v0.12.1"
-
variantslib
= "v0.11.0"
Conflicts
None