package ppx_expect

  1. Overview
  2. Docs
Cram like framework for OCaml

Install

Dune Dependency

Authors

Maintainers

Sources

ppx_expect-v0.11.1.tbz
sha256=78c02b155f89c81357479e678cba6772ac22d7fd5940a04387cf8c348d9ddc19
md5=ee5e03094674de295aadc10efe6bb7b7

Description

Part of the Jane Street's PPX rewriters collection.

Published: 04 Oct 2018

README

README.org

#+TITLE: expect-test - a cram like framework for OCaml
#+PARENT: ../README.md

** Introduction

Expect-test is a framework for writing tests in OCaml, similar to [[https://bitheap.org/cram/][Cram]].
Expect-tests mimic the existing inline tests framework with the =let%expect_test= construct.
The body of an expect-test can contain output-generating code, interleaved with =%expect= extension
expressions to denote the expected output.

When run, these tests will pass iff the output matches what was expected. If a test fails, a
corrected file with the suffix ".corrected" will be produced with the actual output, and the
=inline_tests_runner= will output a diff.

Here is an example Expect-test program, say in =foo.ml=

#+begin_src ocaml
open Core

let%expect_test "addition" =
  printf "%d" (1 + 2);
  [%expect {| 4 |}]
#+end_src

When the test is run (as part of =inline_tests_runner=), =foo.ml.corrected= will be produced with the
contents:

#+begin_src ocaml
open Core

let%expect_test "addition" =
  printf "%d" (1 + 2);
  [%expect {| 3 |}]
#+end_src

=inline_tests_runner= will also output the diff:

#+begin_src
---foo.ml
+++foo.ml.corrected
File "foo.ml", line 5, characters 0-1:
  open Core

  let%expect_test "addition" =
    printf "%d" (1 + 2);
-|  [%expect {| 4 |}]
+|  [%expect {| 3 |}]
#+end_src

Diffs will be shown in color if the =-use-color= flag is passed to the test runner executable.

** Expects reached from multiple places

A [%expect] can exist in a way that it is encountered multiple times, e.g. in a
functor or a function:

#+begin_src ocaml
let%expect_test _ =
  let f output =
    print_string output;
    [%expect {| hello world |}]
  in
  f "hello world";
  f "hello world";
;;
#+end_src

The =[%expect]= should capture the exact same output (i.e. up to string equality) at every
invocation. In particular, this does **not** work:

#+begin_src ocaml
let%expect_test _ =
  let f output =
    print_string output;
    [%expect {| \(foo\|bar\) (regexp) |}]
  in
  f "foo";
  f "bar";
;;
#+end_src

** Output matching

Matching is done on a line-by-line basis. If any output line fails to
match its expected output, the expected line is replaced with the
actual line in the final output.

*** Whitespace

Inside =%expect= nodes, whitespace around patterns are ignored, and
the user is free to put any amount for formatting purposes. The same
goes for the actual output.

Ignoring surrounding whitespace allows to write nicely formatted
expectation and focus only on matching the bits that matter.

To do this, ppx_expect strips patterns and outputs by taking the
smallest rectangle of text that contains the non-whitespace
material. All end of line whitespace are ignored as well. So for
instance all these lines are equivalent:

#+begin_src ocaml
  print blah;
  [%expect {|
abc
defg
  hij|}]

  print blah;
  [%expect {|
                abc
                defg
                  hij
  |}]

  print blah;
  [%expect {|
    abc
    defg
      hij
  |}]
#+end_src

However, the last one is nicer to read.

For the rare cases where one does care about what the exact output is,
ppx_expect provides the =%expect_exact= extension point, which only
succeed when the untouched output is exactly equal to the untouched
pattern.

When producing a correction, ppx_expect tries to respect as much as
possible the formatting of the pattern.

** Integration with Async, Lwt or other cooperative libraries

If you are writing expect tests for a system using Async, Lwt or any
other libraries for cooperative threading, you need some preparation
so that everything works well. For instance, you probably need to
flush some =stdout= channel. The expect test runtime takes care of
flushing =Pervasives.stdout= but it doesn't know about
=Async.Writer.stdout=, =Lwt_io.stdout= or anything else.

To deal with this, expect\_test provides some hooks in the form of a
condifuration module =Expect_test_config=. The default module in scope
define no-op hooks that the user can override. =Async= redefines
this module so when =Async= is opened you can write async-aware
expect test.

This is what you would need to write to do the same with Lwt:

#+begin_src ocaml
module Expect_test_config
  : Expect_test_config.S with module IO = Lwt =
struct
  module IO = Lwt
  let flush () = Lwt_io.(flush stdout)
  let run = Lwt_main.run
end
#+end_src

** Comparing Expect-test and unit testing (e.g. =let%test_unit=)

The simple example above can be easily represented as a unit test:

#+begin_src ocaml
let%test_unit "addition" = [%test_result: int] (1 + 2) ~expect:4
#+end_src

So, why would one use Expect-test rather than a unit test?  There are
several differences between the two approaches.

With a unit test, one must write code that explicitly checks that the
actual behavior agrees with the expected behavior.  =%test_result= is
often a convenient way of doing that, but even using that requires:

- creating a value to compare
- writing the type of that value
- having a comparison function on the value
- writing down the expected value

With Expect-test, we can simply add print statements whose output gives
insight into the behavior of the program, and blank =%expect=
attributes to collect the output.  We then run the program to see if
the output is acceptable, and if so, *replace* the original program
with its output.  E.g we might first write our program like this:

#+begin_src ocaml
let%expect_test _ =
  printf "%d" (1 + 2);
  [%expect {||}]
#+end_src

The corrected file would contain:

#+begin_src ocaml
let%expect_test _ =
  printf "%d" (1 + 2);
  [%expect {| 3 |}]
#+end_src

With Expect-test, we only have to write code that prints things that we
care about.  We don't have to construct expected values or write code
to compare them.  We get comparison for free by using diff on the
output.  And a good diff (e.g. patdiff) can make understanding
differences between large outputs substantially easier, much easier
than typical unit-testing code that simply states that two values
aren't equal.

Once an Expect-test program produces the desired expected output and we
have replaced the original program with its output, we now
automatically have a regression test going forward.  Any undesired
change to the output will lead to a mismatch between the source
program and its output.

With Expect-test, the source program and its output are interleaved.  This
makes debugging easier, because we do not have to jump between source
and its output and try to line them up.  Furthermore, when there is a
mismatch, we can simply add print statements to the source program and
run it again.  This gives us interleaved source and output with the
debug messages interleaved in the right place.  We might even insert
additional empty =%%expect= attributes to collect debug messages.

** Implementation

Every =%expect= node in an Expect-test program becomes a point at which
the program output is captured. Once the program terminates, the
captured outputs are matched against the expected outputs, and interleaved with
the original source code to produce the corrected file. Trailing output is appended in a
new =%expect= node.

** Build system integration

Follow the same rules as for [[https://github.com/janestreet/ppx_inline_test][ppx_inline_test]]. Just make sure to
include =ppx_expect.evaluator= as a dependency of the test runner. The
[[https://github.com/janestreet/jane-street-tests][Jane Street tests]] contains a few working examples using oasis.

** Output patterns

Lines in an =%expect= can end with a "tag" indicating the kind of
match to perform.  This functionality is deprecated because it
interferes with the smooth expect-test workflow of accepting output.
One should instead use output post-processing.

To enable support for output patterns, your =jbuild= file should have:

=((inline_tests ((flags (-allow-output-patterns)))))=

Here are the different kinds of output patterns.

The =(regexp)= tag will perform regexp matching on the given line:

#+begin_src ocaml
printf "foo";
[%expect {| foo\|bar (regexp) |}]
#+end_src

Similarly, the =(glob)= tag will perform glob matching on the given
line:

#+begin_src ocaml
printf "foobarbaz";
[%expect {| {foo,hello}* (glob) |}]
#+end_src

The =(literal)= tag will force a literal match on a line, and can be
useful in edge cases:

#+begin_src ocaml
printf "foo*bar (regexp)";
[%expect {| foo*bar (regexp) (literal) |}]
#+end_src

The =(escaped)= tag will treat the line as an escaped literal string,
which can be useful for matching unprintable characters. It doesn't
support escaped newlines right now.

Dependencies (15)

  1. ocaml >= "4.04.1"
  2. re >= "1.5.0"
  3. ppxlib >= "0.1.0" & < "0.9.0"
  4. ocaml-migrate-parsetree >= "1.0" & < "2.0.0"
  5. jbuilder >= "1.0+beta18.1"
  6. stdio >= "v0.11" & < "v0.12"
  7. ppx_variants_conv >= "v0.11" & < "v0.12"
  8. ppx_sexp_conv >= "v0.11" & < "v0.12"
  9. ppx_inline_test >= "v0.11" & < "v0.12"
  10. ppx_here >= "v0.11" & < "v0.12"
  11. ppx_fields_conv >= "v0.11" & < "v0.12"
  12. ppx_custom_printf >= "v0.11" & < "v0.12"
  13. ppx_compare >= "v0.11" & < "v0.12"
  14. ppx_assert >= "v0.11" & < "v0.12"
  15. base >= "v0.11" & < "v0.12"

Dev Dependencies

None

  1. arrayjit
  2. autofonce
  3. autofonce_config
  4. autofonce_core
  5. autofonce_lib
  6. autofonce_m4
  7. autofonce_misc
  8. autofonce_patch
  9. autofonce_share
  10. bitpack_serializer
  11. bitwuzla < "1.0.0"
  12. camelot >= "1.3.0" & < "1.4.2"
  13. charInfo_width
  14. combinaml
  15. ctypes_stubs_js
  16. cudajit
  17. dap
  18. data-encoding >= "0.6"
  19. dataframe
  20. dream < "1.0.0~alpha5"
  21. dream-pure
  22. drom
  23. drom_lib
  24. drom_toml
  25. dune-action-plugin
  26. electrod >= "0.1.6" & < "0.2.1"
  27. ez_cmdliner >= "0.2.0"
  28. ez_config >= "0.2.0"
  29. ez_file >= "0.2.0"
  30. ez_hash < "0.5.3"
  31. ez_opam_file
  32. ez_search
  33. ez_subst
  34. fiat-p256 < "0.2.0"
  35. fiber >= "3.7.0"
  36. fiber-lwt
  37. GT >= "0.4.0" & < "0.5.0"
  38. gccjit
  39. header-check
  40. hl_yaml
  41. http
  42. http-cookie >= "4.0.0"
  43. http-multipart-formdata >= "2.0.0"
  44. hyper
  45. imguiml
  46. influxdb >= "0.2.0"
  47. js_of_ocaml >= "3.10.0" & < "4.0.0"
  48. js_of_ocaml-compiler >= "3.4.0" & < "3.5.2"
  49. js_of_ocaml-lwt >= "3.10.0" & < "4.0.0"
  50. js_of_ocaml-ocamlbuild >= "3.10.0" & < "5.0"
  51. js_of_ocaml-ppx >= "3.10.0" & < "4.0.0"
  52. js_of_ocaml-ppx_deriving_json >= "3.10.0" & < "4.0.0"
  53. js_of_ocaml-toplevel >= "3.10.0" & < "4.0.0"
  54. js_of_ocaml-tyxml >= "3.10.0" & < "4.0.0"
  55. kdl
  56. knights_tour
  57. kqueue >= "0.2.0"
  58. learn-ocaml >= "0.16.0"
  59. learn-ocaml-client >= "0.16.0"
  60. libbpf
  61. little_logger < "0.3.0"
  62. loga >= "0.0.5"
  63. lsp >= "1.11.3" & < "1.12.2"
  64. merge-fmt >= "0.3"
  65. mlt_parser = "v0.11.0"
  66. module-graph
  67. neural_nets_lib
  68. nice_parser
  69. nloge
  70. nsq >= "0.4.0" & < "0.5.2"
  71. OCanren-ppx >= "0.3.0~alpha1"
  72. ocaml-protoc-plugin
  73. ocp-search
  74. ocplib_stuff >= "0.3.0"
  75. octez-libs
  76. octez-protocol-009-PsFLoren-libs
  77. octez-protocol-010-PtGRANAD-libs
  78. octez-protocol-011-PtHangz2-libs
  79. octez-protocol-012-Psithaca-libs
  80. octez-protocol-013-PtJakart-libs
  81. octez-protocol-014-PtKathma-libs
  82. octez-protocol-015-PtLimaPt-libs
  83. octez-protocol-016-PtMumbai-libs
  84. octez-protocol-017-PtNairob-libs
  85. octez-protocol-018-Proxford-libs
  86. octez-protocol-019-PtParisB-libs
  87. octez-protocol-020-PsParisC-libs
  88. octez-protocol-alpha-libs
  89. octez-shell-libs
  90. odate >= "0.6"
  91. odoc >= "2.0.0"
  92. odoc-parser
  93. omd >= "2.0.0~alpha3"
  94. opam-bin >= "0.9.5"
  95. opam-check-npm-deps
  96. opam_bin_lib >= "0.9.5"
  97. owork
  98. passage
  99. poll
  100. pp
  101. ppx_deriving_jsonschema >= "0.0.2"
  102. ppx_jane = "v0.11.0"
  103. ppx_minidebug
  104. ppx_protocol_conv_json >= "4.0.0"
  105. ppx_relit >= "0.2.0"
  106. ppx_ts
  107. psmt2-frontend >= "0.3.0"
  108. pvec
  109. res_tailwindcss
  110. routes >= "2.0.0"
  111. safemoney >= "0.1.1"
  112. sarif
  113. sedlex >= "3.1"
  114. seqes < "0.2"
  115. solidity-alcotest
  116. solidity-common
  117. solidity-parser
  118. solidity-test
  119. solidity-typechecker
  120. spawn < "v0.9.0" | >= "v0.13.0"
  121. tezos-benchmark
  122. tezos-client-009-PsFLoren >= "14.0"
  123. tezos-client-010-PtGRANAD >= "14.0"
  124. tezos-client-011-PtHangz2 >= "14.0"
  125. tezos-client-012-Psithaca >= "14.0"
  126. tezos-client-013-PtJakart >= "14.0"
  127. tezos-client-014-PtKathma
  128. tezos-client-015-PtLimaPt
  129. tezos-client-016-PtMumbai
  130. tezos-client-017-PtNairob
  131. tezos-client-alpha >= "14.0"
  132. tezos-injector-013-PtJakart
  133. tezos-injector-014-PtKathma
  134. tezos-injector-015-PtLimaPt
  135. tezos-injector-016-PtMumbai
  136. tezos-injector-alpha
  137. tezos-layer2-utils-016-PtMumbai
  138. tezos-layer2-utils-017-PtNairob
  139. tezos-micheline >= "14.0"
  140. tezos-shell >= "15.0"
  141. tezos-smart-rollup-016-PtMumbai
  142. tezos-smart-rollup-017-PtNairob
  143. tezos-smart-rollup-alpha
  144. tezos-smart-rollup-layer2-016-PtMumbai
  145. tezos-smart-rollup-layer2-017-PtNairob
  146. tezos-stdlib >= "14.0"
  147. tezos-tx-rollup-013-PtJakart
  148. tezos-tx-rollup-014-PtKathma
  149. tezos-tx-rollup-015-PtLimaPt
  150. tezos-tx-rollup-alpha
  151. toplevel_expect_test = "v0.11.0"
  152. torch < "v0.16.0"
  153. travesty < "0.6.0" | = "0.6.2"
  154. wtr >= "2.0.0"
  155. wtr-ppx
  156. zanuda

Conflicts

None

OCaml

Innovation. Community. Security.