package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (40)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-lwt-1.1.0.tbz
sha256=212827a49abf4008581c0da53e7ec78a9d639b415380dcb1fdeeb23f3ff083e2
sha512=c47d04b3c7100af703b470b93ff9fe9fe22790415370b6d5972736f46a5c83901717d67caf0c4115d01020d3078dc7f3063838578174921cab352546dad00148

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 03 Apr 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3" & < "1.1.0"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "2.0"

Dev Dependencies (1)

  1. odoc with-doc

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async < "1.0.0" | = "1.1.0"
  4. alcotest-lwt < "1.0.0" | = "1.1.0"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. ambient-context-lwt
  9. angstrom >= "0.7.0"
  10. ansi >= "0.6.0"
  11. anycache >= "0.7.4"
  12. anycache-async
  13. anycache-lwt
  14. archetype >= "1.4.2"
  15. archi
  16. arp
  17. arp-mirage
  18. arrakis
  19. art
  20. asak >= "0.2"
  21. asli >= "0.2.0"
  22. asn1-combinators >= "0.2.2"
  23. atd >= "2.3.3"
  24. atdgen >= "2.10.0"
  25. atdpy
  26. atdts
  27. base32
  28. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  29. bastet
  30. bastet_async
  31. bastet_lwt
  32. bech32
  33. bechamel >= "0.5.0"
  34. bigarray-overlap
  35. bigstring >= "0.3"
  36. bigstring-unix >= "0.3"
  37. bigstringaf
  38. bitlib
  39. blake2
  40. bloomf
  41. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  42. bls12-381-hash
  43. bls12-381-js >= "0.4.2"
  44. bls12-381-js-gen >= "0.4.2"
  45. bls12-381-legacy
  46. bls12-381-signature
  47. bls12-381-unix
  48. blurhash
  49. builder-web
  50. bulletml
  51. bytebuffer
  52. ca-certs
  53. ca-certs-nss
  54. cactus
  55. caldav
  56. calendar >= "3.0.0"
  57. callipyge
  58. camlix
  59. capnp-rpc < "1.2.3"
  60. capnp-rpc-lwt < "0.3"
  61. capnp-rpc-mirage >= "0.9.0"
  62. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  63. carray
  64. carton
  65. cborl
  66. ccss >= "1.6"
  67. cf-lwt
  68. chacha
  69. channel
  70. charrua-client
  71. charrua-client-lwt
  72. charrua-client-mirage < "0.11.0"
  73. checked_oint
  74. checkseum >= "0.0.3"
  75. cid
  76. clarity-lang
  77. class_group_vdf
  78. cohttp >= "0.17.0"
  79. cohttp-curl-async
  80. cohttp-curl-lwt
  81. cohttp-eio >= "6.0.0~beta2"
  82. colombe >= "0.2.0"
  83. color
  84. conan
  85. conan-cli
  86. conan-database
  87. conan-lwt
  88. conan-unix
  89. conduit = "3.0.0"
  90. conex < "0.10.0"
  91. conex-mirage-crypto
  92. conex-nocrypto
  93. cookie
  94. cow >= "2.2.0"
  95. css
  96. css-parser
  97. cstruct >= "3.3.0"
  98. cstruct-sexp
  99. ctypes-zarith
  100. cuid
  101. curly
  102. current_incr
  103. cwe_checker
  104. data-encoding
  105. datakit >= "0.12.0"
  106. datakit-bridge-github >= "0.12.0"
  107. datakit-ci
  108. datakit-client-git >= "0.12.0"
  109. decompress >= "0.8" & < "1.5.3"
  110. depyt
  111. digestif >= "0.8.1"
  112. dispatch >= "0.4.1"
  113. dkim
  114. dkim-bin
  115. dkim-mirage
  116. dns >= "4.0.0"
  117. dns-cli
  118. dns-client >= "4.6.0"
  119. dns-forward < "0.9.0"
  120. dns-forward-lwt-unix
  121. dns-resolver
  122. dns-server
  123. dns-tsig
  124. dnssd
  125. dnssec
  126. docfd >= "2.2.0"
  127. dog < "0.2.1"
  128. domain-name
  129. dream
  130. dream-pure
  131. duff
  132. dune-release >= "1.0.0"
  133. duration >= "0.1.1"
  134. emile
  135. encore
  136. eqaf >= "0.5"
  137. equinoxe
  138. equinoxe-cohttp
  139. equinoxe-hlc
  140. eris
  141. eris-lwt
  142. ezgzip
  143. ezjsonm >= "0.4.2" & < "1.3.0"
  144. ezjsonm-lwt < "1.3.0"
  145. FPauth
  146. FPauth-core
  147. FPauth-responses
  148. FPauth-strategies
  149. faraday != "0.2.0"
  150. farfadet
  151. fat-filesystem >= "0.12.0"
  152. ff
  153. ff-pbt
  154. fiat-p256
  155. flex-array
  156. fsevents-lwt
  157. functoria >= "2.2.0"
  158. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  159. geojson
  160. geoml >= "0.1.1"
  161. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0"
  162. git-mirage < "3.0.0"
  163. git-unix >= "1.10.0" & != "2.1.0"
  164. gitlab-unix
  165. glicko2
  166. gmap >= "0.3.0"
  167. gobba
  168. gpt
  169. graphql
  170. graphql-async
  171. graphql-cohttp >= "0.13.0"
  172. graphql-lwt
  173. graphql_parser != "0.11.0"
  174. graphql_ppx >= "0.7.1"
  175. h1_parser
  176. h2
  177. hacl
  178. hacl-star >= "0.6.0"
  179. hacl_func
  180. hacl_x25519 >= "0.2.0"
  181. highlexer
  182. hkdf
  183. hockmd
  184. html_of_jsx
  185. http
  186. http-multipart-formdata < "2.0.0"
  187. httpaf >= "0.2.0"
  188. hvsock
  189. icalendar >= "0.1.4"
  190. imagelib >= "20200929"
  191. index
  192. inferno >= "20220603"
  193. influxdb-async
  194. influxdb-lwt
  195. inquire < "0.2.0"
  196. interval-map
  197. iomux
  198. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & != "2.3.0"
  199. irmin-bench >= "2.7.0"
  200. irmin-chunk < "1.3.0" | >= "2.3.0"
  201. irmin-cli
  202. irmin-containers
  203. irmin-fs < "1.3.0" | >= "2.3.0"
  204. irmin-git < "2.0.0" | >= "2.3.0"
  205. irmin-http < "2.0.0"
  206. irmin-mem < "1.3.0"
  207. irmin-pack >= "2.4.0" & != "2.6.1"
  208. irmin-pack-tools
  209. irmin-test >= "2.2.0" & < "3.0.0"
  210. irmin-tezos
  211. irmin-tezos-utils
  212. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  213. irmin-watcher
  214. jekyll-format
  215. jerboa
  216. jitsu
  217. jose
  218. json-data-encoding >= "0.9"
  219. json_decoder
  220. jsonxt
  221. junit_alcotest
  222. jwto
  223. ke >= "0.2"
  224. kkmarkdown
  225. lambda-runtime
  226. lambda_streams
  227. lambda_streams_async
  228. lambdapi >= "2.0.0"
  229. lambdoc >= "1.0-beta4"
  230. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  231. letters
  232. lmdb >= "1.0"
  233. logical
  234. logtk >= "1.6"
  235. lp
  236. lp-glpk
  237. lp-glpk-js
  238. lp-gurobi
  239. lru
  240. lt-code
  241. luv
  242. mbr-format >= "1.0.0"
  243. mdx >= "1.6.0"
  244. mec
  245. mechaml >= "1.0.0"
  246. merge-queues >= "0.2.0"
  247. merge-ropes >= "0.2.0"
  248. metrics
  249. minicaml = "0.3.1" | >= "0.4"
  250. mirage >= "4.0.0~beta1"
  251. mirage-block-partition
  252. mirage-block-ramdisk >= "0.3"
  253. mirage-channel >= "4.0.0"
  254. mirage-channel-lwt
  255. mirage-crypto-ec
  256. mirage-flow >= "1.0.2" & < "1.2.0"
  257. mirage-flow-unix
  258. mirage-fs-mem
  259. mirage-fs-unix >= "1.2.0"
  260. mirage-kv >= "2.0.0"
  261. mirage-kv-mem
  262. mirage-kv-unix
  263. mirage-logs >= "0.3.0"
  264. mirage-nat
  265. mirage-net-unix >= "2.3.0"
  266. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  267. mirage-tc
  268. mjson
  269. mmdb
  270. mnd
  271. monocypher
  272. mrmime >= "0.2.0"
  273. mrt-format
  274. msgpck >= "1.6"
  275. mssql >= "2.0.3"
  276. multibase
  277. multihash
  278. multihash-digestif
  279. multipart-form-data
  280. multipart_form
  281. multipart_form-eio
  282. multipart_form-lwt
  283. named-pipe
  284. nanoid
  285. nbd >= "4.0.3"
  286. nbd-tool
  287. nloge
  288. nocoiner
  289. non_empty_list
  290. OCADml >= "0.6.0"
  291. ocaml-r >= "0.5.0"
  292. ocaml-version >= "3.1.0"
  293. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  294. ocamlformat-rpc < "removed"
  295. ocamline
  296. ocluster < "0.3.0"
  297. odoc >= "1.4.0" & < "2.1.0"
  298. ohex
  299. oidc
  300. opam-0install
  301. opam-file-format >= "2.1.1"
  302. opentelemetry >= "0.6"
  303. opentelemetry-client-cohttp-lwt >= "0.6"
  304. opentelemetry-client-ocurl >= "0.6"
  305. opentelemetry-cohttp-lwt >= "0.6"
  306. opentelemetry-lwt >= "0.6"
  307. opium >= "0.15.0"
  308. opium-graphql
  309. opium-testing
  310. opium_kernel
  311. orewa
  312. ortac-core
  313. osx-acl
  314. osx-attr
  315. osx-cf
  316. osx-fsevents
  317. osx-membership
  318. osx-mount
  319. osx-xattr
  320. otoggl
  321. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  322. owl-base < "0.5.0"
  323. owl-ode >= "0.1.0" & != "0.2.0"
  324. owl-symbolic
  325. passmaker
  326. patch
  327. pbkdf
  328. pecu >= "0.2"
  329. pf-qubes
  330. pg_query >= "0.9.6"
  331. pgx >= "1.0"
  332. pgx_unix >= "1.0"
  333. pgx_value_core
  334. pgx_value_ptime
  335. phylogenetics
  336. piaf
  337. polyglot
  338. polynomial
  339. ppx_blob >= "0.3.0"
  340. ppx_deriving_cmdliner
  341. ppx_deriving_rpc
  342. ppx_deriving_yaml
  343. ppx_graphql >= "0.2.0"
  344. ppx_inline_alcotest
  345. ppx_parser
  346. ppx_protocol_conv >= "5.0.0"
  347. ppx_protocol_conv_json >= "5.0.0"
  348. ppx_protocol_conv_jsonm >= "5.0.0"
  349. ppx_protocol_conv_msgpack >= "5.0.0"
  350. ppx_protocol_conv_xml_light >= "5.0.0"
  351. ppx_protocol_conv_xmlm
  352. ppx_protocol_conv_yaml >= "5.0.0"
  353. ppx_repr < "0.4.0"
  354. ppx_subliner
  355. ppx_units
  356. ppx_yojson >= "1.1.0"
  357. pratter
  358. prc
  359. preface
  360. pretty_expressive
  361. prettym
  362. proc-smaps
  363. producer < "0.2.0"
  364. progress < "0.2.0"
  365. prom
  366. prometheus < "1.2"
  367. prometheus-app
  368. protocell
  369. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  370. protocol-9p-unix
  371. psq
  372. qcheck >= "0.18"
  373. qcheck-alcotest
  374. qcheck-core >= "0.18"
  375. quickjs
  376. radis
  377. randii
  378. reason-standard
  379. reparse >= "2.0.0" & < "3.0.0"
  380. reparse-unix < "2.1.0"
  381. resp
  382. resp-unix >= "0.10.0"
  383. rfc1951 < "1.0.0"
  384. routes < "2.0.0"
  385. rpc >= "7.1.0"
  386. rpclib >= "7.1.0"
  387. rpclib-async
  388. rpclib-lwt >= "7.1.0"
  389. rubytt
  390. SZXX >= "4.0.0"
  391. salsa20
  392. salsa20-core
  393. sanddb >= "0.2"
  394. scaml >= "1.5.0"
  395. scrypt-kdf
  396. secp256k1 >= "0.4.1"
  397. secp256k1-internal
  398. semver >= "0.2.1"
  399. sendmail
  400. sendmail-lwt
  401. sendmsg
  402. server-reason-react
  403. session-cookie
  404. session-cookie-async
  405. session-cookie-lwt
  406. sherlodoc
  407. slug
  408. sodium-fmt
  409. spin >= "0.6.0"
  410. squirrel
  411. ssh-agent
  412. ssl >= "0.6.0"
  413. stramon-lib
  414. styled-ppx
  415. syslog-rfc5424
  416. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "8.0.0"
  417. tdigest < "2.1.0"
  418. terminal_size >= "0.1.1"
  419. terminus
  420. terminus-cohttp
  421. terminus-hlc
  422. terml
  423. textrazor
  424. tezos-base-test-helpers < "13.0"
  425. tezos-bls12-381-polynomial
  426. tezos-client-base < "12.0"
  427. tezos-crypto >= "8.0" & < "9.0"
  428. tezos-lmdb
  429. tezos-plompiler = "0.1.3"
  430. tezos-plonk = "0.1.3"
  431. tezos-signer-backends >= "8.0" & < "13.0"
  432. tezos-stdlib >= "8.0" & < "12.0"
  433. tezos-test-helpers < "12.0"
  434. tftp
  435. timedesc
  436. timere
  437. tls >= "0.12.0"
  438. toc
  439. topojson
  440. topojsone
  441. transept
  442. twostep
  443. type_eq
  444. type_id
  445. typebeat
  446. typeid >= "1.0.1"
  447. tyre >= "0.4"
  448. tyxml >= "4.0.0"
  449. tyxml-jsx
  450. tyxml-ppx >= "4.3.0"
  451. tyxml-syntax
  452. uecc
  453. ulid
  454. universal-portal
  455. unix-dirent
  456. unix-errno >= "0.3.0"
  457. unix-fcntl >= "0.3.0"
  458. unix-sys-resource
  459. unix-sys-stat
  460. unix-time
  461. unstrctrd
  462. user-agent-parser
  463. uspf
  464. uspf-lwt
  465. uspf-unix
  466. utop >= "2.13.0"
  467. validate
  468. validator
  469. vercel
  470. vpnkit
  471. wcwidth
  472. websocketaf
  473. x509 >= "0.7.0"
  474. xapi-rrd >= "1.8.2"
  475. xapi-stdext-date
  476. xapi-stdext-encodings
  477. xapi-stdext-std >= "4.16.0"
  478. yaml < "3.2.0"
  479. yaml-sexp
  480. yocaml
  481. yocaml_yaml
  482. yojson >= "1.6.0"
  483. yuscii >= "0.3.0"
  484. zar
  485. zed >= "3.2.2"
  486. zlist < "0.4.0"

Conflicts

None