package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

Authors

Maintainers

Sources

alcotest-1.0.1.tbz
sha256=0c8748838a89df6dee4850aa7ef5e46c573265a9bf1589dec255bd8156a793f6
sha512=f5f52dea5bb143e7001b8d0ac6131f8851389b080f46b9ad1ccacb95cc31a38143dd7122ccba59bb190abe559dbf81f33cc4dc3401ed95772d59be75fa566f19

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 13 Feb 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "1.11"

Dev Dependencies

None

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async != "1.0.0" & < "1.1.0"
  4. alcotest-lwt != "1.0.0" & < "1.1.0"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. ambient-context-lwt
  9. angstrom >= "0.7.0"
  10. ansi >= "0.6.0"
  11. anycache >= "0.7.4"
  12. anycache-async
  13. anycache-lwt
  14. archetype >= "1.4.2"
  15. archi
  16. arp
  17. arp-mirage
  18. arrakis
  19. art
  20. asak >= "0.2"
  21. asli >= "0.2.0"
  22. asn1-combinators >= "0.2.2"
  23. atd >= "2.3.3"
  24. atdgen >= "2.10.0"
  25. atdpy
  26. atdts
  27. base32
  28. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  29. bastet
  30. bastet_async
  31. bastet_lwt
  32. bechamel >= "0.5.0"
  33. bigarray-overlap
  34. bigstring >= "0.3"
  35. bigstring-unix >= "0.3"
  36. bigstringaf
  37. bitlib
  38. blake2
  39. bloomf
  40. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  41. bls12-381-hash
  42. bls12-381-js >= "0.4.2"
  43. bls12-381-js-gen >= "0.4.2"
  44. bls12-381-legacy
  45. bls12-381-signature
  46. bls12-381-unix
  47. blurhash
  48. builder-web
  49. bulletml
  50. bytebuffer
  51. ca-certs
  52. ca-certs-nss
  53. cactus
  54. caldav
  55. calendar >= "3.0.0"
  56. callipyge
  57. camlix
  58. capnp-rpc < "1.2.3"
  59. capnp-rpc-lwt < "0.3"
  60. capnp-rpc-mirage >= "0.9.0"
  61. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  62. carray
  63. carton
  64. cborl
  65. ccss >= "1.6"
  66. cf-lwt
  67. chacha
  68. channel
  69. charrua-client
  70. charrua-client-lwt
  71. charrua-client-mirage < "0.11.0"
  72. checkseum >= "0.0.3"
  73. cid
  74. clarity-lang
  75. class_group_vdf
  76. cohttp >= "0.17.0"
  77. cohttp-curl-async
  78. cohttp-curl-lwt
  79. cohttp-eio >= "6.0.0~beta2"
  80. colombe >= "0.2.0"
  81. color
  82. conan
  83. conan-cli
  84. conan-database
  85. conan-lwt
  86. conan-unix
  87. conduit = "3.0.0"
  88. conex < "0.10.0"
  89. conex-mirage-crypto
  90. conex-nocrypto
  91. cookie
  92. cow >= "2.2.0"
  93. css
  94. css-parser
  95. cstruct >= "3.3.0"
  96. cstruct-sexp
  97. ctypes-zarith
  98. cuid
  99. curly
  100. current_incr
  101. cwe_checker
  102. data-encoding
  103. datakit >= "0.12.0"
  104. datakit-bridge-github >= "0.12.0"
  105. datakit-ci
  106. datakit-client-git >= "0.12.0"
  107. decompress >= "0.8" & < "1.5.3"
  108. depyt
  109. digestif >= "0.8.1"
  110. dispatch >= "0.4.1"
  111. dkim
  112. dkim-bin
  113. dkim-mirage
  114. dns >= "4.0.0"
  115. dns-cli
  116. dns-client >= "4.6.0"
  117. dns-forward < "0.9.0"
  118. dns-forward-lwt-unix
  119. dns-resolver
  120. dns-server
  121. dns-tsig
  122. dnssd
  123. dnssec
  124. docfd >= "2.2.0"
  125. dog < "0.2.1"
  126. domain-name
  127. dream
  128. dream-pure
  129. duff
  130. dune-release >= "1.0.0"
  131. duration >= "0.1.1"
  132. emile
  133. encore
  134. eqaf >= "0.5"
  135. equinoxe
  136. equinoxe-cohttp
  137. equinoxe-hlc
  138. eris
  139. eris-lwt
  140. ezgzip
  141. ezjsonm >= "0.4.2" & < "1.3.0"
  142. ezjsonm-lwt < "1.3.0"
  143. FPauth
  144. FPauth-core
  145. FPauth-responses
  146. FPauth-strategies
  147. faraday != "0.2.0"
  148. farfadet
  149. fat-filesystem >= "0.12.0"
  150. ff
  151. ff-pbt
  152. fiat-p256
  153. flex-array
  154. fsevents-lwt
  155. functoria >= "2.2.0"
  156. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  157. geojson
  158. geoml >= "0.1.1"
  159. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0" & < "3.0.0"
  160. git-mirage < "3.0.0"
  161. git-unix >= "1.10.0" & != "2.1.0" & < "3.0.0"
  162. gitlab-unix
  163. glicko2
  164. gmap >= "0.3.0"
  165. gobba
  166. gpt
  167. graphql
  168. graphql-async
  169. graphql-cohttp >= "0.13.0"
  170. graphql-lwt
  171. graphql_parser != "0.11.0"
  172. graphql_ppx >= "0.7.1"
  173. h1_parser
  174. h2
  175. hacl
  176. hacl_func
  177. hacl_x25519 >= "0.2.0"
  178. highlexer
  179. hkdf
  180. hockmd
  181. html_of_jsx
  182. http
  183. http-multipart-formdata < "2.0.0"
  184. httpaf >= "0.2.0"
  185. hvsock
  186. icalendar >= "0.1.4"
  187. imagelib >= "20200929"
  188. index
  189. inferno >= "20220603"
  190. influxdb-async
  191. influxdb-lwt
  192. inquire < "0.2.0"
  193. interval-map
  194. iomux
  195. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & < "2.3.0"
  196. irmin-bench >= "2.7.0"
  197. irmin-chunk < "1.3.0" | >= "2.3.0"
  198. irmin-cli
  199. irmin-containers
  200. irmin-fs < "1.3.0" | >= "2.3.0"
  201. irmin-git < "2.0.0" | >= "2.3.0"
  202. irmin-http < "2.0.0"
  203. irmin-mem < "1.3.0"
  204. irmin-pack >= "2.4.0" & != "2.6.1"
  205. irmin-pack-tools
  206. irmin-test >= "2.2.0" & < "3.0.0"
  207. irmin-tezos
  208. irmin-tezos-utils
  209. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  210. irmin-watcher
  211. jekyll-format
  212. jerboa
  213. jitsu
  214. jose
  215. json-data-encoding >= "0.9"
  216. json_decoder
  217. jsonxt
  218. junit_alcotest
  219. jwto
  220. ke >= "0.2"
  221. kkmarkdown
  222. lambda-runtime
  223. lambda_streams
  224. lambda_streams_async
  225. lambda_streams_lwt
  226. lambdapi >= "2.0.0"
  227. lambdoc >= "1.0-beta4"
  228. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  229. lmdb >= "1.0"
  230. logical
  231. logtk >= "1.6"
  232. lp
  233. lp-glpk
  234. lp-glpk-js
  235. lp-gurobi
  236. lru
  237. lt-code
  238. luv
  239. mbr-format >= "1.0.0"
  240. mdx >= "1.6.0"
  241. mec
  242. mechaml >= "1.0.0"
  243. merge-queues >= "0.2.0"
  244. merge-ropes >= "0.2.0"
  245. metrics
  246. minicaml = "0.3.1" | >= "0.4"
  247. mirage >= "4.0.0~beta1"
  248. mirage-block-partition
  249. mirage-block-ramdisk >= "0.3"
  250. mirage-channel >= "4.0.0"
  251. mirage-channel-lwt
  252. mirage-crypto-ec
  253. mirage-flow >= "1.0.2" & < "1.2.0"
  254. mirage-flow-unix
  255. mirage-fs-mem
  256. mirage-fs-unix >= "1.2.0"
  257. mirage-kv >= "2.0.0"
  258. mirage-kv-mem
  259. mirage-kv-unix
  260. mirage-logs >= "0.3.0"
  261. mirage-nat
  262. mirage-net-unix >= "2.3.0"
  263. mirage-runtime >= "4.0.0~beta1"
  264. mirage-tc
  265. mjson
  266. mmdb
  267. mnd
  268. monocypher
  269. mrmime >= "0.2.0"
  270. mrt-format
  271. msgpck >= "1.6"
  272. mssql >= "2.0.3"
  273. multibase
  274. multihash
  275. multihash-digestif
  276. multipart-form-data
  277. multipart_form
  278. multipart_form-lwt
  279. named-pipe
  280. nanoid
  281. nbd >= "4.0.3"
  282. nbd-tool
  283. nloge
  284. nocoiner
  285. non_empty_list
  286. OCADml >= "0.6.0"
  287. ocaml-r >= "0.5.0"
  288. ocaml-version >= "3.1.0"
  289. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  290. ocamlformat-rpc < "removed"
  291. ocamline
  292. ocluster < "0.3.0"
  293. odoc >= "1.4.0" & < "2.1.0"
  294. ohex
  295. oidc
  296. opam-0install
  297. opam-file-format >= "2.1.1"
  298. opentelemetry >= "0.6"
  299. opentelemetry-client-cohttp-lwt >= "0.6"
  300. opentelemetry-client-ocurl >= "0.6"
  301. opentelemetry-cohttp-lwt >= "0.6"
  302. opentelemetry-lwt >= "0.6"
  303. opium >= "0.15.0"
  304. opium-graphql
  305. opium-testing
  306. opium_kernel
  307. orewa
  308. ortac-core
  309. osx-acl
  310. osx-attr
  311. osx-cf
  312. osx-fsevents
  313. osx-membership
  314. osx-mount
  315. osx-xattr
  316. otoggl
  317. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  318. owl-base < "0.5.0"
  319. owl-ode >= "0.1.0" & != "0.2.0"
  320. owl-symbolic
  321. passmaker
  322. patch
  323. pbkdf
  324. pecu >= "0.2"
  325. pf-qubes
  326. pg_query >= "0.9.6"
  327. pgx >= "1.0"
  328. pgx_unix >= "1.0"
  329. pgx_value_core
  330. pgx_value_ptime
  331. phylogenetics
  332. piaf
  333. polyglot
  334. polynomial
  335. ppx_blob >= "0.3.0"
  336. ppx_deriving_cmdliner
  337. ppx_deriving_rpc
  338. ppx_deriving_yaml
  339. ppx_graphql >= "0.2.0"
  340. ppx_inline_alcotest
  341. ppx_parser
  342. ppx_protocol_conv >= "5.0.0"
  343. ppx_protocol_conv_json >= "5.0.0"
  344. ppx_protocol_conv_jsonm >= "5.0.0"
  345. ppx_protocol_conv_msgpack >= "5.0.0"
  346. ppx_protocol_conv_xml_light >= "5.0.0"
  347. ppx_protocol_conv_xmlm
  348. ppx_protocol_conv_yaml >= "5.0.0"
  349. ppx_subliner
  350. ppx_units
  351. ppx_yojson >= "1.1.0"
  352. pratter
  353. prc
  354. preface
  355. pretty_expressive
  356. prettym
  357. proc-smaps
  358. producer < "0.2.0"
  359. prom
  360. prometheus < "1.2"
  361. prometheus-app
  362. protocell
  363. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  364. protocol-9p-unix
  365. psq
  366. qcheck >= "0.18"
  367. qcheck-alcotest
  368. qcheck-core >= "0.18"
  369. quickjs
  370. radis
  371. randii
  372. reason-standard
  373. reparse >= "2.0.0" & < "3.0.0"
  374. reparse-unix < "2.1.0"
  375. resp
  376. resp-unix >= "0.10.0"
  377. rfc1951 < "1.0.0"
  378. routes < "2.0.0"
  379. rpc >= "7.1.0"
  380. rpclib >= "7.1.0"
  381. rpclib-async
  382. rpclib-lwt >= "7.1.0"
  383. rubytt
  384. SZXX >= "4.0.0"
  385. salsa20
  386. salsa20-core
  387. sanddb >= "0.2"
  388. scaml >= "1.5.0"
  389. scrypt-kdf
  390. secp256k1 >= "0.4.1"
  391. secp256k1-internal
  392. semver >= "0.2.1"
  393. sendmail
  394. sendmail-lwt
  395. sendmsg
  396. server-reason-react
  397. session-cookie
  398. session-cookie-async
  399. session-cookie-lwt
  400. sherlodoc
  401. slug
  402. sodium-fmt
  403. spin >= "0.6.0"
  404. squirrel
  405. ssh-agent
  406. ssl >= "0.6.0"
  407. stramon-lib
  408. syslog-rfc5424
  409. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "8.0.0"
  410. tdigest < "2.1.0"
  411. terminal_size >= "0.1.1"
  412. terminus
  413. terminus-cohttp
  414. terminus-hlc
  415. textrazor
  416. tezos-base-test-helpers < "13.0"
  417. tezos-client-base < "12.0"
  418. tezos-lmdb
  419. tezos-test-helpers < "11.0"
  420. tftp
  421. timedesc
  422. timere
  423. tls >= "0.12.0"
  424. toc
  425. topojson
  426. topojsone
  427. transept
  428. twostep
  429. type_eq
  430. type_id
  431. typebeat
  432. typeid >= "1.0.1"
  433. tyre >= "0.4"
  434. tyxml >= "4.0.0"
  435. tyxml-jsx
  436. tyxml-ppx >= "4.3.0"
  437. tyxml-syntax
  438. uecc
  439. ulid
  440. unix-dirent
  441. unix-errno >= "0.3.0"
  442. unix-fcntl >= "0.3.0"
  443. unix-sys-resource
  444. unix-sys-stat
  445. unix-time
  446. unstrctrd
  447. user-agent-parser
  448. uspf
  449. uspf-lwt
  450. uspf-unix
  451. utop >= "2.13.0"
  452. validate
  453. validator
  454. vercel
  455. vpnkit
  456. websocketaf
  457. x509 >= "0.7.0"
  458. xapi-rrd >= "1.8.2"
  459. xapi-stdext-date
  460. xapi-stdext-encodings
  461. xapi-stdext-std >= "4.16.0"
  462. yaml < "3.2.0"
  463. yaml-sexp
  464. yocaml
  465. yocaml_yaml
  466. yojson >= "1.6.0"
  467. yuscii >= "0.3.0"
  468. zar
  469. zed >= "3.2.2"
  470. zlist < "0.4.0"

Conflicts

None