Testing LIGO

Adding to the LIGO test suite is one of the more accessible ways to contribute. It exposes you to the compiler structure and primitives without necessarily demanding a deep understanding of OCaml or compiler development. And you'll probably become more familiar with LIGO itself in the process, which is helpful.

Unfortunately right now LIGO itself doesn't have a good way to do automated testing. So the tests are written in OCaml, outside of the LIGO language. Thankfully the test code is typically less demanding than the features being tested. These tests are currently contained in src/test, but the bulk are integration tests which rely on test contracts kept in src/test/contracts. If you're new to LIGO, reading these contracts can be a useful introduction to a given syntax. In the future we plan to have detailed documentation for each syntax, but at the moment we only have a reference manual for PascaLIGO

How To Find Good Test Cases

Your first question is probably "If I'm not already experienced, how do I know what to test?". There's a handful of things you can do to systematically find good test cases. All of them will either get you more familiar with the LIGO code base or LIGO itself.

Extending Existing Test Cases

The fastest way to improve LIGO's test coverage is to extend existing test cases. This means considering the test cases that already exist, and thinking of things they don't cover or situations they'll fail on. A good deal of inference is required for this, but it requires minimal experience with the existing code.

Studying The Parsers For Gaps In Coverage

LIGO is divided into a front end which handles syntax and a backend which optimizes and compiles a core language shared between syntaxes. You can find basic test cases for a particular LIGO syntax by studying its parser. You will find these under src/passes/1-parser. One kind of useful test focuses on coverage, whether we have any testing at all for a particular aspect of a syntax. You can find these by carefully going over the syntax tree for a syntax (probably best read by looking at its Parser.mly) and comparing each branch to the test suite. While these tests are plentiful at the time of writing, they will eventually be filled in reliably as part of writing a new syntax.

Creating Interesting Test Cases By Using LIGO

Another kind of useful test focuses on depth, whether the features are put through a wide variety of complex scenarios to make sure they stand up to real world use. One of the best ways to write these is to use LIGO for a real project. This will require some time and energy, not just to learn LIGO but to write projects complex enough to stretch the limits of what the language can do. At the same time however it will get you used to engaging with LIGO from a developers perspective, asking how things could be better or what features are underdeveloped. If your project has practical uses, you will also be contributing to the Tezos/LIGO ecosystem while you learn. Note that because LIGO is open source, in under for us to incorporate your work as a test case it needs to be licensed in a way that's compatible with LIGO.

Fuzzing (Speculative)

In the future you'll be able to use fuzzing to generate test cases for LIGO. Fuzzing is often useful for finding 'weird' bugs on code paths that humans normally wouldn't stumble into. This makes it a useful supplement to human testing.

Structure of LIGO Tests

LIGO's OCaml-based tests are written in alcotest. However the tests you encounter in src/test/integration_tests.ml are built on top of some abstractions, currently defined in src/test/test_helpers.ml. The use of these can be inferred fairly well from looking at existing tests, but lets break a few of them down for analysis. We'll first analyze a short integration test for assignment:

Assignment Test

let assign () : unit result =
  let%bind program = type_file "./contracts/assign.ligo" in
  let make_expect = fun n -> n + 1 in
  expect_eq_n_int program "main" make_expect

assign.ligo

function main (const i : int) : int is begin i := i + 1 ; end with i

So what's going on here? We have a function which takes no arguments and returns a unit result. We then define two variables, a program which is read from disk and fed to the LIGO compiler; and a comparison function make_expect which takes an integer and adds one to it. Using expect_eq_n_int the program's main function is run and compared to the result of providing the same input to make_expect. This gives us some flavor of what to expect from these integration tests. Notice that the main argument given to expect_eq_n_int corresponds to the name of the function in assign.ligo. We can see in more complex tests that we're able to pull the values of arbitrary expressions or function calls from LIGO test contracts. Consider:

Annotation Test

let annotation () : unit result = let%bind program = type_file "./contracts/annotation.ligo" in let%bind () = expect_eq_evaluate program "lst" (e_list []) in let%bind () = expect_eq_evaluate program "address" (e_address "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx") in let%bind () = expect_eq_evaluate program "address_2" (e_address "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx") in ok ()

annotation.ligo

const lst : list(int) = list [] ;

const address : address = "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx" ;

const address_2 : address = ("tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx" : address) ;

Here what's going on is similar to the last program; expect_eq_evaluate runs a program and then pulls a particular named value from the final program state. For example, once the program stops running the value of address is "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx". The comparison however is made to a constructed expression. Remember that we're testing from OCaml, but the program is written and evaluated as LIGO. In order to provide a proper comparison, we convert our expected test values into LIGO expressions and data. Constructors such as e_list and e_address provide a bridge between LIGO and OCaml. Their definitions can be found in files such as src/stages/ast_simplified/combinators.ml, or using Merlin's definition point finder. These same functions are used during the simplification stage of LIGO compilation, so becoming familiar with them will help prepare you to work on the front end.

How To Write A Test For LIGO

What if we want to write a test of our own? If the test is in the integration test vein (which it probably is if you're testing new syntax or features), then the process looks something like:

Write a test contract which uses the new syntax or feature in src/test/contracts.
Write an integration test in src/test/integration_tests.ml in the vein of existing tests, make sure you add it to the test runner that is currently located at the bottom of the file.
Write the feature, assuming it doesn't already exist. Build the resulting version of LIGO without errors.
Run the test suite, see if your test(s) pass. If they do, you're probably done. If not it's time to go debugging.

7.9 KiB Raw Blame History