From ac0a685769f6c37d6469c714e3caa60a1da537aa Mon Sep 17 00:00:00 2001 From: Benjamin Canou Date: Sun, 18 Feb 2018 19:28:26 +0100 Subject: [PATCH] Docs: add doc on OPAM packages/source code architecture --- docs/introduction/contributing.rst | 2 +- docs/tutorials/entering_alpha.rst | 2 + docs/whitedoc/packages.svg | 3030 ++++++++++++++++++++++++++++ docs/whitedoc/the_big_picture.rst | 268 ++- 4 files changed, 3300 insertions(+), 2 deletions(-) create mode 100644 docs/whitedoc/packages.svg diff --git a/docs/introduction/contributing.rst b/docs/introduction/contributing.rst index e469f7fb0..0efbfc941 100644 --- a/docs/introduction/contributing.rst +++ b/docs/introduction/contributing.rst @@ -35,7 +35,7 @@ Where to start While you familiarize yourself with the basics as suggested above, you can have a look at the :ref:`software architecture -` of Tezos. It will +` of Tezos. It will give you the main components and their interactions, and links to the documentations for the various parts. diff --git a/docs/tutorials/entering_alpha.rst b/docs/tutorials/entering_alpha.rst index ebeca006e..7fe168ac9 100644 --- a/docs/tutorials/entering_alpha.rst +++ b/docs/tutorials/entering_alpha.rst @@ -1,3 +1,5 @@ +.. _entering_alpha: + How to start reading protocol Alpha =================================== diff --git a/docs/whitedoc/packages.svg b/docs/whitedoc/packages.svg new file mode 100644 index 000000000..bed597efd --- /dev/null +++ b/docs/whitedoc/packages.svg @@ -0,0 +1,3030 @@ + + + +image/svg+xml \ No newline at end of file diff --git a/docs/whitedoc/the_big_picture.rst b/docs/whitedoc/the_big_picture.rst index 58e47535e..8df173c59 100644 --- a/docs/whitedoc/the_big_picture.rst +++ b/docs/whitedoc/the_big_picture.rst @@ -1,8 +1,19 @@ -.. _the_big_picture: +.. _software_architecture: Tezos Software Architecture =========================== +This document contains two section. The first section, which should be +readable by anyone, describes the main elements of Tezos from a +distance. It abstracts from all plumbing and both internal and system +dependencies to give a simple view of the main components, their +responsibilities and interactions. The second part is written for +developpers, and is at the level of OPAM packages. + +The Big Picture +--------------- +.. _the_big_picture: + The diagram below shows a very coarse grained architecture of Tezos. |Tezos architecture diagram| @@ -58,3 +69,258 @@ is fully inter-operable, and auto descriptive, using JSON schema. .. |Tezos architecture diagram| image:: octopus.svg + +Software Architecture and Package Relashionships +------------------------------------------------ +.. _packages: + +The diagram below shows the main OPAM packages present in the source +code of Tezos, and their dependencies. The ``tezos-`` prefix has been +dropped for clarity. + +|Tezos source packages diagram| + +In green at the bottom are binaries. Hilighted in yellow are the OPAM +packages (sometimes with shortened names). Black arrows show direct +dependencies. Orange arrows show other indirect relashionships (code +generation, interface sharing), explained below. The part circled in +blue, contains modules that bear no dependency to Unix, and can thus +be compiled to JavaScript. External dependencies are not shown in this +illustration. + +Base and below +~~~~~~~~~~~~~~ + +At the center, the :package:`tezos-base` package is where +the blockchain specific code starts. Before it are the set of libraries +that are used everywhere for basic operations. + + - :package:`tezos-stdlib` contains a few extensions over the + OCaml standard library (a few string primitives, an ``Option`` + module, etc.), a few ``Lwt`` utilities, and a ``Compare`` module + that implements monomorphic comparison operators. + - :package:`tezos-data-encoding` is the in-house + comibnator-based serialization library. From a single type + description ``t encoding``, the code can read to and write from + values of type ``t`` both binary and JSON representations. For + both, the library provides machine and human-redable documentations + by the use of documentation combinators. The JSON part depends on + :opam:`ocplib-json-typed`. + A :ref:`tutorial` is available for this library. + - :package:`tezos-error-monad` is an in-house monadic + interface to the OCaml ``('a, 'b) result`` type, that fixes the + ``'b`` to an extensible type ``error`` (actually a list, to hold an + error trace). When extending the type, programmers must also call + the ``register_error`` function that registers a pretty printer and + an encoding for serialization. + A :ref:`tutorial` is available for this library. + - :package:`tezos-rpc` provides the basics of Tezos' RPC service + mechanism. It provides combinators for building service hierarchies + à la Ocsigen/Eliom, registering and calling services. This module + is based on :opam:`ocplib-resto`, that allows for automatic + generation of a machine and human-readable of the hierarchy of + services: the structure of URLs and the expected formats for input + and output bodies, via the use of ``data_encoding``. + - :package:`tezos-crypto` wraps the external cryptography + libraries that we use. We try to use minimal references + implementations, with as thin as possible bindings. A possible plan + is to use libraries from the HACL projet, so that all of our crypto + is extracted from Fstar, either with thin C bindings or directly in + OCaml. + - :package:`tezos-micheline` is the concrete syntax used by + Michelson, the language of smart contracts. It mostly contains the + generic, untyped AST, a printer and a parser. + - :package:`tezos-base` wraps all these module in a common foundation + for all the other components of Tezos, and introduces the data + structures of the blockchain (e.g. ``Block_hash``, + ``Block_header``, ``Block_locator``, ``Fitness``, ``P2p_identity``) + that are shared between the shell, economic protocol, client, + daemons and third party software. It also rewraps some modules from + ``crypto`` as functors that build all-in-one modules for a given + type (for instance, the module for block hashes contains all + possible converters, command line and RPC argument builders, pretty + printers, an autocompleter, etc.). This package also contains the + ``cli_entries`` module that we use to handle command line parsing + in some executables. + +The Shell +~~~~~~~~~ + +The shell is the part of the node responsible for all communications, +peer-to-peer and RPC, acting as a cocoon around the economic +protocols. + + - :package:`tezos-shell-services` contains the definition of the + node's service hierarchy, and calling functions to use in the + client (or any third party software). As this library is linked + into the client to call the services in a type-safe way, only the + description of services is done here. The registration of handlers + is done in the rest of the node's implementation. + - :package:`tezos-rpc-http` uses :opam:`cohttp` to implement the RPC + over HTTP server and client, allowing to make actual use of + services declared using :package:`tezos-rpc`. + - :package:`tezos-p2p` is the in-house peer-to-peer layer. + - :package:`tezos-storage` contains the raw simple key-value store + used for the chain data, and the raw versioned key-value store + used for storing the ledger's context (one version per + block). This is implemented using :opam:`irmin` and currently + :opam:`leveldb`. + - :package:`tezos-protocol-updater` maintains the table of available + protocol versions, embedded or dynamically linked. + - :package:`tezos-shell` implements the scheduling of block + validations, the mempool management, and the distributed database. + A description is available in :ref:`this document`. + +The Economic Protocol Environment and Compiler +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Economic protocols are compiled by a specific version of the OCaml +compiler. This compiler restricts the set of modules available to the +economic protocol, as a form of static sandboxing. It also generates a +functorized version of the protocol, to make the execution of the +protocol in alternative environment possible. + + - :package:`tezos-protocol-environment-sigs` contains the modules + that are available to the economic protocol. A review of this + sandbox is available :ref:`here`. This + modules include a stripped down standard library, and interfaces + to the crypto APIs, RPC definitions, and key-value store. + + - :package:`tezos-protocol-compiler` is the compiler for economic + protocols: an alternative driver to the OCaml + :opam:`ocaml-compiler-libs` that typechecks within the protocol + environment, and performs some more checks on the protocol code. + + - ``tezos-protocol-xxx`` is produced by the protocol compiler + and contains a functorized version of protocol ``xxx`` that takes its + standard library as parameter. This parameter can be filled with + any of the implementations described in the two points below. + + - :package:`tezos-protocol-environment-shell` is the instance of the + environment whose RPC service registration and storage access are + the ones of the node. This is the environment that is fed by the + node when loading new protocols. + + - :package:`tezos-protocol-environment` contains three alternative + instances of the protocol environment: one whose context access + function are dummy ones which can be used when only the types and + non contextual functions of the protocol are needed, another that + simulates the key-value store in memory for testing, and a functor + that let you build an environment from a few context accessors. + + - ``tezos-embedded-protocol-xxx`` contains a version of protocol + ``xxx`` whose standard library is pre-instanciated to the shell's + implementation, these are the ones that are linked into the + node. It alse contains a module that registers the protocol in the + node's protocol table. + +The Embedded Economic Protocols +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Three economic protocols are included in the main Tezos repository. + + - :package:`tezos-protocol-genesis` + (:package:`tezos-embedded-protocol-genesis`) is the protocol of + the genesis block. It accepts a single block, signed by a dictator + whose public key is hardcoded, that single action is to switch to + a new protocol chosen by the dictator. + - :package:`tezos-protocol-alpha` + (:package:`tezos-embedded-protocol-alpha`) is the first real + protocol of Tezos. A :ref:`tutorial` is available + to start reading the protocol's code. + - :package:`tezos-protocol-demo` + (:package:`tezos-embedded-protocol-demo`) is just a demo protocol + that does nothing interesting but has the right shape. + +The Client Library +~~~~~~~~~~~~~~~~~~ + +The client is split into many packages, to enforce three separation +lines: shell vs economic protocol, Unix dependent vs Javascript +compatible, and library vs command line interface. + + - :package:`tezos-client-base` define the client context, which is + an object whose methods allow for: accessing a wallet of keys, + interacting via the user, making RPC calls, and signing data using + signer plug-ins. Most of the, including RPC calling functions from + :package:`tezos-shell-services` and + :package:`tezos-protocol-alpha`, are abstracted over this object + type. That way, it is possible to use the same code for different + platforms ot toolkits. + - :package:`tezos-client-alpha` provides some functions to perform + the operations of protocol alpha using the wallet and signers from + the client context. + - :package:`tezos-client-commands` plugs the basic context access + functions from :package:`tezos-client-base` as handlers for the + commands of the ``tezos-client`` command line wallet. + - :package:`tezos-client-alpha-commands` plugs the functions from + :package:`tezos-client-alpha` as handlers for the alpha specific + commands of the ``tezos-client`` command line wallet. + - :package:`tezos-client-genesis` contains the basic dictator + commands available on the genesis protocol. + - :package:`tezos-client-base-unix` implements configuration file + and wallet storage in Unix files, user interaction via the Unix + console, and terminal based signer plug-ins. + +Tests Packages +~~~~~~~~~~~~~~ + +The tests are splitted into various packages, testing more and more +elements while following the dependency chain. Use ``make test`` to +run them. + + - :package-name:`tezos-client` + (in directory :src:`src/bin_client/test/`): + end-to-end tests as shell scripts that launch a local sandboxed node + and performs various tasks using the client + - :package-name:`tezos-p2p` + (in directory :src:`src/lib_p2p/test/`): + tests of the peer-to-peer layer, independently of the Tezos gossip + protocol (establishing connections, propagating peers, etc.) + - :package-name:`tezos-protocol-environment` + (in directory :src:`src/lib_protocol_environment/test/`): + tests for the in-memory context implementation. + - :package-name:`tezos-shell` + (in directory :src:`src/lib_shell/test/`): + tests for the chain data storage. + - :package-name:`tezos-stdlib` + (in directory :src:`src/lib_stdlib/test/`): + tests for the basic data structures. + - :package-name:`tezos-storage` + (in directory :src:`src/lib_storage/test/`): + tests for the versioned key-value context. + - :package-name:`tezos-client-alpha`, :package-name:`tezos-baking-alpha` + (in directory :src:`src/proto_alpha/lib_baking/test/`): + tests of the OCaml client library, performing various short + scenarios via RPCs to a sandboxed node. + - :package-name:`tezos-protocol-alpha` + (in directory :src:`src/proto_alpha/lib_protocol/test/`): + tests of the alpha protocol (without launching a node). + - :package-name:`tezos-crypto` + (in directory :src:`src/lib_crypto/test/`): + tests for the in-house merkle trees. + - :package-name:`tezos-data-encoding` + (in directory :src:`src/lib_data_encoding/test/`): + tests for the JSON and binary serialization and deserialization. + +The Final Executables +~~~~~~~~~~~~~~~~~~~~~ + + - :package:`tezos-node` provides the node launcher binary + ``tezos-node``. All the algorithmic being implemented in the + shell, this package only implements the node's CLI. It also + provides the sandboxed node shell script launcher (see the main + readme). + - :package:`tezos-client` provides the ``tezos-client`` and + ``tezos-admin-client`` binaries. The former contains a small + comand line wallet, the latter an administration tool for the + node. It also provides a shell script that configures a shell + environment to interact with a sandboxed node. + - :package:`tezos-baker-alpha` provides the ``tezos-alpha-baker`` + binary. + - :package:`tezos-protocol-compiler` provides the + ``tezos-protocol-compiler`` binary that is used by the node to + compile new protocols on the fly, and that can be used for + developping new protocols. + +.. |Tezos source packages diagram| image:: packages.svg