Docs: add doc on OPAM packages/source code architecture

docs/introduction/contributing.rst

@@ -35,7 +35,7 @@ Where to start
 
 While you familiarize yourself with the basics as suggested above, you
 can have a look at the :ref:`software architecture
-<the_big_picture>` of Tezos. It will
+<software_architecture>` of Tezos. It will
 give you the main components and their interactions, and links to the
 documentations for the various parts.
 

docs/tutorials/entering_alpha.rst

@@ -1,3 +1,5 @@
+.. _entering_alpha:
+
 How to start reading protocol Alpha
 ===================================
 

docs/whitedoc/the_big_picture.rst

@@ -1,8 +1,19 @@
-.. _the_big_picture:
+.. _software_architecture:
 
 Tezos Software Architecture
 ===========================
 
+This document contains two section. The first section, which should be
+readable by anyone, describes the main elements of Tezos from a
+distance. It abstracts from all plumbing and both internal and system
+dependencies to give a simple view of the main components, their
+responsibilities and interactions. The second part is written for
+developpers, and is at the level of OPAM packages.
+
+The Big Picture
+---------------
+.. _the_big_picture:
+
 The diagram below shows a very coarse grained architecture of Tezos.
 
 |Tezos architecture diagram|
@@ -58,3 +69,258 @@ is fully inter-operable, and auto descriptive, using JSON schema.
 
 .. |Tezos architecture diagram| image:: octopus.svg
 
+
+Software Architecture and Package Relashionships
+------------------------------------------------
+.. _packages:
+
+The diagram below shows the main OPAM packages present in the source
+code of Tezos, and their dependencies. The ``tezos-`` prefix has been
+dropped for clarity.
+
+|Tezos source packages diagram|
+
+In green at the bottom are binaries. Hilighted in yellow are the OPAM
+packages (sometimes with shortened names). Black arrows show direct
+dependencies. Orange arrows show other indirect relashionships (code
+generation, interface sharing), explained below. The part circled in
+blue, contains modules that bear no dependency to Unix, and can thus
+be compiled to JavaScript. External dependencies are not shown in this
+illustration.
+
+Base and below
+~~~~~~~~~~~~~~
+
+At the center, the :package:`tezos-base` package is where
+the blockchain specific code starts. Before it are the set of libraries
+that are used everywhere for basic operations.
+
+ - :package:`tezos-stdlib` contains a few extensions over the
+   OCaml standard library (a few string primitives, an ``Option``
+   module, etc.), a few ``Lwt`` utilities, and a ``Compare`` module
+   that implements monomorphic comparison operators.
+ - :package:`tezos-data-encoding` is the in-house
+   comibnator-based serialization library. From a single type
+   description ``t encoding``, the code can read to and write from
+   values of type ``t`` both binary and JSON representations. For
+   both, the library provides machine and human-redable documentations
+   by the use of documentation combinators. The JSON part depends on
+   :opam:`ocplib-json-typed`.
+   A :ref:`tutorial<data_encoding>` is available for this library.
+ - :package:`tezos-error-monad` is an in-house monadic
+   interface to the OCaml ``('a, 'b) result`` type, that fixes the
+   ``'b`` to an extensible type ``error`` (actually a list, to hold an
+   error trace). When extending the type, programmers must also call
+   the ``register_error`` function that registers a pretty printer and
+   an encoding for serialization.
+   A :ref:`tutorial<error_monad>` is available for this library.
+ - :package:`tezos-rpc` provides the basics of Tezos' RPC service
+   mechanism. It provides combinators for building service hierarchies
+   à la Ocsigen/Eliom, registering and calling services. This module
+   is based on :opam:`ocplib-resto`, that allows for automatic
+   generation of a machine and human-readable of the hierarchy of
+   services: the structure of URLs and the expected formats for input
+   and output bodies, via the use of ``data_encoding``.
+ - :package:`tezos-crypto` wraps the external cryptography
+   libraries that we use. We try to use minimal references
+   implementations, with as thin as possible bindings. A possible plan
+   is to use libraries from the HACL projet, so that all of our crypto
+   is extracted from Fstar, either with thin C bindings or directly in
+   OCaml.
+ - :package:`tezos-micheline` is the concrete syntax used by
+   Michelson, the language of smart contracts. It mostly contains the
+   generic, untyped AST, a printer and a parser.
+ - :package:`tezos-base` wraps all these module in a common foundation
+   for all the other components of Tezos, and introduces the data
+   structures of the blockchain (e.g. ``Block_hash``,
+   ``Block_header``, ``Block_locator``, ``Fitness``, ``P2p_identity``)
+   that are shared between the shell, economic protocol, client,
+   daemons and third party software. It also rewraps some modules from
+   ``crypto`` as functors that build all-in-one modules for a given
+   type (for instance, the module for block hashes contains all
+   possible converters, command line and RPC argument builders, pretty
+   printers, an autocompleter, etc.). This package also contains the
+   ``cli_entries`` module that we use to handle command line parsing
+   in some executables.
+
+The Shell
+~~~~~~~~~
+
+The shell is the part of the node responsible for all communications,
+peer-to-peer and RPC, acting as a cocoon around the economic
+protocols.
+
+  - :package:`tezos-shell-services` contains the definition of the
+    node's service hierarchy, and calling functions to use in the
+    client (or any third party software). As this library is linked
+    into the client to call the services in a type-safe way, only the
+    description of services is done here. The registration of handlers
+    is done in the rest of the node's implementation.
+  - :package:`tezos-rpc-http` uses :opam:`cohttp` to implement the RPC
+    over HTTP server and client, allowing to make actual use of
+    services declared using :package:`tezos-rpc`.
+  - :package:`tezos-p2p` is the in-house peer-to-peer layer.
+  - :package:`tezos-storage` contains the raw simple key-value store
+    used for the chain data, and the raw versioned key-value store
+    used for storing the ledger's context (one version per
+    block). This is implemented using :opam:`irmin` and currently
+    :opam:`leveldb`.
+  - :package:`tezos-protocol-updater` maintains the table of available
+    protocol versions, embedded or dynamically linked.
+  - :package:`tezos-shell` implements the scheduling of block
+    validations, the mempool management, and the distributed database.
+    A description is available in :ref:`this document<validation>`.
+
+The Economic Protocol Environment and Compiler
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Economic protocols are compiled by a specific version of the OCaml
+compiler. This compiler restricts the set of modules available to the
+economic protocol, as a form of static sandboxing. It also generates a
+functorized version of the protocol, to make the execution of the
+protocol in alternative environment possible.
+
+  - :package:`tezos-protocol-environment-sigs` contains the modules
+    that are available to the economic protocol. A review of this
+    sandbox is available :ref:`here<protocol_environment>`. This
+    modules include a stripped down standard library, and interfaces
+    to the crypto APIs, RPC definitions, and key-value store.
+
+  - :package:`tezos-protocol-compiler` is the compiler for economic
+    protocols: an alternative driver to the OCaml
+    :opam:`ocaml-compiler-libs` that typechecks within the protocol
+    environment, and performs some more checks on the protocol code.
+
+  - ``tezos-protocol-xxx`` is produced by the protocol compiler
+    and contains a functorized version of protocol ``xxx`` that takes its
+    standard library as parameter. This parameter can be filled with
+    any of the implementations described in the two points below.
+
+  - :package:`tezos-protocol-environment-shell` is the instance of the
+    environment whose RPC service registration and storage access are
+    the ones of the node. This is the environment that is fed by the
+    node when loading new protocols.
+
+  - :package:`tezos-protocol-environment` contains three alternative
+    instances of the protocol environment: one whose context access
+    function are dummy ones which can be used when only the types and
+    non contextual functions of the protocol are needed, another that
+    simulates the key-value store in memory for testing, and a functor
+    that let you build an environment from a few context accessors.
+
+  - ``tezos-embedded-protocol-xxx`` contains a version of protocol
+    ``xxx`` whose standard library is pre-instanciated to the shell's
+    implementation, these are the ones that are linked into the
+    node. It alse contains a module that registers the protocol in the
+    node's protocol table.
+
+The Embedded Economic Protocols
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Three economic protocols are included in the main Tezos repository.
+
+  - :package:`tezos-protocol-genesis`
+    (:package:`tezos-embedded-protocol-genesis`) is the protocol of
+    the genesis block. It accepts a single block, signed by a dictator
+    whose public key is hardcoded, that single action is to switch to
+    a new protocol chosen by the dictator.
+  - :package:`tezos-protocol-alpha`
+    (:package:`tezos-embedded-protocol-alpha`) is the first real
+    protocol of Tezos. A :ref:`tutorial<entering_alpha>` is available
+    to start reading the protocol's code.
+  - :package:`tezos-protocol-demo`
+    (:package:`tezos-embedded-protocol-demo`) is just a demo protocol
+    that does nothing interesting but has the right shape.
+
+The Client Library
+~~~~~~~~~~~~~~~~~~
+
+The client is split into many packages, to enforce three separation
+lines: shell vs economic protocol, Unix dependent vs Javascript
+compatible, and library vs command line interface.
+
+  - :package:`tezos-client-base` define the client context, which is
+    an object whose methods allow for: accessing a wallet of keys,
+    interacting via the user, making RPC calls, and signing data using
+    signer plug-ins. Most of the, including RPC calling functions from
+    :package:`tezos-shell-services` and
+    :package:`tezos-protocol-alpha`, are abstracted over this object
+    type. That way, it is possible to use the same code for different
+    platforms ot toolkits.
+  - :package:`tezos-client-alpha` provides some functions to perform
+    the operations of protocol alpha using the wallet and signers from
+    the client context.
+  - :package:`tezos-client-commands` plugs the basic context access
+    functions from :package:`tezos-client-base` as handlers for the
+    commands of the ``tezos-client`` command line wallet.
+  - :package:`tezos-client-alpha-commands` plugs the functions from
+    :package:`tezos-client-alpha` as handlers for the alpha specific
+    commands of the ``tezos-client`` command line wallet.
+  - :package:`tezos-client-genesis` contains the basic dictator
+    commands available on the genesis protocol.
+  - :package:`tezos-client-base-unix` implements configuration file
+    and wallet storage in Unix files, user interaction via the Unix
+    console, and terminal based signer plug-ins.
+
+Tests Packages
+~~~~~~~~~~~~~~
+
+The tests are splitted into various packages, testing more and more
+elements while following the dependency chain. Use ``make test`` to
+run them.
+
+ - :package-name:`tezos-client`
+   (in directory :src:`src/bin_client/test/`):
+   end-to-end tests as shell scripts that launch a local sandboxed node
+   and performs various tasks using the client
+ - :package-name:`tezos-p2p`
+   (in directory :src:`src/lib_p2p/test/`):
+   tests of the peer-to-peer layer, independently of the Tezos gossip
+   protocol (establishing connections, propagating peers, etc.)
+ - :package-name:`tezos-protocol-environment`
+   (in directory :src:`src/lib_protocol_environment/test/`):
+   tests for the in-memory context implementation.
+ - :package-name:`tezos-shell`
+   (in directory :src:`src/lib_shell/test/`):
+   tests for the chain data storage.
+ - :package-name:`tezos-stdlib`
+   (in directory :src:`src/lib_stdlib/test/`):
+   tests for the basic data structures.
+ - :package-name:`tezos-storage`
+   (in directory :src:`src/lib_storage/test/`):
+   tests for the versioned key-value context.
+ - :package-name:`tezos-client-alpha`, :package-name:`tezos-baking-alpha`
+   (in directory :src:`src/proto_alpha/lib_baking/test/`):
+   tests of the OCaml client library, performing various short
+   scenarios via RPCs to a sandboxed node.
+ - :package-name:`tezos-protocol-alpha`
+   (in directory :src:`src/proto_alpha/lib_protocol/test/`):
+   tests of the alpha protocol (without launching a node).
+ - :package-name:`tezos-crypto`
+   (in directory :src:`src/lib_crypto/test/`):
+   tests for the in-house merkle trees.
+ - :package-name:`tezos-data-encoding`
+   (in directory :src:`src/lib_data_encoding/test/`):
+   tests for the JSON and binary serialization and deserialization.
+
+The Final Executables
+~~~~~~~~~~~~~~~~~~~~~
+
+  - :package:`tezos-node` provides the node launcher binary
+    ``tezos-node``. All the algorithmic being implemented in the
+    shell, this package only implements the node's CLI. It also
+    provides the sandboxed node shell script launcher (see the main
+    readme).
+  - :package:`tezos-client` provides the ``tezos-client`` and
+    ``tezos-admin-client`` binaries. The former contains a small
+    comand line wallet, the latter an administration tool for the
+    node. It also provides a shell script that configures a shell
+    environment to interact with a sandboxed node.
+  - :package:`tezos-baker-alpha` provides the ``tezos-alpha-baker``
+    binary.
+  - :package:`tezos-protocol-compiler` provides the
+    ``tezos-protocol-compiler`` binary that is used by the node to
+    compile new protocols on the fly, and that can be used for
+    developping new protocols.
+
+.. |Tezos source packages diagram| image:: packages.svg