Worked on the Actor Runtime section of the new paper.

doc/BlocktreeCloudPaper/BlocktreeCloudPaper.tex

@@ -91,11 +91,11 @@ By freeing the developer from dealing with the numerous failure modes that occur
 they are able to focus on the functionality of their system.
 
 Blocktree is implemented in the Rust programming language.
-Its source code is licensed under the Affero GNU Public License.
+Its source code is licensed under the Affero GNU Public License Version 3.
 It can be downloaded at the project homepage at \url{https://blocktree.systems}.
 Anyone interested in contributing to development is welcome to submit a pull request
 to \url{https://gogs.delease.com/Delease/Blocktree}.
-If you larger changes or architectural suggestions,
+If you have larger changes or architectural suggestions,
 please submit an issue for discussion prior to spending time implementing your idea.
 
 % Describe the remainder of the paper.
@@ -111,17 +111,139 @@ The remainder of this paper is structured as follows:
 \end{itemize}
 
 \section{Actor Runtime}
-% message passing interface
+% Motivation for using the actor model. 
+Building scalable fault tolerant systems requires us to distribute computation over
+multiple computers.
+Rather than switching to a different programming model when an application scales beyond the
+capacity of a single computer,
+it is beneficial in terms of programmer time and program simplicity to begin with a model that 
+enables multi-computer scalability.
+Fundamentally, all communication over an IP network involves the exchange of messages,
+namely IP packets.
+So if we wish to build scalable fault-tolerant systems,
+it makes sense to choose a programming model built on message passing,
+as this will ensure low impedance with the underlying networking technology.
 
-% btmsg and how it functions as a secure transport.
+% Overview of message passing interface.
+That is why Blocktree is built on the actor model
+and why its actor runtime is at the core of its architecture.
+The runtime can be used to spawn new actors, register services, and dispatch messages.
+Messages can be dispatched in two different ways: with \texttt{send} and \texttt{call}.
+A message is dispatched with the \texttt{send} method when no reply is required,
+and with \texttt{call} when exactly one is.
+The \texttt{Future} returned by \texttt{call} can be awaited to obtain the reply.
+If a timeout occurs while waiting for the reply,
+then the \texttt{Future} completes with an error.
+The name \texttt{call} was chosen to bring to mind a remote procedure call,
+which is the primary use case this method was intended for.
+Awaiting replies to messages serves as a simple way to synchronize a distributed computation.
 
-% security model based on filesystem permissions
+% Delivering messages over the network.
+Messages can be forwarded between actor runtimes using a secure transport layer called
+\texttt{bttp}.
+Messages are addressed using \emph{actor names}.
+An actor name is a pair consisting of the filesystem path of the runtime
+and a UUID specifying an actor in that runtime.
+Every message has a header containing the name of the sender and receiver.
+The transport is implemented using the QUIC protocol, which integrates TLS for security.
+The TLS handshake between runtimes is performed using mutual TLS authentication.
+This handshake cryptographically verifies the credentials of each runtime.
+These credentials contain the filesystem path where each runtime is located,
+which ensures that messages addressed to a specific path will only be delivered to the runtime
+at that path.
 
-% service discovery.
+% Delivering messages locally.
+When a message is sent between actors in the same runtime it is delivered into the queue of the recipient without any copying,
+while ensuring immutability (move semantics).
+This is possible thanks to the Rust ownership system,
+because the message sender gives ownership to the runtime when it dispatches the message,
+and the runtime gives ownership to the recipient when it delivers the message.
+
+% Security model based on filesystem permissions.
+A runtime is represented in the filesystem as a file.
+This file contains the authorization attributes which are associated with the runtime's security
+principal.
+The credentials used by the runtime specify the file, so other runtimes are able to locate it.
+The metadata of the file contains authorization attributes just like any other file
+(e.g. UID, GID, and mode bits).
+In order for a principal to be able to send a message to an actor in the runtime,
+it must have execute permissions for this file.
+Thus communication between runtimes can be controlled using simple filesystem permissions.
+Permissions checking is done during the \texttt{bttp} handshake.
+Note that it is possible for messages to be sent in one direction in a \texttt{bttp} connection
+but not in the other.
+In this situation replies are permitted but unsolicited messages are not.
+An important trade-off which was made when designing this model was that messages which are
+sent between actors in the same runtime are not subject to any authorization checks.
+This was done for two reasons: performance and security.
+By eliminating authorization checks messages can be more efficiently delivered between actors in the
+same process,
+which helps to reduce the performance penalty of the actor runtime over directly using threads.
+Security is enhanced by this decision because it forces the user to separate actors with different
+security requirements into different operating system processes,
+which ensures all of the process isolation machinery in the operating system will be used to
+isolate the different security domains.
+
+% Representing resources as actors.
+As in other actor systems, it is convenient to represent resources in Blocktree using actors.
+This allows the same security model used to control communication between actors to be used for
+controlling access to resources,
+and for resources to be shared by many actors.
+For instance, a Point-to-Point Protocol connection could be owned by an actor.
+This actor could forward traffic delivered to it in messages over this connection.
+The set of actors which are able to access the connection is controlled by setting the filesystem
+permissions on the file for the runtime executing the actor with the connection.
+
+% Service discovery.
+In addition to spawning actors, the runtime can also be used to register actors as service
+providers.
+A service is identified by a filesystem path.
+One or more actors may be register as providing a service.
+Services are resolved to actor names by the runtime.
+The service resolution method takes the path of a service and a scope path.
+The scope path defines the filesystem path where service resolution will begin.
+Resolution produces the name of an actor which is registered in a runtime which is closest to the
+scope, or \texttt{None} if no service provider can be found.
+To be more precise, consider the following cases:
+\begin{enumerate}
+  \item If the scope is the path of a runtime, and there are service providers registered in the
+    runtime, then one of their names if returned. Otherwise, service resolution is retried using a
+    new scope which is obtained by removing the last path component of the current scope.
+  \item If a directory is specified, then all of the runtimes in the directory are checked for
+    registered service providers, and if one is found its name is returned. Otherwise, service
+    resolution is retried using a new scope which is obtained by removing the last path component of
+    the current scope.
+  \item If the scope is the empty string, then \texttt{None} is returned.
+\end{enumerate}
+In order to contact other runtimes and query their service registrations,
+their IP addresses need to be known.
+To enable this a file with the runtime's IP address is maintained in the same directory as the
+runtime.
+The runtime is granted write permissions on the file,
+and it is updated by the transport layer when it begin listening on a new endpoint.
+
+% The sector and filesystem service.
+The filesystem is itself implemented as a service.
+A filesystem service provider can be passed messages to delete files, list directory contents,
+open files, or perform several other standard filesystem operations.
+When a file is opened,
+a new actor is spawned which owns the newly created file handle and its name is returned to the
+caller in a reply.
+Subsequent read and write messages are sent to this actor.
+The filesystem service does not persist any data itself,
+its job is to function as an integration layer,
+conglomerating sector data from many different sources into a single unified interface.
+The sector service is what is ultimately responsible for storing data,
+and thus maintaining the persistent state of the system.
+It stores sector data in the local filesystem of each computer on which it is registered.
+The details of how this is accomplished are deferred to the next section.
 
 % protocol contracts, and runtime checking of protocol adherence. Emphasize the benefits to
 % system composability that this enables, where errors can be traced back to the actor which
 % violated the contract.
+To facilitate the creation of composable systems,
+a protocol contract checking system based on session types has been designed.
+This system operates on a state transition model of a communications protocol.
 
 \section{Filesystem}
 % Benefits of using a distributed filesystem as the sole source of persistent state for the system,