|
@@ -91,11 +91,11 @@ By freeing the developer from dealing with the numerous failure modes that occur
|
|
they are able to focus on the functionality of their system.
|
|
they are able to focus on the functionality of their system.
|
|
|
|
|
|
Blocktree is implemented in the Rust programming language.
|
|
Blocktree is implemented in the Rust programming language.
|
|
-Its source code is licensed under the Affero GNU Public License.
|
|
|
|
|
|
+Its source code is licensed under the Affero GNU Public License Version 3.
|
|
It can be downloaded at the project homepage at \url{https://blocktree.systems}.
|
|
It can be downloaded at the project homepage at \url{https://blocktree.systems}.
|
|
Anyone interested in contributing to development is welcome to submit a pull request
|
|
Anyone interested in contributing to development is welcome to submit a pull request
|
|
to \url{https://gogs.delease.com/Delease/Blocktree}.
|
|
to \url{https://gogs.delease.com/Delease/Blocktree}.
|
|
-If you larger changes or architectural suggestions,
|
|
|
|
|
|
+If you have larger changes or architectural suggestions,
|
|
please submit an issue for discussion prior to spending time implementing your idea.
|
|
please submit an issue for discussion prior to spending time implementing your idea.
|
|
|
|
|
|
% Describe the remainder of the paper.
|
|
% Describe the remainder of the paper.
|
|
@@ -111,17 +111,139 @@ The remainder of this paper is structured as follows:
|
|
\end{itemize}
|
|
\end{itemize}
|
|
|
|
|
|
\section{Actor Runtime}
|
|
\section{Actor Runtime}
|
|
-% message passing interface
|
|
|
|
|
|
+% Motivation for using the actor model.
|
|
|
|
+Building scalable fault tolerant systems requires us to distribute computation over
|
|
|
|
+multiple computers.
|
|
|
|
+Rather than switching to a different programming model when an application scales beyond the
|
|
|
|
+capacity of a single computer,
|
|
|
|
+it is beneficial in terms of programmer time and program simplicity to begin with a model that
|
|
|
|
+enables multi-computer scalability.
|
|
|
|
+Fundamentally, all communication over an IP network involves the exchange of messages,
|
|
|
|
+namely IP packets.
|
|
|
|
+So if we wish to build scalable fault-tolerant systems,
|
|
|
|
+it makes sense to choose a programming model built on message passing,
|
|
|
|
+as this will ensure low impedance with the underlying networking technology.
|
|
|
|
|
|
-% btmsg and how it functions as a secure transport.
|
|
|
|
|
|
+% Overview of message passing interface.
|
|
|
|
+That is why Blocktree is built on the actor model
|
|
|
|
+and why its actor runtime is at the core of its architecture.
|
|
|
|
+The runtime can be used to spawn new actors, register services, and dispatch messages.
|
|
|
|
+Messages can be dispatched in two different ways: with \texttt{send} and \texttt{call}.
|
|
|
|
+A message is dispatched with the \texttt{send} method when no reply is required,
|
|
|
|
+and with \texttt{call} when exactly one is.
|
|
|
|
+The \texttt{Future} returned by \texttt{call} can be awaited to obtain the reply.
|
|
|
|
+If a timeout occurs while waiting for the reply,
|
|
|
|
+then the \texttt{Future} completes with an error.
|
|
|
|
+The name \texttt{call} was chosen to bring to mind a remote procedure call,
|
|
|
|
+which is the primary use case this method was intended for.
|
|
|
|
+Awaiting replies to messages serves as a simple way to synchronize a distributed computation.
|
|
|
|
|
|
-% security model based on filesystem permissions
|
|
|
|
|
|
+% Delivering messages over the network.
|
|
|
|
+Messages can be forwarded between actor runtimes using a secure transport layer called
|
|
|
|
+\texttt{bttp}.
|
|
|
|
+Messages are addressed using \emph{actor names}.
|
|
|
|
+An actor name is a pair consisting of the filesystem path of the runtime
|
|
|
|
+and a UUID specifying an actor in that runtime.
|
|
|
|
+Every message has a header containing the name of the sender and receiver.
|
|
|
|
+The transport is implemented using the QUIC protocol, which integrates TLS for security.
|
|
|
|
+The TLS handshake between runtimes is performed using mutual TLS authentication.
|
|
|
|
+This handshake cryptographically verifies the credentials of each runtime.
|
|
|
|
+These credentials contain the filesystem path where each runtime is located,
|
|
|
|
+which ensures that messages addressed to a specific path will only be delivered to the runtime
|
|
|
|
+at that path.
|
|
|
|
|
|
-% service discovery.
|
|
|
|
|
|
+% Delivering messages locally.
|
|
|
|
+When a message is sent between actors in the same runtime it is delivered into the queue of the recipient without any copying,
|
|
|
|
+while ensuring immutability (move semantics).
|
|
|
|
+This is possible thanks to the Rust ownership system,
|
|
|
|
+because the message sender gives ownership to the runtime when it dispatches the message,
|
|
|
|
+and the runtime gives ownership to the recipient when it delivers the message.
|
|
|
|
+
|
|
|
|
+% Security model based on filesystem permissions.
|
|
|
|
+A runtime is represented in the filesystem as a file.
|
|
|
|
+This file contains the authorization attributes which are associated with the runtime's security
|
|
|
|
+principal.
|
|
|
|
+The credentials used by the runtime specify the file, so other runtimes are able to locate it.
|
|
|
|
+The metadata of the file contains authorization attributes just like any other file
|
|
|
|
+(e.g. UID, GID, and mode bits).
|
|
|
|
+In order for a principal to be able to send a message to an actor in the runtime,
|
|
|
|
+it must have execute permissions for this file.
|
|
|
|
+Thus communication between runtimes can be controlled using simple filesystem permissions.
|
|
|
|
+Permissions checking is done during the \texttt{bttp} handshake.
|
|
|
|
+Note that it is possible for messages to be sent in one direction in a \texttt{bttp} connection
|
|
|
|
+but not in the other.
|
|
|
|
+In this situation replies are permitted but unsolicited messages are not.
|
|
|
|
+An important trade-off which was made when designing this model was that messages which are
|
|
|
|
+sent between actors in the same runtime are not subject to any authorization checks.
|
|
|
|
+This was done for two reasons: performance and security.
|
|
|
|
+By eliminating authorization checks messages can be more efficiently delivered between actors in the
|
|
|
|
+same process,
|
|
|
|
+which helps to reduce the performance penalty of the actor runtime over directly using threads.
|
|
|
|
+Security is enhanced by this decision because it forces the user to separate actors with different
|
|
|
|
+security requirements into different operating system processes,
|
|
|
|
+which ensures all of the process isolation machinery in the operating system will be used to
|
|
|
|
+isolate the different security domains.
|
|
|
|
+
|
|
|
|
+% Representing resources as actors.
|
|
|
|
+As in other actor systems, it is convenient to represent resources in Blocktree using actors.
|
|
|
|
+This allows the same security model used to control communication between actors to be used for
|
|
|
|
+controlling access to resources,
|
|
|
|
+and for resources to be shared by many actors.
|
|
|
|
+For instance, a Point-to-Point Protocol connection could be owned by an actor.
|
|
|
|
+This actor could forward traffic delivered to it in messages over this connection.
|
|
|
|
+The set of actors which are able to access the connection is controlled by setting the filesystem
|
|
|
|
+permissions on the file for the runtime executing the actor with the connection.
|
|
|
|
+
|
|
|
|
+% Service discovery.
|
|
|
|
+In addition to spawning actors, the runtime can also be used to register actors as service
|
|
|
|
+providers.
|
|
|
|
+A service is identified by a filesystem path.
|
|
|
|
+One or more actors may be register as providing a service.
|
|
|
|
+Services are resolved to actor names by the runtime.
|
|
|
|
+The service resolution method takes the path of a service and a scope path.
|
|
|
|
+The scope path defines the filesystem path where service resolution will begin.
|
|
|
|
+Resolution produces the name of an actor which is registered in a runtime which is closest to the
|
|
|
|
+scope, or \texttt{None} if no service provider can be found.
|
|
|
|
+To be more precise, consider the following cases:
|
|
|
|
+\begin{enumerate}
|
|
|
|
+ \item If the scope is the path of a runtime, and there are service providers registered in the
|
|
|
|
+ runtime, then one of their names if returned. Otherwise, service resolution is retried using a
|
|
|
|
+ new scope which is obtained by removing the last path component of the current scope.
|
|
|
|
+ \item If a directory is specified, then all of the runtimes in the directory are checked for
|
|
|
|
+ registered service providers, and if one is found its name is returned. Otherwise, service
|
|
|
|
+ resolution is retried using a new scope which is obtained by removing the last path component of
|
|
|
|
+ the current scope.
|
|
|
|
+ \item If the scope is the empty string, then \texttt{None} is returned.
|
|
|
|
+\end{enumerate}
|
|
|
|
+In order to contact other runtimes and query their service registrations,
|
|
|
|
+their IP addresses need to be known.
|
|
|
|
+To enable this a file with the runtime's IP address is maintained in the same directory as the
|
|
|
|
+runtime.
|
|
|
|
+The runtime is granted write permissions on the file,
|
|
|
|
+and it is updated by the transport layer when it begin listening on a new endpoint.
|
|
|
|
+
|
|
|
|
+% The sector and filesystem service.
|
|
|
|
+The filesystem is itself implemented as a service.
|
|
|
|
+A filesystem service provider can be passed messages to delete files, list directory contents,
|
|
|
|
+open files, or perform several other standard filesystem operations.
|
|
|
|
+When a file is opened,
|
|
|
|
+a new actor is spawned which owns the newly created file handle and its name is returned to the
|
|
|
|
+caller in a reply.
|
|
|
|
+Subsequent read and write messages are sent to this actor.
|
|
|
|
+The filesystem service does not persist any data itself,
|
|
|
|
+its job is to function as an integration layer,
|
|
|
|
+conglomerating sector data from many different sources into a single unified interface.
|
|
|
|
+The sector service is what is ultimately responsible for storing data,
|
|
|
|
+and thus maintaining the persistent state of the system.
|
|
|
|
+It stores sector data in the local filesystem of each computer on which it is registered.
|
|
|
|
+The details of how this is accomplished are deferred to the next section.
|
|
|
|
|
|
% protocol contracts, and runtime checking of protocol adherence. Emphasize the benefits to
|
|
% protocol contracts, and runtime checking of protocol adherence. Emphasize the benefits to
|
|
% system composability that this enables, where errors can be traced back to the actor which
|
|
% system composability that this enables, where errors can be traced back to the actor which
|
|
% violated the contract.
|
|
% violated the contract.
|
|
|
|
+To facilitate the creation of composable systems,
|
|
|
|
+a protocol contract checking system based on session types has been designed.
|
|
|
|
+This system operates on a state transition model of a communications protocol.
|
|
|
|
|
|
\section{Filesystem}
|
|
\section{Filesystem}
|
|
% Benefits of using a distributed filesystem as the sole source of persistent state for the system,
|
|
% Benefits of using a distributed filesystem as the sole source of persistent state for the system,
|