Blocktree/doc/Paper/Paper.tex

\documentclass{article}
\usepackage{amsfonts,amssymb,amsmath}
\usepackage[scale=0.75]{geometry}
\usepackage{hyperref}
\usepackage{biblatex}
\bibliography{../citations.bib}

\title{Blocktree \\
\large A platform for distributed computing.}
\author{Matthew Carr}
\date{May 28, 2022}
\begin{document}
\maketitle
\begin{abstract}
Blocktree is a platform which aims to make developing reliable distributed systems easy and safe. 
It does this by defining a format for data stored in the system, a mechanism for replicating
that data, and a programming interface for accessing it. It aims to solve the difficult problems
of cryptographic key management, consensus, and global addressing so that user code can focus on
solving problems germane to their application.
\end{abstract}

\section{Introduction}
The online services that users currently have access to are incredible. They handle all
the details of backing up user data and implementing access controls to facilitate
safe sharing. However, because these are closed systems, users are forced to trust that
the operators are benevolent, and they lack any real way of ensuring that the access
controls they prescribe will actually be enforced. There have been several systems proposed
as alternatives to the conventional model, but these systems suffer from their own shortcomings.
They either assume the need for cloud storage providers (Blockstack \cite{blockstack}) or implement
all operations using a global blockchain, limiting performance (Filecoin \cite{filecoin}).
Blocktree takes a different approach.

The idea behind blocktree is to organize a user's computers into a cooperative unit, called a
blocktree. The user is said to own the blocktree, and they wield sovereign authority over it.
The artifact granting them this authority is the root private key of the blocktree. Measures for protecting
this key and delegating its authority are important design considerations of the system.
The owners of blocktrees are encouraged to collaborate with each other to replicate data by
means of a cryptocurrency known as blockcoin. The blockchain implementing this cryptocurrency
is the source of global state for the system, and allows for the creation of global paths.

All data stored in blocktree is contained in units called blocks. As the name would suggest, the blocks
in a blocktree form a tree. Each block has a path corresponding to its location in the tree. The first
component of a fully qualified blocktree path is the fingerprint of the root public key of the
blocktree. Thus a blocktree path can globally specify a block. If a block is not a leaf,
then it is called a directory, and the data it contains is managed by the system.
This information includes the list of blocks which are children of the directory as well as the
list of nodes which are attached to the block tree at this directory. In addition
to its payload of data,
each block has a header containing cryptographic access control mechanisms. These mechanisms ensure
that only authorized users can read and optionally write to the block.
Users and nodes in the blocktree system are identified by hashes of their public keys. These hashes
are referred to as principals, and they are used for setting access control policy.

This paper is intended to be short introduction to the ideas of blocktree. A book is planned which
will specify the system in greater detail. In keeping with the agile software methodology, this
book is being written concurrently with an open source implementation of the system.
The remainder of this paper is organized as follows:
\begin{itemize}
\item A description of the operations of a single blocktree.
\item The definition of a blockchain which provides global state and links individual blocktrees
together.
\item The programming interface for interacting with blocktrees and sending messages.
\item An exploration of applications that could be written using this platform.
\item Conclusion.
\end{itemize}

\section{An Individual Blocktree}
The atomic unit of data storage, confidentiality and authenticity is the block. A
block contains a payload of data. Confidentiality of this data is achieved by encrypting it using 
a symmetric cipher using a random key. This random key is known as the block key.
The block key can be encapsulated using the public key of a principal whose is to be given access.
The resulting ciphertext is stored in the header of the block. Thus
the person possessing the corresponding private key will be able to access the contents of
the block. Blocks are arranged into trees, and the parent block also has a block key.
The child's block key is always encapsulated using the parent's key and stored in the block
header. This ensures that when a principal is given read access to a block, it automatically has
access to every child of that block. The encapsulated block key is known as a read capability,
or readcap, as it grants the holder the ability to read the block. In the case of the root block
there is no parent block key to issue a readcap to. Instead, the root block always contains a
readcap for the the root key.

A note on visualizing a blocktree. In this paper I will describe the root block as being at the
bottom of the tree, and describe the descendants of a block as being above it. The reader may
recognize this as the ``trees grow up" model of formal logic, and it was chosen because such trees
appear visually similar to real world trees when rendered in a virtual environment.

Authenticity guarantees are provided using a digital signature scheme. In order to change the
contents of a block a data structure called a write capability, or writecap
\footnote{The names readcap and writecap were taken from the Tahoe Least-Authority Filesystem
\cite{tahoe}. The access control mechanism described in the Tahoe system heavily influenced the
design of Blocktree.},
is needed. A
writecap is approximately an x509 certificate chain. A writecap contains the following data:
\begin{itemize}
\item The path the writecap can be used under.
\item The principal that the writecap was issued to.
\item The timestamp when the writecap expires.
\item The public key of the principal who issued the writecap.
\item A digital signature produced by the private key corresponding to the above public key.
\item Optionally, the next writecap in the chain.
\end{itemize}
The last item is only excluded in the case of a self-signed writecap, i.e. one that was signed by
the same principal it was issued to. A writecap is considered valid for use in a block if all
of the following conditions are met:
\begin{itemize}
\item The signature on every writecap in the chain is valid.
\item The signing principal matches the principal the next writecap was issued to for every
writecap in the chain.
\item The path of the block is contained in the path of every writecap in the chain.
\item The current timestamp is strictly less than the expiration of all the writecaps in the
chain.
\item The principal corresponding to the public key used to sign the last writecap in the chain,
is the owner of the blocktree.
\end{itemize}
The intuition behind these rules is that a writecap is only valid if there is a chain of trust
that leads back to the owner of the blocktree. The owner may delegate their trust to any number
of intermediaries by issuing them writecaps. These writecaps are scoped based on the path
specified when they are issued. These intermediaries can then delegate this trust as well.
A block is considered valid if it contains a valid writecap, it was signed using the key
corresponding to the first writecap's public key, and this signature is valid.

Blocks are used for more than just organizing data, they also organize computation. A program
participating in a blocktree is referred to as a node. Multiple nodes may be run on
a single computer. Every node is attached to the blocktree at a specific path. This information
is recorded in the directory where the node is attached. A node is responsible for the storage of
the directory where it is attached and all of the blocks that are recursively contained with in it.
If there is a child node attached to a subdirectory contained in the directory the node is
responsible for, then the child node is responsible for the subdirectory.
In this way data storage can be delegated, allowing the system to scale. When more than one
node is attached to the same directory they form a cluster.
Each node in the cluster contains a copy of the data that the cluster is responsible for. They
maintain consistency of this data by running the Raft \cite{raft} consensus protocol.

When a new blocktree is created a node generates a key pair to serve as the root keys.
It is imperative for the security of the system that the root private key is protected, and it
is highly recommended that it be stored in a Trusted Platform Module (TPM) \cite{tpm} and that
the TPM
be configured to disallow unauthenticated use of this key. The node then generates its own key pair
and uses the root private key to issue itself a writecap for the root of the tree. Once it has
this writecap, it creates the root block and generates a block key for it. A readcap is added to
this block for the root public key and the node's public key. Additional
cryptographic operations are performed using the node's key pair, and only when a new writecap
needs to be created for an addition root node is the root private key used.

When a new node comes online and wishes to join the blocktree, it generates its own key pair.
The public key of this
node then needs to be transmitted to another node that's already part of the user's blocktree. The
mechanism used will depend on the nature of the device on which the new node is running.
For example, a phone could scan a QR code which contains
the IP address of the user's root node, and then transmit its public key to that internet host.
In order for the new node to be added to the user's blocktree, it needs to be issued a writecap
and a readcap must be added to the directory where it will be attached.
This could be accomplished
by providing a user interface on the node which received the public key from the new node.
This interface would show the requests that have been received from nodes attempting
to join the blocktree. The user can then choose to approve or deny the request, and can specify
the path where the new node will attach. If the user chooses to approve the request, then the writecap
is signed using the node's key and transmitted to then new node.

The ability to cope with key compromise is an important design consideration in any real-world
cryptosystem. In blocktree the compromise of a node key is handled by re-keying every block under
the directory where the node was attached. Specifically, this means that a new block key is generated for
each block, and the readcap for the compromised node is removed. This ensures that new writes to
these blocks will not be visible to the holder of the compromised key. To ensure that writes
will not be allowed, the root directory contains a revocation list containing the public keys
which have been revoked by the tree. These only need to be maintained until their writecaps
expire, after which time the list can be cleaned. Note that if the root private key is compromised or lost,
then the blocktree must be abandoned, there is no recovery. This is real security, the artifact
which grants control over the blocktree is the root private key which is why storing the root
private key in multiple secure cryptographic co-processors is so important.

Nodes in a blocktree communicate with one another by sending blocktree messages. These messages
are used for implementing consensus and distributed locking,
as well as notifying parent nodes when a 
child has written new data. The later mechanism allows the parent to replicate the data stored in
a child for redundancy. User code is also able to initiate the sending of messages. Messages are
addressed using blocktree paths. When a node receives a message that is not addressed to it,
but is addressed to its blocktree, it forwards it to the closest node to the recipient that it
is connected to. In order to enable efficient low-latency message transfers, nodes maintain open
TCP connections to the other nodes in their cluster, and the cluster leader maintains a connection to
its parent. Diffie-Hellman key exchange is used to exchange a key for use in an AEAD cipher, and
once this cipher context is established, the two nodes mutually authenticate each other using their
respective key pairs. When a node comes online, it uses the global blocktree (described later)
to find the other nodes in its cluster. If it is not part of a cluster, or this information is
not stored in the global blocktree, then it instead looks up the
IP address of a root node and connects to it. The root node may direct the node to connect to
one of root's children, and this process repeats until the new node is connected to its parent.

A concept that has proven to be very useful in the world of filesystems is the symbolic link.
This is a short file that contains the path to another file, and is interpreted by most programs
as being a "link" to that file. Blocktree supports a similar system, where a block can be
marked as a symbolic link and a blocktree path placed in its body. This also provides us with
a convenient way of storing readcaps for data that a node would otherwise not have access to.
For instance a symbolic link could be created which points to a block in another user's blocktree.
The other user only knows the public key of the owner of our blocktree, so they issue
a readcap to it. But the root nodes can open this readcap and extract
the block key. This key can then be encapsulated using the public key of the node which
requires access, and placed in the symbolic link. When the node needs to read the data
in the block, it opens the readcap in the symbolic link, follows the link to the block (how
that actually happens will be discussed below) and decrypts its contents.

While the consistency of individual blocks can be maintained using Raft, a distributed locking
mechanism is employed to enable transactions which span multiple blocks. This is
accomplished by exploiting the hierarchical arrangement of nodes in the tree. In order to describe
this, its first helpful to define a new term. The \emph{nodetree} of a blocktree is tree obtained
from the blocktree by collapsing all the blocks that a node (or cluster of nodes) is responsible
for into a single logical block representing the node itself. Thus we can talk about a node having
a parent, and by this we mean its parent in the nodetree. In terms of the blocktree, the parent
of a node is the first node encountered when the path back to the root is traversed.
Now, distributed locking works as follows:
\begin{itemize}
\item A node sends a request to lock a block to the current consensus leader in its cluster.
  If the node is not part of a cluster, then it is the leader. This request contains a timestamp
  for when the lock expires.
\item If the leader is responsible for the block then it moves on to the next step. Otherwise
  it contacts its parent and forwards the lock request and this step is repeated for the parent.
\item The responsible node checks to see if there is already a lock for this block. If there is
  then the request fails. Otherwise the request succeeds and a lock is placed on the block. A
  message indicating the result is then passed back up the tree ending at the original node. This
  message includes the principal of the node enforcing the lock.
\item Once the locking node is done making its updates it sends a message directly to the node
  enforcing the lock, causing it to be removed.
\end{itemize}
Locking a block locks the subtree rooted at that block. Thus no writes to any path contained in
the path of the locked block will be allowed, unless they come from locking node. If the locking
node does not send the message
unlocking the block before the lock expires, then the modifications which had been performed by
it are dropped and the block reverts to its prior state. Since the locking node is the leader
of the consensus cluster that is responsible for the block, this guarantees that
writes from other nodes will not be accepted.

\section{Connecting Blocktrees}
In order to allow nodes to access blocks in other blocktrees, a global ledger of events is used.
This ledger is implemented using a proof of work (PoW) blockchain and a corresponding 
cryptocurrency known as blockcoin. Nodes mine chain blocks (not to be confused with the tree 
blocks we've been discussing up till now) in the same way they do in other PoW blockchain
systems such as Bitcoin \cite{bitcoin}. The node which manages to mine the next chain block receives
a reward,
which is the sum of the fees for each event in the chain and a variable amount of newly minted
blockcoin. The amount of new blockcoin created by a chain block is directly proportional to the
amount of data storage events contained in the chain block. Thus the total amount of blockcoin
in circulation has a direct relationship to the amount of data stored in the system, reflecting
the fact that blockcoin exists to provide an accounting mechanism for data.

When a node writes data to a tree block, and it wishes this block to be globally accessible or
replicated for redundancy,
it produces what are called fragments. Fragments are the output symbols of the RaptorQ code
\cite{raptorq}. This algorithm is an example of an Erasure Code, which is a class of fountain codes
which have the property that only $m$ out of $n$ (where $m < n$) symbols are needed to reconstruct
the original data. Such a code ensures that even if some of the fragments are lost, as long as $m$
remain, the original data can be recovered.

Once these fragments have been computed an event is created for each one and published to the
blockchain. This event indicates to other nodes that this node wishes to store a fragment and
states the amount of blockcoin it will pay for each of the fragment's maintenance payments. When
another nodes wishes to accept the offer, it directly contacts the first node, which then sends 
it the fragment and publishes an event stating that the fragment is stored with the second 
node. This event includes the path of the block the fragment was computed from, the fragment's 
ID (the sequence number from the erasure code), and the principal of the node which stored it.
Thus any other node in the network can use the information contained in this event to
determine the set of nodes which contain the fragments of any given path.

In order for the node which stored a fragment to receive its next payment, it has to pass
a time-bound challenge-response protocol initiated by the node that owns the fragment.
The owning node select a leaf in the Merkle tree of the fragment and sends the index of
this leaf to the storing node. The storing node then walks the path from this leaf back to
the root of the Merkle tree, and updates a hash value using the data in each node it traverses.
It sends this result back to the owning node which verifies that this value matches its
own computation. If it does then the owning node signs a message indicating that the challenge
passed and that the storing node should be paid. The storing node receives this message and uses
it to construct an event, which it signs and publishes to the blocktree. This event causes
the blockcoin amount specified to be transferred from the owning node's to the storing
node's blocktree.

The fact that payments occur over time provides a simple incentive for nodes to be honest and
store the data they agree to. In banking terms, the storing node views the fragment as an
asset, it is a loan of its disk space which provides a series of payments over time.
On the other hand the owning node views the fragment as a liability, it requires payments to
be made over time. In order for a blocktree owner to remain solvent, it must balance its
liabilities with its assets, incentivizing it to store data for others so that its own data
will be stored.

In order for nodes to be able to contact other nodes, a mechanism is required for associating
an internet protocol (IP) address with a principal. This is done by having nodes publish events
to the blockchain when their IP address changes. This event includes their new IP address,
their public key, and a digital signature computed using their private key. Other nodes can
then verify this signature to ensure that an attacker cannot bind the wrong
IP address to a principal in order to receive messages not meant for it.

While this event ledger is useful for appending new 
events, and ensuring previous events
cannot be changed, another data structure is required to enable efficient queries.
In particular, it's important to be able to quickly perform the
following queries:
\begin{itemize}
\item Find the set of nodes storing the fragments for a given path.
\item Find the IP address of a node or owner given a principal.
\item Find the public key associated with a principal.
\end{itemize}
To enable these queries a special blocktree is maintained by each node in the network: the global
blocktree. This tree does not support the usual writing and locking semantics of local blocktrees.
In functional programming terms, it can be thought of as a left fold over all of the events in the
blockchain.
The above queries are facilitated by the following blocks:
\begin{itemize}
\item \emph{/global/fragments}: this block contains a hashtable where the key is a path and the
value is the list of nodes storing the fragments for the block at that path.
\item \emph{/global/principals}: contains a hashtable where the key is the a principal and the value
is the tuple containing the public key of that principal, its current IP address, and its current
blockcoin balance.
\end{itemize}
To compute the entries in these tree blocks, the nodes in the network iterate over all the chain blocks, updating
their local copy of each tree block appropriately. The experienced reader will recognize that this is an event 
sourced architecture. Currently only these two tree blocks are known to be needed, but if new events are
added to the system it can be easily extended to enable queries that have yet to be envisioned.

\section{Programming Environment}
Enabling an excellent developer experience is one of the primary goals of this system (the others being security
and scalability). Nodes execute user code that has been compiled into WebAssembly modules \cite{wasm}. Such code
running on a blocktree node is referred to as an "app". An app
executes in a sandbox that isolates it from other code, as well as the security critical operations of the node
itself. The sandbox provides the code with an extension of the WebAssembly System Interface (WASI), with extra 
system calls to interact with the particulars of the blocktree system.
The extra system calls fall into these categories:
\begin{itemize}
\item Distributed Locking
\item Messaging
\item Supervision Trees
\item Protocol Contracts (Session Types)
\end{itemize}
The standard WASI filesystem APIs are used
to interact with the contents of blocktrees. For instance a file descriptor for a block can be obtained
by calling path\_open. Writes and reads of blocks are performed using the privileges of the node on which
the app is running, but the node may limit the app's access depending on its permissions.

When an app is installed it is given a directory under which it can store data that is shared between all nodes
in the blocktree. The path of this block is formed by prefixing the path the app was published at
with the string ``/apps". When an app is runs on a node, it is confined to a block contained
in the node's directory. It is only allowed read and write blocks in this block, but to allow it to
access shared data, a symbolic link is created to the app's shared directory in ``/apps". 

% App Publishing
Apps are published by writing them into a blocktree. The path of the directory used to publish an app is used to
identify it. Only one app per directory is allowed. This directory contains the WebAssembly module itself as
well as a JSON manifest. This manifest defines the app's user-friendly name as well as the list
of permissions it requires. This list of permissions is used to determine which APIs the app has access to.

% Privacy Safe vs Unsafe
Apps are broken into two categories: those that are privacy safe and those that are not.
An app is privacy unsafe if it requests any permissions which allow it to send data
outside of the blocktree it's part of. Thus requesting the ability to
open a TCP socket would cause an app to be privacy unsafe. Similarly, the creation of a protocol
handler for HTTP would also be privacy unsafe. Privacy unsafe apps can limit the scope of their
unsafety by
imposing limits on the unsafe APIs that they request. For instance an app which needs to send
blocktree message back to the blocktree it was published in can request the messaging permission
for a path in this tree. Similarly, an app which only wants to open a TCP socket listening on the
local network, can limit the scope of its requested permission.

% Protocol Contracts
In order to make it easy to write apps which interface with existing systems, most notably
those using HTTP, apps are able to define protocols using a contract language and then register
call backs for these protocols. This works by providing a system call which the app can
supply a protocol contract to. This contract is then compiled into a state machine by the node
and a handle is returned to the app. This handle is then used to register callbacks for different
parts of the protocol. This ensures that the protocol is handled by the node itself and that
protocols can be shared between many different apps as a library. For instance, the HTTP protocol
would be compiled to a particularly simply state machine, with only one state: the listening state.
This state would expose a hook that where a callback can be registered to handle a request.
This state also defines the record format used to pass the request information to the callback
and the record format of the return value that is expected in order to produce the response.
More complicated (stateful) protocols would have more states, each defining their own request and
response records, as well as hooks. One nice thing about this setup is that it will enable
optimizations where the state machine and the user callbacks can be compiled into a program
which can be safely run in the node itself, or even in a SmartNIC. This would require that
the callbacks only use an approved set of APIs, but could enable much higher performance.

% Supervision Trees
Apps can arrange themselves into supervision trees, in the same way that Erlang
processes are arranged \cite{armstrong}. In this scheme, when a child app crashes, or the node its
running on dies (which is detected by other nodes), then the app receives a message.
In the
simplest case this can be used to implement a logging system, where crashes and node
deaths are recorded. More interestingly, this can be used to integrate with a control
plane. For instance, if a blocktree were running in AWS, when a message is received indicating
that a node has died, a new EC2 instance could be started to replace it. The reliability of Erlang
and other system employing the Actor Model have shown the robustness of this approach.
Apps can form this relationship after they've started running, provided that both of the apps
have permission to send messages to each other. Apps, with the appropriate permissions, can also
spawn other apps on descendent nodes. This can be used to implement map-reduce workloads, where
an app spawns mapping jobs on descendent nodes containing the data of interest, and it processes
their messages to compute the final reduction. Due to the tiny size of most programs, this is a much
more efficient approach than moving the data to the nodes performing the computation.

\section{A Brave New Web}
In order to explore how blocktree can be used, the design of several hypothetical systems
is discussed. It's important to note that blocktree does not try to force all computation
to be local to a user's device, it merely tries to enable this for applications where it
is possible.

\subsection{Contacts and Mail}
The first application we'll consider is one which manages a user's contacts. This would expose
the usual create, read, update and delete operations, allowing a user to input the name
of a person they know and associate
that name with their public key. Once the principal of a person is known, then their public
key can be looked up in the global blocktree. This principal needs to be communicated to the
user via some out-of-band method. They could receive it in an email, a text message, or embedded
in a QR code. Of course this out-of-band communication needs to be authenticated, otherwise
it would be easy to fool the user into associating an attacker's key for the person.

The user now has a way of associating a blocktree with the name of this person. However, the
root public key of this blocktree is not enough to establish secure communications, because
the root private key is not available to every node in the person's blocktree. In particular
it would be inadvisable for the root private key to be stored on a user's mobile device. To
address this mailbox directories are created.

For each contact two directories are created: the inbox and the outbox. The user creates a readcap
for another user's root key and adds it to their outbox. The inbox for the other user is a symbolic
link to the user's outbox in the blocktree of the other user. Thus each user
can write messages into their own blocktree at a location where the other party knows how
to find them. But, in order for a node to read these messages it requires a its own readcap.
Only the root
nodes can issue this readcap as only they have access to the root key. Once permission has been
granted to a node, a root node can use the root key to decrypt the readcap issued to it, and then
encrypt it using the public key of the node. The resulting readcap is then stored in the header
of the inbox.

In addition to being able to check the inbox for mail, a blocktree message is sent to the receiving
blocktree when mail is sent. This message may contain the entire contents of the mail, if the
contents are short enough. But if the mail contains a lot of data, for instance a video, then
the message just serves as a notification that new mail is available.

\subsection{Social Network}
Building a social network on top of the contacts app is fairly straight-forward. Once
a contacts entry has been created for a person, most interactions between the user and that
person can be implemented by sending mail. For example, when the
user sends a direct message to a person mail is placed in their outbox
and a blocktree message is sent to the root cluster of their blocktree. If the user wishes to 
send a message to a group of people, mail is sent to each one.

This same mechanism can be used to implement status updates. When a user updates their status,
they send mail to each of their "friends". The social networking app running on each of the contacts'
devices will then display the latest status update from the user as their current status. This
setup allows the user to "unfriend" anyone by simply omitting them from the list of recipients
of these updates. To allow comments and likes on the status update to be visible to everyone
that it was shared with, the block created in the outbox of each of the friends
is a symbolic link to a single status update block. This symbolic link contains the
readcap for the status update block.

Comments and likes on status updates are implemented by sending mail to the user who posted the
update. When one of the user's nodes receives this mail, it then updates the block containing
the status update with the new comment, or increments the like counter.
It then sends mail to the people this status update was shared with, notifying them
that the data has changed.

\subsection{An e-commerce website.}
The previous two examples show how to build decentralized versions of existing web applications,
but blocktree also excels at building traditional web applications. Imagine an e-commerce website
with multiple warehouses, all of whose inventory is to be visible to customers of the site. Part
of the design consideration for the site is that the warehouses need to be able to update their
inventory even when their internet service goes down, and that these updates need to be visible
on the website once connectivity is restored.

To accomplish this a designer could create a directory for each warehouse. This directory would have
nodes attached to it that are physically located at each warehouse. The inventory of the warehouse
is then maintained in the warehouse's directory, satisfying the first requirement. Now, in order to
enable efficient queries of the overall available inventory, the data from each of the warehouses
needs to be merged. This is accomplished by creating another directory containing the merged data.
In event sourcing terms this is called a read-model. This directory will have another cluster of
nodes attached which will act as web servers. These nodes will subscribe to events published by the
warehouse clusters, events indicating changing levels of inventory, and digest this information into
a format that can be efficiently queried. These nodes will also publish events to the warehouses
when orders are placed, cancelled or amended.

When a warehouse goes offline, its previous inventory counts are
still recorded in the read-model, so queries can still be answered using the cluster's best
knowledge of the warehouse. Once the warehouse comes back online, then the events that were recorded by the
warehouse cluster while it was offline can be replayed to the web server cluster, and their read-model
can be brought up to date.

One of the advantages of this approach is that the cluster of web servers need not be physically
close to each other. In fact they could be spread over several data centers, even in different
cloud providers. This allows for greater fault tolerance and reliability. Of course, running a
consensus cluster over a larger area means more latency and thus reduced performance, but if
the workload is read-heavy, and writes can be handled mostly in the warehouse clusters, then this
tradeoff may be worthwhile.

I hope this example shows that having a standard format for data and the federation of servers,
can provide designers with much greater flexibility, even if they do not care about decentralization
or their user's privacy.

\subsection{The Open Metaverse}
As a final example I'd like to consider a platform for recording spacial information. The key insight
that enables this is very general: blocktree enables the creation of distributed tree-like data structures.
For instance, its straight forward to imagine creating a distributed hashtable implemented as a red-black tree.
This impedance match between efficient query structures and the design of blocktree is one
of the reasons why I believe it is so useful. The particular data structure germane to building the metaverse
is the Binary Space Partition (BSP) tree.

I don't see the metaverse as being one world, but rather many. Each world would be hosted by its own blocktree.
The world will have a directory in the blocktree, that directory will be the root of a BSP. read
access to the world is controlled by adding readcaps to this directory. If the world is to be
publicly accessible, then then its block can be stored in plaintext.

Thinking of worlds
like ours (those that can be reasonably approximated as the surface of a sphere) we can impose the usual latitude
and longitude coordinate system. We can then define parcels in the world by specifying the polygonal boundary as
a set of these coordinates (more precisely, the parcel is the convex hull of this set of points). These parcels
are then recorded as blocks in the world's directory, whose path is determined by its virtual location using
the BSP algorithm. If a parcel
is owned by the same user who owns the world, then the data contained in the parcel is stored in
the same blocktree. However, if a parcel is to be owned by another user, then a symbolic link is created
pointing to a block in the owner's blocktree. They can then write whatever data they want into this block, defining
the contents of their parcel. Collaboration on a single parcel is accomplished by issuing a read and writecap to
another user.

It's easy to imagine that one world would be more important than the rest and that he creation of a metaverse 
representation of Earth will be an important undertaking. The hierarchical nature of permissions in blocktree
make such a shared world possible. National blocktrees could be given ownership of their virtual territory,
this would then
be delegated down to the state and municipal levels. Finally municipalities would delegate individual parcels
to their actual owners. Owners could then use these as they see fit, including leasing them to third parties.
The ending date of such a lease would be enforced technically by the writecap issued to the lessee; when it
expires so too does the lease.

\section{Conclusion}
In this paper I have given the outline of a decentralized method of organizing information, computation,
and trust in a way that I hope will be useful and easy to use. The use of cryptographic primitives for
implementing access control were discussed, as well as methods of protecting private keys. A blockchain
and corresponding cryptocurrency was proposed as a means of incentivizing the distribution of data.
Erasure coding was used to ensure that distributed data is resilient to the loss of nodes and can be
reconstructed efficiently. A programming environment based on WASM and WASI was proposed as a way
of providing an interface to this data. APIs for defining protocol contracts and efficient web servers
were indicated. APIs for constructing supervision trees were mentioned as a means for building reliable
systems.

When Sir Tim Berners-Lee invented HTTP he could not have anticipated the applications
that his inventions would bring about. It has been said that the key to the success of his system is
that it made networking programming so easy that anyone could do it. I don't know what the future will
bring, but I hope that this system, or one like it, will enable anyone to fearlessly build distributed
systems.
One thing is certain however, it is a moral imperative that we provide users with viable alternatives
to online services which harvest their data and weaponize it against them. Only then will the web
become the place it was meant to be.

\printbibliography

\end{document}