|
@@ -1031,18 +1031,236 @@ one can begin monitoring files as soon as they are created.
|
|
|
|
|
|
|
|
|
\section{Examples}
|
|
|
-This section contains examples of systems built using Blocktree. The hope is to illustrate how this
|
|
|
-platform can be used to implement existing applications more easily and to make it possible to
|
|
|
-implement systems which are currently out of reach.
|
|
|
-
|
|
|
-\subsection{A personal cloud for a home user.}
|
|
|
-% Describe my idealized home Blocktree setup.
|
|
|
-
|
|
|
-\subsection{An ecommerce website.}
|
|
|
-% Describe a blocktree which runs a cluster of webservers, a manufacturing process, a warehouse
|
|
|
-% inventory management system, and an order fulfillment system.
|
|
|
-
|
|
|
-\subsection{A smart home.}
|
|
|
+This section contains examples of systems that could be built using Blocktree.
|
|
|
+The hope is to illustrate how this platform can be used to implement existing applications more
|
|
|
+easily and to make it possible to implement systems which are currently out of reach.
|
|
|
+
|
|
|
+\subsection{A distributed AI execution environment.}
|
|
|
+Neural networks are just vector-valued functions with vector inputs,
|
|
|
+albeit very complicated ones with potentially billions of parameters.
|
|
|
+But, just like any other computation,
|
|
|
+these functions can be conceptualized as computational graphs.
|
|
|
+Imagine that you have a set of computers equipped AI accelerator hardware
|
|
|
+and you have a neural network that is too large to be processed by any one of them.
|
|
|
+By partitioning the graph into small enough subgraphs,
|
|
|
+we can break the network down into pieces which can be processed by each of the accelerators.
|
|
|
+The full network can be stitched together by passing messages between each of these pieces.
|
|
|
+
|
|
|
+Let us consider how this could be accomplished with Blocktree.
|
|
|
+We begin by provisioning a runtime on each of the accelerator machines,
|
|
|
+each of which will have a new accelerator service registered.
|
|
|
+Messages will be sent to the accelerator service describing the computational graph to execute,
|
|
|
+as well as the name of the actor to which the output is to be sent.
|
|
|
+When such a message is received by an accelerator service provider,
|
|
|
+it spawns an actor which compiles its subgraph to a kernel for its accelerator
|
|
|
+and remembers the name of the actor to send its output to.
|
|
|
+An orchestrator service will be responsible for partitioning the graph and sending these messages.
|
|
|
+Ownership of the actors spawned by the accelerator service is given to the orchestrator service,
|
|
|
+ensuring that they will all be stopped when the orchestrator returns.
|
|
|
+When one of the spawned actors stops,
|
|
|
+it unloads the kernel from the accelerator's memory and returns it to its initial state.
|
|
|
+Note that the orchestrator actor must have execute permissions on each of the accelerator runtimes
|
|
|
+in order to send messages to them.
|
|
|
+The orchestrator dispatches messages to the accelerator service in reverse order of the flow of data
|
|
|
+in the computational graph,
|
|
|
+so that it can tell each service provider where its output should be sent.
|
|
|
+The actors responsible for the last layer in the computational graph send their output to the
|
|
|
+orchestrator.
|
|
|
+To begin the computation,
|
|
|
+the actors which are responsible for input are given the filesystem path of the input data.
|
|
|
+The orchestrator learns of the completion of the computation once it receives the output from
|
|
|
+final layer.
|
|
|
+It can then save these results to the file system and return.
|
|
|
+Because inference and training can both be modeled by computational graphs,
|
|
|
+this same procedure can be used for both.
|
|
|
+
|
|
|
+\subsection{A decentralized social media network.}
|
|
|
+One of the original motivations for designing Blocktree was to create a platform for a social
|
|
|
+network that puts users in fully in control of their data.
|
|
|
+In the opinion of the author,
|
|
|
+the only way to actually accomplish this is for users to host the data themselves.
|
|
|
+One might think it is possible to use client-side encryption to solve the privacy issue,
|
|
|
+but this does not solve the full problem.
|
|
|
+While it is true that good client-side encryption will prevent the service provider from reading
|
|
|
+the user's data,
|
|
|
+the user could still loose everything if the service provider goes out of business or simply
|
|
|
+decides to stop offering its service.
|
|
|
+Similarly, putting data in a federated system, as has been proposed by the Mastodon developers,
|
|
|
+also puts the user at risk of loosing their data if the operator of the server they use decides to
|
|
|
+shut it down.
|
|
|
+To have real control the user must host the data themselves.
|
|
|
+Then they decide how its encrypted, how its served, and to whom.
|
|
|
+
|
|
|
+Let us explore how Blocktree can be used to build a social media platform which provides this
|
|
|
+control.
|
|
|
+To participate in this network each user will need to setup their own domain by generating new root
|
|
|
+credentials
|
|
|
+and provisioning at least one runtime to host the social media service.
|
|
|
+A technical user could do this on their own hardware by reading the Blocktree documentation,
|
|
|
+but a non-technical user might choose to purchase a new router with Blocktree pre-installed.
|
|
|
+By connecting this router directly to their WAN,
|
|
|
+the user ensures that the services running on it will always have direct internet access.
|
|
|
+The user can access the \texttt{btconsole} web GUI via the router's WiFi interface to generate their
|
|
|
+root credentials and provision new runtimes on their network.
|
|
|
+
|
|
|
+A basic function of any social network is keeping track of a user's contacts.
|
|
|
+This would be handled by maintaining the contacts as files in a well-known directory in the user's
|
|
|
+domain.
|
|
|
+Each file in the directory would be named using the user defined nickname for the contact
|
|
|
+and its contents would include the root principal of the contact as well as any additional user
|
|
|
+defined attributes,
|
|
|
+such as address or telephone number.
|
|
|
+The root principal would be used to discover runtimes controlled by the contact
|
|
|
+so that messages can be sent to the social media service running in them.
|
|
|
+When a user adds a new contact,
|
|
|
+a connection message would be sent to it,
|
|
|
+which the contact could choose to accept or reject.
|
|
|
+If accepted,
|
|
|
+the contact would create an entry in its contacts directory for the user.
|
|
|
+The contact's social media service would then accept future direct messages from the user.
|
|
|
+When the user sends a direct message to the contact,
|
|
|
+its runtime discovers runtimes controlled by the contact and delivers the message.
|
|
|
+Once delivered the contact's social media service stores the message in a directory for the user's
|
|
|
+correspondence,
|
|
|
+sort of like an mbox directory but where messages are sorted into directories based on sender
|
|
|
+instead of receiver.
|
|
|
+
|
|
|
+Note that this procedure only works if a contact's root principal can be resolved using the
|
|
|
+search domain configured in the user's runtime.
|
|
|
+We can ensure this is the case by configuring the runtime to use a search domain that operates
|
|
|
+a Dynamic DNS (DDNS) service
|
|
|
+and by arranging with this service to create the correct records to resolve the root principal.
|
|
|
+The author intends to operate such a service to facilitate the use of Blocktree by home users,
|
|
|
+but a more long-term solution is to implement a blockchain for resolving root principals.
|
|
|
+Only then would the system be fully decentralized.
|
|
|
+
|
|
|
+Making public posts is accomplished by creating files in a directory with the HTML contents of the
|
|
|
+post.
|
|
|
+This file, the directory containing it, and all parents of it,
|
|
|
+would be configured to allow others to read, and in the case of directories, execute them.
|
|
|
+At least one runtime with the filesystem service registered would need to have the execute
|
|
|
+permission granted to others to allow anyone to access these files.
|
|
|
+When someone wanted to view the posts of another user,
|
|
|
+they would use the filesystem service to read these files from the well-known posts directory.
|
|
|
+
|
|
|
+Of course user's would not be using a file manager to interact with this social network,
|
|
|
+they would use their browsers as they do now.
|
|
|
+This web interface would be served by the social media service in their domain.
|
|
|
+A normal user who has a Blocktree enabled router would just type in a special hostname into their
|
|
|
+browser to open this interface.
|
|
|
+Because the router provides DNS services to their network,
|
|
|
+it can generate the appropriate records to ensure this name resolves to the address where the social
|
|
|
+media service is listening.
|
|
|
+The social media service would be responsible for sending message to other user's domains to
|
|
|
+get their posts,
|
|
|
+and to read the filesystem to display the user's direct messages.
|
|
|
+All this file data would be used to populate the web interface.
|
|
|
+It is not hard to see how the same system could be used to serve any type of media: text, images,
|
|
|
+video, immersive 3D worlds.
|
|
|
+All of these can be stored in files in the filesystem,
|
|
|
+and so all of them are accessible to Blocktree actors.
|
|
|
+
|
|
|
+One issue that must be addressed with this design is how it will scale to a large number of users
|
|
|
+accessing data at once.
|
|
|
+In other words,
|
|
|
+what happens if the user goes viral?
|
|
|
+Currently, the way to solve this would be to add more computers to the user's network which run
|
|
|
+the sector and filesystem services.
|
|
|
+This is not ideal as it means the user would need to buy more hardware to serve their dank memes.
|
|
|
+A better solution would be implement peer-to-peer distribution of sector data in the filesystem
|
|
|
+service.
|
|
|
+This would reduce the load on the user's computers and allow their follows to share the posted
|
|
|
+data with each other.
|
|
|
+This work is planned as a future enhancement.
|
|
|
+
|
|
|
+\subsection{A smart lock.}
|
|
|
+The access control language provided by Blocktree's filesystem can be used for more than just
|
|
|
+authorizing access to data.
|
|
|
+To illustrate this point,
|
|
|
+consider a smart lock installed on the front door of a company's office building.
|
|
|
+When the company first got the lock they used NFC to configure the lock
|
|
|
+and connect it to their WiFi network.
|
|
|
+The lock then used link-local runtime discovery to perform automatic provisioning.
|
|
|
+An IT administrator accessed \texttt{btconsole} to approve the provisioning request
|
|
|
+and position the lock in a specific directory in the company's domain.
|
|
|
+Permission to actuate the lock is granted if a principal has execute permission on the lock's file.
|
|
|
+To verify the physical presence of an employee,
|
|
|
+NFC is used for the authentication handshake.
|
|
|
+When an employee presses their NFC device, for instance their phone, to the lock,
|
|
|
+it generates a nonce and transmits it to the device.
|
|
|
+The device then signs the nonce using the credentials it used during provisioning in the company's
|
|
|
+domain.
|
|
|
+It transmits this signature to the lock along with the path to the principal's file in the domain.
|
|
|
+The lock then reads this file to obtain the principal's authorization attributes and its public key.
|
|
|
+It uses the public key to validate the signature presented by the device.
|
|
|
+If this is successful,
|
|
|
+it then checks the authorization attributes of the principal against the authorization attributes on
|
|
|
+its own file.
|
|
|
+If execute permissions are granted, the lock actuates, allowing the employee access.
|
|
|
+The administrators of the company's domain create a group specifically for controlling physical
|
|
|
+access to the building.
|
|
|
+All employees with physical access permission are added to this group,
|
|
|
+and the group is granted execute permission on the lock,
|
|
|
+rather than individual users.
|
|
|
+
|
|
|
+\subsection{A traditional three-tier web application.}
|
|
|
+While it is hoped that Blocktree will enable interesting and novel applications,
|
|
|
+it can also be used to build the kind of web applications that are common today.
|
|
|
+Suppose that we wish to build a three-tier web application.
|
|
|
+Let us explore how Blocktree could help.
|
|
|
+
|
|
|
+First, let us consider which database to use.
|
|
|
+It would be desirable to use a traditional SQL database,
|
|
|
+preferably one which is open source and not owned by a large corporation with dubious motivations.
|
|
|
+These constraints lead us to choose Postgres,
|
|
|
+but Postgres was not designed to run on Blocktree.
|
|
|
+However, Postgres does have a container image available on docker hub,
|
|
|
+we can create a service to run this container image in our domain.
|
|
|
+But Postgres stores all of its data in the local filesystem of the machine it runs on.
|
|
|
+How can we ensure this does not become a single point of failure?
|
|
|
+First, we should create a directory in our domain to hold the Postgres cluster.
|
|
|
+Then we should procure at least three servers for our storage cluster
|
|
|
+and provision runtimes hosted on each of them in this directory.
|
|
|
+The sector service is registered on each of the runtimes,
|
|
|
+so all the data stored in the directory will be replicated on each of the server.
|
|
|
+Now, the Postgres service should be register in one and only one of these runtimes,
|
|
|
+as Postgres requires exclusive access to its database cluster.
|
|
|
+We now have to decide how other parts of the system are going to communicate with Postgres.
|
|
|
+We could have the Postgres service setup port forwarding for the container,
|
|
|
+so that ordinary network connection can be used to talk to it.
|
|
|
+But we will have to setup TLS if we want this to be secure.
|
|
|
+The alternative is to use Blocktree as a VPN and proxy network communications in messages.
|
|
|
+This is accomplished by registering a proxy service in the same runtime as the Postgres service
|
|
|
+and configuring it to allow traffic it receives to pass to the Postgres container on TCP port 5432.
|
|
|
+
|
|
|
+In a separate directory,
|
|
|
+a collection runtimes are provisioned which will host the webapp service.
|
|
|
+This service will use axum to serve the static assets to our site,
|
|
|
+including the Wasm modules which make up our frontend,
|
|
|
+as well as our site's backend.
|
|
|
+In order to do this,
|
|
|
+it will need to connect to the Postgres database.
|
|
|
+This is accomplished by registering the proxy service in each of the runtimes hosting the
|
|
|
+webapp service.
|
|
|
+The proxy service is configured to listen on TCP 127.0.0.1:5432 and forwards all traffic
|
|
|
+to the proxy service in the Postgres directory.
|
|
|
+The webapp can then use the \texttt{tokio-postgres} crate to establish a TCP connection to
|
|
|
+127.0.0.1:5432
|
|
|
+and it will end up talking to the containerized Postgres instance.
|
|
|
+
|
|
|
+Although the data in our database is stored redundantly,
|
|
|
+we do still have a single point of failure in our system,
|
|
|
+namely the Postgres container.
|
|
|
+To handle this we can implement a failover service.
|
|
|
+It will work by calling the Postgres service with heartbeat messages.
|
|
|
+If too many of these timeout,
|
|
|
+we assume the service is dead and start a new instance one of the other runtimes in the Postgres
|
|
|
+directory.
|
|
|
+This new instance will have access to all the same data the old,
|
|
|
+including its journal file.
|
|
|
+Assuming it can complete any in progress transactions,
|
|
|
+the new service will come up after a brief delay
|
|
|
+and the system will recover.
|
|
|
|
|
|
\subsection{A realtime geo-spacial environment.}
|
|
|
% Explain my vision of the metaverse.
|