Foresight Ventures: Arweave explained: the 🐘 in the room of Web3.0

Peter Li, msfew@Foresight Ventures

Arweave

1. Overview

Arweave is a decentralized network focused on storing data permanently. Compared to Bitcoin’s global ledger and Ether’s global computer positioning, Arweave is similar to a global hard drive that never loses data. In a public chain of storage, Arweave has the potential to become the “Layer 0” of Web 3.0, existing as a critical infrastructure.

2. Why store data permanently important?

Pre-Web

Before the advent of the Web, data was usually recorded in books such as history books, and libraries were the storage centers of today. In ancient times, the Library of Alexandria in Egypt was the source or database of knowledge at the time, containing hundreds of thousands of valuable manuscript books. But after wars, all the books accumulated during the 600-year history of the Library of Alexandria were burned. All these precious data were destroyed. Because of the destruction of this data, the course of human history may have been set back by decades.

Web 1.0 — Web 2.0

In the Web 2 era, there is a site similar to Arweave: Internet Archive (https://archive.org). It keeps snapshots of almost all websites from all periods (for example, we can see the Apple’s website from 2001), as well as music, books and other data. It is the permanent storage of the Web2 era, preserving all the precious data of websites and media, which can be seen in its original form after the rapid changes of the Internet. But it is run by centralized databases and servers. At the time of this writing, the Internet Archive’s website snapshot service is inaccessible, so its service is rather unstable. At the same time, the organization behind the Internet Archive has to pay a lot to maintain the servers and databases, which is another unstable factor. Also, although the Internet Archive is an open source non-profit organization, the data stored on their servers is still theirs, and there is the problem of centralization. Because of these factors, the Internet Archive may lose data under extreme conditions, even though it certainly does have disaster recovery and backups.

Web 3.0

In the Web 3.0 era, Arweave emerged as a perfect solution. A unique consensus mechanism guarantees permanent data storage. The token economy and mining mechanism allow miners to store data and the developer ecosystem to be expanded, solving the problem of the huge cost of server and development funding. The decentralized architecture ensures the data ownership of users. Arweave also continues to actively cooperate with the Internet Archive to permanently preserve data on the chain. Arweave is the Library of Alexandria for permanent data storage in the Web3 era.

3. Data Structures and Consensus Mechanisms

Arweave enables permanent file storage through the Blockweave data structure and the SPoRA consensus mechanism.

a. Blockweave

Arweave is different from the block structure of traditional blockchains. A traditional blockchain has a chain structure where blocks are continuously connected to each other. Arweave’s block, on top of that, also points to a random previous recall block (previously called recall block, now called recall chunk), so from a three-dimensional perspective it is a spun-like structure. From the point of view of storage content, the conventional on-chain storage is mainly indexed and data is stored in distributed nodes, while Blockweave’s block stores data information as transactions, which can be seen in Arweave’s blockchain browser exists in types like image type. Blockweave is actually sharded, and the performance is extremely scalable. Compared to other storage chains, Arweave can increase TPS by many times and significantly reduce long-term storage costs.

Blockweave is mainly designed to serve Arweave’s consensus mechanism above all else.

b. SPoRA Consensus Mechanism

Before we explain the SPoRA consensus mechanism that Arweave is using now, we can first understand the PoA + PoW mechanism that it used before. The PoW mechanism is already familiar to us, and we have seen the classic PoW mechanism on many blockchains. So let’s focus on the PoA mechanism.

PoA stands for Proof of Access. If a miner wants to generate a new block, then he must access a randomly recall block. Ideally, if a miner wants to satisfy the conditions of this PoA mechanism, he would choose to store the data of all blocks, because each next block would point to the current block and a random block before it. The PoA incentivizes miners to store more Arweave blocks, and it also encourages miners to store blocks that are not stored by others to increase the probability of satisfying the PoA. Of course, PoA alone is not feasible, because if everyone stores all the data, then everyone’s probability of getting a block is 1. So there is a PoW mechanism in addition to PoA to ensure decentralization. With PoA and PoW consensus, a miner’s block probability = the probability of having a randomly recalled block * the probability of being the first to find the hash. This unique design of PoA incentivizes miners to store more data and to store rare data, enabling permanent data storage.

In addition, in order to regulate the block generation rate in the network, the PoW algorithm allows to adjust the difficulty, the higher the PoW difficulty, the longer the computation period, if the block generation rate in the network exceeds the target rate, it increases the difficulty of the PoW problem for future block generation; similarly, as the block generation rate in the network decreases, the difficulty setting will be adjusted downward. In this way, the network is able to regulate the block generation rate without without being limited by the number of nodes and computing power in the network.

Based on the consensus mechanism of PoA, Arweave upgraded PoA to SPoRA (Succinct Proofs of Random Access) in this year. The PoA + PoW mechanism alone only achieves the goal of permanent data storage, but does not incentivize data access speed, so miners are likely to place the database in other regions where it is cheaper, rather than locally. This can lead to slower network access due to distance and other factors. The SPoRA mechanism reduces the weight of miners’ block probability and adds a consideration for data access speed. The reduced weight of PoW in block probability reduces the wasted energy in finding hash, reduces energy consumption and is more friendly to the environment. Upgrading PoA to SPoRA allows miners to focus more on maintaining their own local hardware and nodes, avoiding miners storing data in the same low-cost data center, diversifying the geographic location of nodes, and making it more decentralized.

Through these uniquely designed consensus mechanisms, Arweave guarantees permanent data storage, access speed and block generation rate.

c. Decentralized Storage

Another great feature of Arweave is that the miner’s experience is good. Compared to traditional blockchains, Arweave does not require miners to download data from all nodes of the blockchain, but only a few blocks to start mining (although the probability of success is low at this point). This is actually a very important point. If a storage chain is designed without this, it is particularly difficult to join the network as new node.

By the design of blockweave and consensus mechanism, miners will spontaneously download rare data, which will ensure that the average number of data stored remains uniform. The first row of the diagram below shows each node of a traditional blockchain. Each node needs to download all the data, which wastes a lot of resources. The second row shows the situation before miners spontaneously download rare data, which means data may be missed by miners, leading to possible data loss and overstorage of some data. The third line shows the current situation of Arweave, which represents the equilibrium reached by miners after spontaneously downloading rare data. Every copy of the data has the same number of copies, which avoids wasting resources while maximizing data storage security.

In addition, Arweave miners can focus more on upgrading their hard drive devices with the consensus of SPoRA. Also, in future releases, Arweave will continue to lower CPU requirements and focus on making data storage and read/write experience even better.

4. Network and Application Architecture

Arweave’s DApp does not run directly on the Arweave blockchain. Arweave is just a blockchain like a hard disk or a database. On top of the Arweave blockchain network layer, there is a permaweb (permanent web) layer. The permaweb layer runs Arweave’s decentralized applications. The DApp smart contracts in Ethereum are all running in the Ethereum EVM, which runs directly in the Ethereum network. The blockchain structure of Arweave and the positioning of the storage are very similar to the paper tape of Turing machines. Developers can manage their own application state in code, and use Arweave to record that state on the chain.

Arweave’s architecture gives developers a high degree of freedom. Just like in the Web 2.0 era, it doesn’t matter if the backend of an application uses PHP or Java, any developer who is familiar with Arweave’s HTTP API (which has almost zero learning cost for developers) can use Arweave as a blockchain database in a storage-based consensus paradigm to develop their own decentralized applications. The storage-based consensus paradigm (SCP) is a smart contract design paradigm that brings unlimited potential to Arweave.

The problem with this development flexibility is that Arweave cannot directly specify the contract and token standards for DApps since they do not run on Arweave. We will discuss Arweave’s smart contracts in a later section.

DApp = Front-end Page = Smart Contract

• Definition of Smart Contracts
• When Nick Szabos coined the term “smart contract” in 1996, he defined that a smart contract should have the following four characteristics: a set of commitments; be specified in digital form, not on paper; contain an agreement; and involve the fulfillment of commitments by the people involved in the contract. In addition, I believe that smart contracts need to be data-store-trustworthy and computationally-trustworthy.
• Traditional DApp vs Arweave DApp
• A traditional DApp usually consists of a front-end of the application (web page) and a smart contract. The front-end of the application transmits the user’s data to the smart contract. Such a DApp fulfills the four conditions and two characteristics of a smart contract. The developers of Arweave think about Arweave’s DApp and smart contracts in this way: The front-end page is the smart contract and vice versa. The idea is that a DApp doesn’t need to have both a front page and a smart contract on the chain, but just a front page and a smart contract in one.
• How front page and smart contract become one?
• In the previous article, we mentioned that Arweave can store different content-type files directly. We can store the front-end page directly into Arweave, and open the transaction data to see the complete page directly! (For example, the data stored in this transaction. Click on it and it is directly a rendered front-end page). From this example, we can actually understand why the front-end page can be a smart contract. Because when we open the Arweave data, the browser also renders the page for us. Not only can we see the source code of the page, but we can also see the functioning page. This can’t be done on Ethereum, where the smart contract is just a string of text to the browser, and the browser can’t render the page. When the developer is developing, the logic of DApp and the logic of contract can be both written in the front-end, and then the transactions and data that need to be stored will be stored on Arweave through HTTP API.
• How to ensure the data storage and computation is trustworthy?
• The design style that Arweave’s developers and Sam Williams prefer is storage-based consensus paradigm (first proposed by everFinance Founder outprog). The source code of the contract pages is stored on Arweave, deterministically, and can be viewed at will. The data for each interaction is also stored on the chain, which ensures that the data is stored with trust. While we render the contract pages, modern browsers provide us with an almost completely uniform execution environment, which ensures that the computation is trustworthy. By doing so, Arweave’s smart contracts also guarantee data perpetuation/data immutability/data traceability/low data cost.
• How can the contract code be upgraded?
• When upgrading a contract or front-end page layouts, the developer simply republishes a new version of the contract to Arweave, which generates a new Arweave transaction, updates the front-end page, and updates the contract content.

Implementations of Smart Contracts

1. Smartweave (Arweave acts as a paper tape for Turing machines, Smartweave is dedicated to managing the state of Turing machines)

• Overview:
• Remember the Turing machine paper tape concept we mentioned in the previous article? To build a Turing machine, it’s not enough to have Arweave’s paper tape to record state, we need a way to effectively manage and change that state of the application. What Smartweave does is an important missing piece of the puzzle. Smartweave is the official Arweave standard for smart contracts, and uses the same concept of “front-end code = smart contract”, where developers can use JavaScript, a front-end scripting language, to manage state. Front-end pages are basically the logic and state of various UI’s, so I believe it is much easier for front-end developers to use Smartweave to write smart contracts to manage state than Solidity.
• Design concept and features:
• Smartweave is designed with Lazy-execution philosophy, only executing when it has to. When the user has to get the latest contract status on the front end, it will be executed. Since the language used is JavaScript, the user can run all transactions directly in the browser, without the need to run them on the chain side. Smartweave stores the initial state and the transactions associated with the contract (an ordered list in the form of [transaction1, transaction2, transaction3]). To get the latest state of a contract, the user runs all contract-related transactions in the browser, as if the user were a node itself, and finally gets the latest state and submits a new transaction.
• Example:
• CommunityXYZ is the dashboard and management platform for the Arweave profit sharing community. CommunityXYZ’s smart contract is a standard implementation of Smartweave. It uses TypeScript (an extended version of JavaScript). It contains information about the user’s balance, vault, roles, votes, etc. Through a JSON-like data structure, the contract also implements the necessary functions such as transfer, increaseVault, and check balance. The SmartWeave contract implemented by CommunityXYZ can be placed in the front-end page code base, so that the contract logic can be stored in the front-end.
• Disadvantages and optimization:
• If you read the design of Smartweave carefully, you will see that the TPS cap becomes the cap of user browser performance and network bandwidth. However, leaving all transactions to the user raises the question of whether a contract with a large number of transactions will take a long time to run once the latest state is available, causing too much blocking.
• Although in reality, a normal but large number of transactions does not cause a lot of user experience problems (the JavaScript file size for loading and running in modern web pages is actually very large itself, for example, the JavaScript code of the official website notion.so that I am using is about 2MB but it can be opened in seconds), but if there is a DDoS attack and someone uploads a lot of malicious transaction data (e.g. several GB), then there will definitely be a problem. To solve this problem, we can filter the transaction data through a separate Smartweave contract before submitting the transaction, receive a large amount of transaction data, and then batch rollup to AR.
• Also, developers can refer to the Smartweave contract implemented by Redstone. Redstone reduces the unnecessary reloading and running of transaction data through the multi-layer of browser cache, which can greatly optimize the performance of the contract on the client side. Alternatively, a trusted user can generate a snapshot of the transaction state to reduce the transaction computation.

2. everFinance’s token-demo (also based on storage-based consensus paradigm)

• The token-demo implemented by everFinance is based on the same idea as Smartweave, which is also based on the idea of Arweave serves as a paper belt of Turing machines. The token-demo consists of two application modules: issuer (token issuance operator) and detector (token checker). The token issuer, which provides the web page of Token, provides an interface to check balance, transaction history, and submit tokens for transaction. The token checker, when running, automatically loads the data on Arweave (just like the Lazy-execution we described in the Smartweave section) and provides an interface to query token transactions.

In theory, Arweave’s smart contracts can be implemented in a myriad of different forms, so each project may have its own implementation, but most projects use Smartweave. Arweave actually has a token standard like ERC-20, which specifies various APIs. On Arweave, Smartweave’s smart contracts greatly simplify the development process, and if Arweave is used for data storage, project teams may not need a dedicated contract engineer at all.

5. PSC (Profit Sharing Community) explained

PSC Overview

PSCs are profit-sharing communities on Arweave, a new and more equitable structure for startup projects. These communities give founders more active control and flexibility over their projects, while giving contributors more power. In traditional startups, employee-owned equity is less likely to be realized into cash quickly, is not granted governance rights, and receives little to no dividends. At the same time, PSCs combine DAO and tokens. The voting power of a PSC depends on how much community PST a person holds, and how long they hold those tokens.

In contrast, all PSC contributors earn PST tokens, which give them fair governance rights, instant liquidity, and interest generated by using the application. See more PSC projects on the Profit Sharing Community Hub at communityXYZ.

PST (Profit Sharing Token) Overview

PST is an additional mechanism, Arweave allows developers to attach a PST attribute to the protocol tokens when creating the protocol. When users interact with the application on Arweave to generate gas fees, the holder of the PST attribute tokens will receive a rebate of AR tokens from the fees in the proportion agreed upon when creating the protocol. The return will be distributed randomly in proportion to the PST holders’ relative PST holdings, with a higher chance of receiving the distribution if the number of PST tokens held is larger (similar to PoS). This is equivalent to Arweave incentivizing projects with a fee rebate, which is an important source of income for developers and a bonus for the holders in addition to the increase in the price of the tokens during the development phase of the project.

PST Development

You may wonder why such tokens are not designed and standardized by Arweave? Is it safe to let the project set up? In fact, PST on Arweave has a standard like ERC20, there are some general interfaces. But for applications, there are no restrictions, and developers can flexibly use the advantages of Smartweave and PST, so Arweave does not enforce standards. With regard to setting up the project in a decentralized way, as we mentioned in our previous article, Arweave developers generally believe that the front end of the project can be a smart contract. So if such profit sharing is written in the contract, then it’s actually quite safe and trust-worthy. Users can check whether the tokens are correctly allocated, ensuring that Code is Law.

PST be on CEX?

From a technical point of view, PSTs are eligible for centralized exchanges, but the main constraint for not seeing them yet comes from the following two points.

One is that the Arweave ecosystem as a whole is in an early state, and the liquidity of PST tokens and the activity of PSCs is in early stages. In Verto, the decentralized exchange for PST, only one transaction of all PST tokens took place this month (Note: The main reason for this is that Verto is undergoing a code refactoring, so there has been no trading for the last three months. Other than that, the demand for PST tokens is actually very high. Sorry for being misleading). And Evermore, for example, has not changed in price since its launch in April, reflecting the poor liquidity. But if it is listed on an exchange, then the liquidity problem may be solved.

Second is the technical limitation of PST tokens. The biggest value of PST tokens is that holding PST allows you to receive tips from the application. However, PST revenue distribution is calculated based on the address’s holding of PST, which means that if PST tokens are transferred to an exchange, the user cannot receive PST revenue without a targeted adaptation from the CEX. To solve this problem, the exchange needs to calculate the user’s share and then distribute the corresponding profit to the user (similar to a mining pool). In addition, since different projects may have different smart contract implementations (e.g. Smartweave and token-demo), the token implementations may also be different, so the exchange may have to make different adjustments and adaptations (similar to how applications connected to APIs in the Web 2.0 era).

6. Current Storage Status

Node Distribution

source: https://viewblock.io/arweave/nodes

The current number of nodes is 1019, including 868 nodes in mainland China and 103 nodes in the United States, and the rest are scattered in various countries. The current number of nodes is still small and the storage limit of a single node is not high enough, but compared with 150 nodes in April this year, it has made a great leap.

Total Storage Volume

source：https://viewblock.io/arweave/nodes

The explosion of NFT boosted Arweave’s storage, followed by more blockchains backing up data, and the total storage volume quickly climbed to 27.5TiB, which is more than 500,000 times different from Filecoin’s storage volume of 13.5EiB, but Filecoin stores a low percentage of valid information due to its incentive mechanism, while Arweave can upload 43.88GB of data in a single block.

Storage Price

Arweave has a low cost of consensus, with permanent storage of 1GB of data (which can handle 10 million transactions) costing only about $20 USD. In comparison, Filecoin’s user storage cost is about $0.06 for 1GB, but the pricing model offered by Filecoin is calculated on annual fee, with no guarantee of perpetual storage, and Arweave is a buyout, with at least 200 years of guaranteed storage for a single fee. So Arweave has some reasonable premium on data storage.

To ensure that the storage fees provided to miners are competitive in the long run, Arweave deposits most of the transaction fees currently paid by users into an endowment fund, which will be gradually released to miners as its ecosystem grows.

7. Arweave Ecosystem

Ecosystem Overview

ArDrive

Website

Google Drive is the storage client of Google Cloud, and the corresponding ArDrive is the storage client of Arweave. ArDrive is a set of applications for both PC and mobile that can upload various forms of data such as text, images, videos, etc. to Arweave for permanent storage. Currently, ArDrive is based on a one-time fee for permanent use, and the storage price is about $20 for 1G data, which is slightly higher than the current traditional storage price, but the value of permanent storage and data security is much better than the traditional storage method.

It is not just for the cryptocurrency community, but for storing, all the good things that deserve to be recorded forever.

KYVE

Website

The KYVE network can be seen as the middle layer of Arweave’s cross-chain storage. Through KYVE, data from other blockchains can be permanently stored on Arweave as a backup, and different blockchains can interact with each other through the KYVE network. In addition, KYVE guarantees the accuracy of uploaded data by means of a storage pool, where the uploader is responsible for storing the latest data and the verifier ensures the accuracy of the uploaded data, and both parties pledge KYVE tokens to the storage pool to ensure accuracy and fairness.

Currently, we can see from KYVE’s website that the public chain integration will cover Avalanche, Cosmos, Near, Polkadot and Solana.

everPay

Website

everPay is a DApp for cross-chain transfer, currently it has realized real time token transfer on Ethereum, and Arweave, users can top up assets on Ethereum or Arweave to Everpay and take out assets from the other end arbitrarily by mapping, and there is no waiting and no gas fee. Therefore, everPay is more like a Layer2 application, which guarantees speed through off-chain computing and security by rolling up data to Arweave, and becomes a channel to connect Arweave ecosystem with other blockchains.

In the future, everPay will host and integrate more blockchains and support smart contract accounts to provide a seamless transfer experience for more investors who want to participate in Arweave’s development dividends.

Verto

Website

As mentioned above, PST tokens are not yet available on centralized exchanges, and Verto is a decentralized exchange for PST tokens, facing the profit sharing mechanism of PST tokens. The more important role of Verto is to serve as a revenue swap tool, both through Verto to hold popular projects’ token to get higher fee interest. In addition, its token, VRT, receives a 0.5% commission on all PST transaction fees. After Verto completed its Ethereum bridge in February this year, it allowed users to exchange ETH through Verto while allowing VRT holders to earn Ethereum-denominated transaction fees.

Although the current number of transactions in Verto is very small, with the increasing economic effect of Arweave, the promotion of PST type tokens and the value capture of VRT, Verto is bound to have its highlight moment.

ArConnect

Website

If you want to use Arweave, you can’t beat ArConnect. As the first browser plug-in wallet in the Arweave ecosystem, it puts an end to the awkwardness of having to paste keys into the app page when using the DApp. It is now deeply tied to the Arweave ecosystem and is the official and main wallet for many protocols, but it is worth noting that there are currently no fees for in-program AR and PST transfers, and that a portion of the fees generated when interacting with the DApp are distributed to randomly selected VRT token holders.

Pianity

Website · Twitter · Discord · Telegram

As the Arweave ecosystem as a whole is in its early stages, the kingpin of it, apart from the infrastructure protocols, should be Pianity. This project is a music NFT platform that proposes a model that provides collectors with a certain annualized revenue. The revenue is then distributed to each NFT holder according to the ratio of the value of the NFT purchased by the user to the total value of all NFTs. Since the ratio is calculated based on the price of NFT at the time of purchase, it may lead users to deliberately raise the price of NFT in order to obtain a higher allocation ratio, which in turn stimulates the market to gradually increase the distributable revenue and attracts traffic attention. In addition, its token PIA also belongs to the PST type, so users will also get rewards based on Pianity’s traffic in addition to the NFT revenue they can get.

In the relatively blue ocean market of music NFT, Pianity has the opportunity to stand out through its perpetual storage and dual revenue model.

Does Arweave require smart contracts and token standards?

Before I really got to know Arweave, I always thought that smart contracts and token standards were necessary for a public chain’s ecosystem to thrive. But Arweave, being a more underlying chain, with DApps that don’t run inside the chain, certainly can’t restrict developers in a specific secure sandbox environment like Ethereum. However, the Arweave developers’ ideas of “front-end = smart contract”, “storage-based consensus paradigm”, and “Arweave as a Turing machine paper belt” are slowly gaining consensus among developers, so more and more project ecosystems are using Smartweave, implementing the “front-end = smart contract” design idea.

8. In summary: Arweave’s true value will explode in Web 3.0.

Let’s start with a list of Arweave’s advantages. Arweave’s original consensus and miner incentives guarantee the permanence of storage. Preserving data permanently has been a major issue in human history and is of great importance to humanity. Arweave can serve as the Library of Alexandria in the Web 3.0 era, preserving data and continuing to accumulate human wisdom. The decentralized nature of Arweave can disrupt the monopoly of centralized cloud service platforms on the storage market, allowing users to avoid censorship of their data, while also allowing data to be stored in the cloud, bringing freedom of data distribution. Arweave’s storage is positioned with the ability to serve as “Layer0”, an indispensable part of blockchain development, and is versatile and extremely easy to use, making it an ideal breeding ground for all kinds of decentralized applications. Next, let’s look at the disadvantages of Arweave. Arweave’s positioning is too low-level to limit the standard of tokens and contracts due to its architecture, and the development ecosystem is too flexible and may not be unified, making the ecosystem sometimes uncooperative.

We may not understand Arweave for now, but we didn’t understand Bitcoin in the first place. Charlie Munger said: “The human brain is reluctant to change in order to save computing space. It’s a form of avoiding inconsistency.” We can’t look at Arweave through the lens of a normal blockchain, because that misses a lot of the important value of Arweave. Web 3.0 is likely to arrive within 2 years, and by then a blockchain infrastructure like Arweave will have demonstrated its merits and be understood by all.

References:

• Arweave technical details:
• https://www.chainnews.com/articles/543779897127.htm
• https://www.youtube.com/watch?v=YO3zyQvL3Yg
• https://esl-conf-static.s3.eu-central-1.amazonaws.com/media/files/000/000/841/original/Sam_Williams_-_Blockchains__blockweaves__and_the_decentralised_revolution.pdf?1513004189
• https://arweave.medium.com/the-arweave-network-is-now-running-succinct-random-proofs-of-access-spora-e2732cbcbb46
• Arweave storage:
• https://news.ycombinator.com/item?id=24791621
• Arweave ecosystem:
• https://arweave.medium.com/nfts-and-attention-rewards-verto-and-koi-spotlight-c7fa1ca94827
• https://arweave.news/profit-sharing-communities-a-deep-dive-by-arweave/
• https://www.verto.exchange
• https://arweave.medium.com/profit-sharing-tokens-a-new-incentivization-mechanism-for-an-open-web-1f2532411d6e
• https://coinmarketcap.com/alexandria/article/profit-sharing-communities-a-deep-dive-by-arweave
• Arweave true value:
• https://www.chainnews.com/articles/124011762292.htm
• https://www.chainnews.com/articles/930028948285.htm