10T distributed storage

In the 5G times, our demand for video quality is increasing from SD to HD, to 4k and Blu-ray, bringing with it increasing pressure on data storage. Especially with the advent of the 5G Internet of Everything times, various smart devices and sensors will generate massive amounts of data. According to IDC, the human data demand in 2025 will be 175ZB (1ZB = 1040GB).

In 2022, some flagship phones have already been launched with 1TB hardware storage, such as VERTU phones. 1TB of mobile phone storage capacity was not launched by mobile phone manufacturers to sell more expensive individual products at a higher selling price, thus boosting high selling prices and high profits, but rather there is a real and growing demand from users for this large capacity.

The power of mobile phones now means that more than half of the office no longer needs a computer to operate. Although some specialised software is limited and cannot be used to 100% effect on the small screen of a mobile phone, it can still do the job, so there is a greater demand for the capacity of the mobile phone's hard drive, as many office documents such as PPTs need to be stored, and chat logs and videos need to be saved for a long time in chat tools.

In recent years, the high-speed iteration of the mobile phone's image system, every once in a while there is a stronger photo performance of the new machine released, with RAW mode shooting photos gradually used in mobile devices, resulting in a photo in the past 2, 3M occupies space, into 20, 30M or even higher, photos a lot, then occupy the space is simply not enough to use. Most mobile phones now also support 4K video recording, and 8K video recording phones may be born by the end of the year, when a video will be dozens or even more than a hundred G. Therefore, the 1TB capacity for shooting video footage with mobile phones may only be the minimum threshold. However, the higher the storage capacity, the higher the price of the phone. The latest 1TB Apple phone, priced at RMB 13,000, is even more expensive than most computers.

Of course, there are also many mobile phone manufacturers offering cloud storage solutions that push large files, photos, videos and installation files that take up space on the phone to the cloud. Users' need for local storage will be significantly reduced, helping to solve the problem of insufficient memory on the phone. Users can conveniently access their data anytime, anywhere, from any internet-enabled device connected to the cloud. Cloud storage space is where data is stored, both large and small, and just like a normal hard drive, the size of cloud storage space can be chosen according to the user's needs.

A typical example is Apple's iCloud Photos, which automatically uploads photos and videos to the cloud and downloads them when you want to see them, thus saving storage space. However, as anyone who has used iCloud Photos will have noticed, iCloud only starts downloading these images when you want to play them back, and you still have to wait before you can view them.

Although cloud storage can solve the problem of insufficient storage on the user's body to a certain extent, cloud storage itself requires a network connection to use, for example, photos can be uploaded to the cloud storage space, but if the network conditions are not good, it will take longer to upload/download a photo and the experience is far less convenient than local storage. In places where there is no internet access at all, cloud storage can be considered a non-starter.

Secondly, in order to protect the privacy of the user, the cloud storage has added a more complex encryption algorithm, the reality that cannot be ignored is that it will make the cloud platform consume a lot of resources, which will make the whole cloud platform work much less efficient and the cost will naturally increase; however, if the data is encrypted with a simpler encryption algorithm, the data stored in the cloud platform or in the process of processing may be The cost increases. And although the price of cloud storage may seem low, you need to pay storage fees every month, and the same as traffic, the capacity and price of the cloud storage provided is slowly becoming unreasonable, and the price of buying 100G of cloud storage a year is around 300 yuan, and in the times of 1TB storage 100G capacity is obviously not enough.

And it's not just photos and videos that take up space, but all sorts of app files too. Imagine that there are around 200-300 million King's Quest users in China, and King's Quest has around 2GB of installed files, which adds up to 400,000-600,000 TB of storage space. The current Cloud-Original God, for example, is simply a live streaming video of the game images back to our mobile phones, which send operating instructions to the remote gaming terminal in real time. Meanwhile, the industry is currently looking at ways to reduce the amount of traffic required for cloud gaming and further reduce latency through video compression. The industry also has optimised mobile phone screen casting solutions, where the network transmits drawing instructions that are simply used to control the screen image, rather than data-intensive video, enabling further traffic control and latency reduction. And Google's cloud gaming solution uses a lot of AI algorithms, light models and even separate training for individual games to get higher quality and higher refresh rates.

In theory, decentralised cloud storage has advantages over traditional cloud storage in terms of privacy protection, high security, low cost, scalability, low bandwidth limitation and de-trust; however, in practice, the reliability of large-scale applications has yet to be verified, and it faces many challenges at the technical and regulatory levels.

As a super terminal for the Internet of everything, traditional storage means can no longer meet the effective storage of huge amounts of data.

We believe that decentralised distributed storage is the cornerstone of the Web3 mobile system layer and can be a good solution to many of the problems associated with traditional centralised storage. This is mainly reflected in the following areas.

• Data security:The data is encrypted using encryption algorithms and then stored in a distributed manner, so that only the private key of the data owner can be accessed and downloaded, and no one else can access the data content. In addition, multi-point-in-time snapshot technology allows users to back up each version of data at a certain time, extract samples from multiple points in time at the same time, and be able to restore data at the same time to avoid losses arising from storage problems.
• Massive storage:Free storage space and unused bandwidth can also be shared by ordinary users or enterprises. As more storage resources are shared, more resources are available to users, breaking the limitation of limited centralised storage resources. In addition, the system's storage capacity, computing power and performance are all increased due to the scaling of the cluster servers, allowing it to meet the higher storage requirements of enterprises.
• Storage performance:The storage system is able to manage the read and write cache effectively and is able to grade the storage itself. Write caching technology with cache storage enables overall storage performance to be improved by mapping data in hotspots directly to cache storage, which can better improve system response times.
• Data consistency:the storage system will slice the data when the user stores the data, keeping the data after the slice on the cluster nodes, when one copy is written while the other copies are read, in case of data read failure, the system can read the data from the other copies and continue to write to that copy, ensuring the consistency of the total number of copies, and when the data is in inconsistency for a long time, the system will automatically rebuild and recover, thus minimising the impact on the data owner.

As for the decentralised storage solution for the Web3 mobile operating system, we believe it includes the following main technical requirements.

1. Participant

• Users: users of storage network resources, who join the ecology through an access mechanism and pay a corresponding cost (set by the ecological organisation) to use the corresponding storage network resources.
• Storage Service Provider: The storage service provider that builds the cornerstone of the entire network. You need to apply and be approved before you can join.
• Validation Node: A designated node in the ISS Network is responsible for packing out blocks and performing validation to maintain the security of the entire storage network. The validation node can exist on its own or be a storage service provider.

2. Core technologies

• Distributed hash tables: cutting data into 512KB fragmented files, hashing them and finally forming a file indexed hash table based on content addressing (hash + IP address where the file is stored).
• BitTorrent: A P2P file transfer protocol that enables peer-to-peer file sharing and efficient and fast uploading or downloading of data between different decentralised storage servers.
• GIT versioning protocol: efficient update of uploaded files without re-uploading overwrites.
• Self-verification system: content objects addressed by content, containing a hash of the uploading user's signature, can be verified by a public key to derive the authenticity of the document publication.
• Privacy protection mechanism: Support a variety of mainstream privacy protection and security mechanisms, multi-faceted multi-channel security system, allowing users to balance security and privacy in the actual application.

For the actual situation of the current storage servers, we suggest that a three-step route, first centralised, weakly centralised and finally decentralised, can be adopted for implementation. The data is first saved through a server storage cluster, then the nodes in the storage server that complete the proof of workload, replicate the stored data, and finally broadcast it to all nodes, thus achieving decentralised storage. In other words, the data stored on the storage server must be allowed to be block-packed and uploaded to the decentralised server by a certain number of working nodes for preservation, before the whole storage process is considered complete.

The core idea is to use the private computing architecture of TEE+SE to build a distributed storage server with two layers of storage, compatible with IPFS protocol, integrated with P2P network transmission technology, BitTorrent, asymmetric encryption and decryption technology, privacy protection technology, data fragmentation technology, users can encrypt themselves, who store and who use or authorize the use, more can guarantee data privacy and security.

File upload process:On the server's main network, the user packages the relevant file information for the transaction, signs it and broadcasts it to the storage node. The file is then uploaded to one of the storage nodes via the SDK interface provided by the storage server storage. The node checks the file content Hash, file size and other information accordingly. Once the check is successful, the node broadcasts the transaction information to all other producers, indicating that the file has been saved successfully; the other nodes synchronise the file information via the P2P network and upload proof of successful saving.

Download file process:The user selects one from the server's storage node ranking list, initiates a query/download transaction, waits for the transaction to be packaged successfully, obtains the transaction credentials, and then initiates a query/download request to the node via the credentials. Users can download directly from the secondary storage network according to their needs.

With such a Web3 mobile distributed storage system solution, under the combined TEE+SE architecture, the mobile phone and storage server combine a two-tier storage model that can support unlimited storage and effective storage, provide commercial-grade big data uplink, enable the advantages of sharing economy and reduce the construction cost of centralised storage by data cropping and releasing redundant data. At the same time, such a storage solution has strong privacy protection functions. The data is fragmented and encoded, using a configurable number of redundant pieces, which are encrypted and then scattered on numerous nodes to ensure that the overall data is not compromised, and the protocol's content addressing architecture ensures that file names and file contents are strongly correlated, and the protocol's content addressing architecture ensures data integrity and verifiability. The protocol's content addressing architecture ensures data integrity and verifiability. 99.9999% high availability QoS (Quality of Service) is achieved through the use of code correction technology.

At the same time, as files are downloaded, the sharded data is reorganised and the sharded data is sourced from multiple storage providers, making the speed of decentralised storage much greater than that of centralised storage. In addition, centralised storage will only deploy a few core server rooms. Decentralised storage, on the other hand, has storage providers everywhere and uses the principle of proximity transfer, which will also be faster.

We believe that such a complete set of decentralised storage solutions provides a technical construct for data ownership and trust in the times of Web 3.0, and can nearly extend the Web 3.0 ecological blueprint, making commercial-grade big data on the chain a reality, and also helping human society to master and utilise data more effectively and realise the Internet of Value.