Storage layer design #

The storage layer is responsible for data storage, and the data storage here includes different storage media of web2 and web3. For web2, the storage media we face includes disk, OSS, and for web3, the storage media we face includes IPFS, blockchain, and smart contracts.Currently, the storage types supported by IceFireDB mainly include the following.

Engine type	describe	Driver directory
LevelDB	LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.	Default
Badger	BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go.	Badger
OSS	Object storage is a technology that stores and manages data in an unstructured format called objects.	OSS
IPFS	IPFS (the InterPlanetary File System) is a hypermedia distribution protocol addressed by content and identities. It enables the creation of completely distributed applications, and in doing so aims to make the web faster, safer, and more open.	IPFS
CRDT-KV	The IceFireDB-CRDT-KV engine can support decentralized P2P networking, data synchronization and consistency between nodes. It is a component of the IceFireDB software ecosystem, thanks to the open source of IPFS.	CRDT-KV
IPFS-LOG	icefiredb-ipfs-log is a distributed immutable, operation-based conflict-free replication data structure that relies on ipfs to store data and merges each peer node data based on pubsub conflict-free. You can easily implement custom data structures such as kv, event, nosql, etc. based on icefiredb-ipfs-log.	IPFS-LOG
OrbitDB	OrbitDB is a serverless, distributed, peer-to-peer database. OrbitDB uses IPFS as its data storage and IPFS Pubsub to automatically sync databases with peers.	OrbitDB

Storage model #

The NoSQL storage layer of each individual IceGiant mainly includes the codec layer and the underlying KV storage layer. the underlying KV engine currently supports levelDB, badgerDB, IPFS and OSS, and the main data storage includes two ways:

instruction broadcast model based on IPFS-LOG\CRDT_KV\OrbitDB
Native data storage model based on LevelDB\Badger\OSS\IPFS

Multiple IceFireDB nodes will be divided into groups according to data sets, and each group will form a highly available storage area structure.

NoSQL storage engine #

The core of each node is the database engine. By default, IceGiant node integrates KV storage engines such as levelDB, badgerDB, IPFS, OSS, etc., and implements the protocol coding layer of NoSQL on the KV storage relationship. Currently, the data storage of NoSQL mainly includes the following two ways:

Instruction broadcast model #

Based on ipfs-log,crdt and libp2p(pubsub), an immutable and operation-based conflict-free replication data model for distributed systems is implemented. Based on ipfs-log, various data structures such as event and kv are encapsulated, and multi-node database instruction broadcast is implemented based on this engine;At that bottom of IceFireDB, we abstract the variable kv engine base on badgerdb and leveldb. any node will broadcast the whole network when it is writing instruction, and the bottom driver of IceFireDB of each node will execute the broadcast instruction to ensure the final consistency of data.

           Log A                Log B
             |                    |
     logA.append("one")   logB.append("hello")
             |                    |
             v                    v
          +-----+             +-------+
          |"one"|             |"hello"|
          +-----+             +-------+
             |                    |
     logA.append("two")   logB.append("world")
             |                    |
             v                    v
       +-----------+       +---------------+
       |"one","two"|       |"hello","world"|
       +-----------+       +---------------+
             |                    |
             |                    |
       logA.join(logB) <----------+
             |
             v
+---------------------------+
|"one","hello","two","world"|
+---------------------------+

Full storage model #

In addition to the first implementation mode, we are also building the structure of the second type of data, so that the complete data will grow on ipfs. At first, there is an ipfs driver in the IceFireDB driver layer, which will encode and process the upper-level commands into a unified kv data structure, store and change the value, and the generated new cid will be connected with key. However, at present, there is no key broadcast network with multiple nodes and data synchronization. When we connect with the broadcast network, we can build a data model originally grown on LevelDB\Badger\OSS\IPFS.

+-------------------------------------------------------------+
|                            Codec                            |
|    +-----------+                           +-----------+    |
|    |   Encode  |                           |   Decode  |    |
|    +-----------+                           +-----------+    |
|                 support: kv、list、hash、set                 |
+----------+---------------------------------------^----------+
           |                                       |
+----------+---------------------------------------+----------+
|          |put              KV Engine             |Get       |
|    +-----v----+                            +-----+----+     |
|    | put(a,b) |                            |  Get(a)  |     |
|    +-----+----+                            +-----+----+     |
|          | a:b            +-------+              | a        |
|    +-----v----+    +------> store <----+   +-----v----+     |
|    |  CID(b)  +----+      +-------+    +---+ cat(hash)|     |
|    +-----+----+                            +-----+----+     |
|          | add(b)                                | cat      |
|  --------v---------------------------------------v-----     |
|           Leveldb\Badger\OSS\IPFS\CRDT\IPFS-LOG             |
+-------------------------------------------------------------+

IceGiant Synchronizer #

The storage layer of IceFireDB not only includes a complete storage server, but IceGiant Synchronizer, which is currently under construction, also belongs to the ecological software layer of the storage layer.

IceGiant Synchronizer is an application directly above the database engine. All incoming database requests pass through IceGiant Synchronizer, which determines whether the requests should be processed, whether data writes should be propagated to other parts of the network, and whether local data should be written and customer requests should be responded to.

IceGiant Synchronizer can also provide data write aggregation function, allowing multiple data requests to be merged and written into a single network storage request. It also allows users to cross-mix data sets between different nodes, encouraging further data decentralization, while keeping the operation overhead low.