Network layer design #
The network layer undertakes the work of inter-node networking, inter-node data distribution, and inter-node data consistency consensus. The network layer of IceFireDB is divided into two layers according to the distance of the physical network link:
- Data consistency network layer for short-distance networks.
- Decentralized database network layer for wide-distance network.
The above two different network layers are supported by different technologies and have different requirements for data consistency sensitivity and timeliness.
Parallel cluster network #
The smallest component unit of the IceFireDB network cluster is a parallel cluster network guaranteed by the RAFT algorithm, rather than a decentralized P2P network.
The IceFireDB parallel cluster uses the RAFT algorithm to form a data consistency layer. Each cluster has a master and multiple slave nodes. They synchronize with each other and maintain the master-slave state of the cluster. Each node stores the same data and writes data. The input is undertaken by the master node, and the master node is responsible for distributing data to the slave nodes. A RAFT cluster internally guarantees the consistency of each data write.
Decentralized network #
The data jumps out of the parallel cluster network and enters the decentralized network. The IceFireDB decentralized network mainly uses P2P technology for automatic networking, and relies on the P2P PubSub communication method to synchronize data commands.
IceFireDB’s use of P2P networking is not only in node discovery, but also provides users with decentralized publish and subscribe middleware by bridging the PubSub network to the RESP protocol. Increase the decentralization capability of the SQLite database by bridging the PubSub network to the SQL protocol.
Decentralized log synchronization network #
Relying solely on P2P and PubSub technologies cannot meet the database requirements for data decentralized synchronization and decentralized data consistency.The current popular CRDT technology can meet the final data consistency, but it cannot guarantee the synchronization order of the data, so it will castrate some data instruction functions of IceFireDB-NoSQL, and the IPFS-LOG technology very well makes up for the functional gap of the decentralized log.
IceFireDB-NoSQL has established a decentralized log synchronization network based on IPFS-LOG technology. The decentralized IceFireDB nodes broadcast database command logs and build orderly logs to complete the function of data decentralization and consistency.
IceGiant Network structure #
IceFireDB’s RAFT has exactly the same data set within the same group of nodes, and in a larger network, we are providing the IceGiant network structure, which aims to break up different tenant database data.
Although all IceGiant nodes are in the same P2P network, the structure of the network can be decomposed according to the data set, which is the data tenant isolation area, which refers to the database area used by a specific application on top of the IceGiant protocol. Different from the traditional blockchain network, IceGiant nodes are only responsible for interacting with peers operating the same data set, and are not responsible for any data of other data sets. Peer IceGiant node sets form a high-availability data storage area.
In-process network IO model #
IceFireDB supports the following two IO models:
Golang classic netpoll model: goroutine-per-connection, suitable for situations where the number of connections is not a bottleneck.
RawEpoll model: that is, Reactor mode, I/O multiplexing (I/O multiplexing) + non-blocking I/O (non-blocking I/O) mode. For scenarios where there are a large number of long links between the access layer and the gateway, it is more suitable for the RawEpoll model.