Codec layer

Codec layer design #

The codec layer is the glue of the IceFireDB data expression layer, because the bottom layer of IceFireDB supports many storage engines, including centralized storage such as web2 disk, OSS, leveldb, and badger, as well as web3’s IPFS, crdt-kv, and IPFS-LOG For this kind of decentralized storage, the storage interface provided by any kind of storage is simple and not standardized. The codec layer of IceFireDB-NoSQL is abstracted through a unified driver layer, and by encoding and decoding many instruction semantics into a KV model, a richer data expression layer is built to support more data scenarios, such as Strings\Hashs\Sets\Lists \Sorted Sets.

+-------------------------------------------------------------+
|                        Query Processor                      |
|   +-----------------------------------------------------+   |
|   |                     Query Parser                    |   |
|   +-----------------------------------------------------+   |
|   +-----------------------------------------------------+   |
|   |                   Query Optimizer                   |   |
|   +-----------------------------------------------------+   |
+---+--------------------------+--------------------------+---+
                               |
+------------------------------v------------------------------+
|                            Codec                            |
|    +-----------+                           +-----------+    |
|    |   Encode  |                           |   Decode  |    |
|    +-----------+                           +-----------+    |
|                 support: kv、list、hash、set                 |
+----------+---------------------------------------^----------+
           |                                       |
+----------+---------------------------------------+----------+
|          |put              KV Engine             |Get       |
|    +-----v----+                            +-----+----+     |
|    | put(a,b) |                            |  Get(a)  |     |
|    +-----+----+                            +-----+----+     |
|          | a:b            +-------+              | a        |
|    +-----v----+    +------> store <----+   +-----v----+     |
|    |  CID(b)  +----+      +-------+    +---+ cat(hash)|     |
|    +-----+----+                            +-----+----+     |
|          | add(b)                                | cat      |
|  --------v---------------------------------------v-----     |
|           Leveldb\Badger\OSS\IPFS\CRDT\IPFS-LOG             |
+-------------------------------------------------------------+

As an important glue layer, the codec layer connects the network layer, request layer, and KV storage layer.

Codec advantages #

In the decentralized database scenario, although the community also has a decentralized storage solution similar to the KV model, the KV model cannot meet the complex use of upper-level applications. We believe that rich data structures need to be supported to support the decentralized development of applications, including NoSQL data models (such as strings\hashs\lists\sets\sorted sets data structures). Just as the prosperity of the web2 application ecology is inseparable from the contribution of database infrastructure such as memcached, redis, and mysql, IceFireDB provides richer data types and can support data decentralization in more complex scenarios.

The encoding and decoding layer encodes data commands and parameters to meet the conversion of many rich instruction data models to KV models. Under the function of this conversion layer, the complex shielding layer of the data model layer and the storage engine layer is effectively constructed.

  1. The data instruction layer does not need to care about the underlying storage engine, so it can adapt to various storage engine drivers with the help of the codec layer.

  2. The underlying storage engine continues to provide a simple data operation interface (put\get\del\iterator), without direct coupling and customization with the data presentation layer, and the abstract storage driver layer of the codec layer maintains the easy work of the two layers.

Any problem in computer science can be solved by another layer of indirection.

Codec Layer value #

In addition to adding a richer data model and isolating the complexity of the request layer and storage layer, the codec layer of IceFireDB also has the following functions:

  1. Improve data access speed: A memory buffer layer is added to the encoding layer to effectively improve the performance of data access.
  2. Calculate the middle layer：The support of complex data structures often requires a certain amount of calculation. At present, the coding layer mainly performs CPU calculations, and subsequent calculations can be combined with GPU and FPGA, which can expand high-performance data calculations and add more possibilities for decentralized databases.
  3. blockchain/web3 connection layer: At present, whether it is a web2 or web3 database scenario, data access is the focus of attention, but with the development of blockchain and web3, trusted computing of data and non-tamperable proofs are becoming more and more important. The codec layer of IceFireDB is playing an important role , can be used to combine the blockchain data structure to build web3 side chain infrastructure, reducing the calculation and storage burden of the blockchain layer.