dodo
0.0.1
A C++ library to create containerized Linux services
|
A persistent, multi-threaded key-value store backed by sqlite3 database (file). More...
#include <kvstore.hpp>
Data Structures | |
struct | MetaData |
Meta data concerning the key. More... | |
Public Member Functions | |
KVStore (const std::filesystem::path &path) | |
Create the KVStore object against the path. More... | |
~KVStore () | |
Destructor, cleanup sync and close the SQLLite database. More... | |
void | checkpoint () |
Sync all to disk - issue a SQLite full checkpoint. More... | |
void | commitTransaction () |
Commit a transaction. More... | |
bool | deleteKey (const std::string &key) |
Delete the key or return false if the key does not exist. More... | |
std::string | ensureWithDefault (const std::string &key, const std::string &def) |
If the key does not exist, create it with the default and return the default. More... | |
double | ensureWithDefault (const std::string &key, double &def) |
If the key does not exist, create it with the default and return the default. More... | |
int64_t | ensureWithDefault (const std::string &key, int64_t &def) |
If the key does not exist, create it with the default and return the default. More... | |
bool | exists (const std::string &key) const |
Check if the key exists. More... | |
void | filterKeys (std::list< std::string > &keys, const std::string &filter) const |
Return a list of keys that match the filter. More... | |
MetaData | getMetaData (const std::string &key) const |
Get MetaData for the key. More... | |
bool | getValue (const std::string &key, common::Bytes &value) const |
If the key exists, returns true and copies (overwrites) data to the Bytes. More... | |
bool | getValue (const std::string &key, double &value) const |
If the key exists, returns true and sets the value parameter. More... | |
bool | getValue (const std::string &key, int64_t &value) const |
If the key exists, returns true and sets the value parameter. More... | |
bool | getValue (const std::string &key, std::string &value) const |
If the key exists, returns true and sets the value parameter. More... | |
bool | insertKey (const std::string &key, const common::Bytes &value) |
Insert a (key, Bytes) pair. More... | |
bool | insertKey (const std::string &key, const double &value) |
Insert a (key, double) pair. More... | |
bool | insertKey (const std::string &key, const int64_t &value) |
Insert a (key, int64_t) pair. More... | |
bool | insertKey (const std::string &key, const std::string &value) |
Insert a (key, string) pair. More... | |
void | optimize () |
Optimzime, preferably called after workload and implicitly called by the destructor. More... | |
void | rollbackTransaction () |
Rollback a transaction. More... | |
bool | setKey (const std::string &key, const common::Bytes &value) |
Set the binary data/Bytes value of an existing key. More... | |
bool | setKey (const std::string &key, const double &value) |
Set the double value of an existing key. More... | |
bool | setKey (const std::string &key, const int64_t &value) |
Set the int64_t value of an existing key. More... | |
bool | setKey (const std::string &key, const std::string &value) |
Set the string value of an existing key. More... | |
void | startTransaction () |
Start a transaction. More... | |
void | vacuum () |
Vaccum - clean and defragment, which may increase efficiency after heavy deletion and/or modification. More... | |
Protected Member Functions | |
void | createSchema () |
Create the SQLite schema. More... | |
void | prepareSQL () |
Prepare all SQL statements. More... | |
Protected Attributes | |
sqlite::Database * | db_ |
The database handle. More... | |
std::filesystem::path | path_ |
The filesystem path to the kvstore. More... | |
sqlite::DML * | stmt_delete_ |
Delete key pair statement handle. More... | |
sqlite::Query * | stmt_exists_ |
check key existence statement handle. More... | |
sqlite::Query * | stmt_getvalue_ |
Get-value-for-key statement handle. More... | |
sqlite::DML * | stmt_insert_ |
Insert key pair statement handle. More... | |
sqlite::Query * | stmt_key_filter_ |
Key filter statement handle. More... | |
sqlite::Query * | stmt_key_value_filter_ |
Key + value filter statement handle. More... | |
sqlite::Query * | stmt_metadata_ |
Get metadata statement handle. More... | |
sqlite::DML * | stmt_update_ |
Update key pair statement handle. More... | |
A persistent, multi-threaded key-value store backed by sqlite3 database (file).
If each thread uses its own KVStore object, no explicit synchronization is required between the threads. Cluster filesystems are not supported - two processes can only write to a single SQLite file if the processes run on the same host.
The KVStore is backed by the kvstore table. In SQLite, values in the same column can have different data types, so the table can store values of any of the supported data types:
DataType | C++ type | SQLite type |
---|---|---|
dtInteger | int64_t | SQLITE_INTEGER |
dtFloat | double | SQLITE_FLOAT |
dtText | std::string | SQLITE3_TEXT |
dtBlob | common::Bytes | SQLITE_BLOB |
dtUnknown | used to indicate the unknown dataType of a non-existing key |
In typical use the code is aware of the datatype of a key, so typical use would be
Moreover, SQLite will convert (CAST) between types where possible, but revert to defaults when it can't. For example,
will output 0 as the string "value" does not convert to an integer. Use getMetaData to check if a key is of the expected DataType.
It is also possible to store dodo::common::Bytes (dtBlob / SQLITE_BLOB) types.
The SQLite database is initailized in WAL mode for performance. Make sure to use transactions to group bulk insertKey or setKey as that is much faster.
The examples/kvstore/kvstore.cpp is a simple speed test using a KVStore:
Definition at line 77 of file kvstore.hpp.
dodo::persist::KVStore::KVStore | ( | const std::filesystem::path & | path | ) |
Create the KVStore object against the path.
If the KVStore does not exist yet, it is created. If it already exists, it is opened.
path | The filesystem path of the KVStore file. |
Definition at line 34 of file kvstore.cpp.
References createSchema(), db_, path_, prepareSQL(), stmt_delete_, stmt_exists_, stmt_getvalue_, stmt_insert_, stmt_key_filter_, stmt_key_value_filter_, stmt_metadata_, and stmt_update_.
dodo::persist::KVStore::~KVStore | ( | ) |
Destructor, cleanup sync and close the SQLLite database.
Definition at line 50 of file kvstore.cpp.
References checkpoint(), db_, optimize(), stmt_delete_, stmt_exists_, stmt_getvalue_, stmt_insert_, stmt_key_filter_, stmt_key_value_filter_, stmt_metadata_, and stmt_update_.
void dodo::persist::KVStore::checkpoint | ( | ) |
Sync all to disk - issue a SQLite full checkpoint.
Definition at line 64 of file kvstore.cpp.
References dodo::persist::sqlite::Database::checkPointFull(), and db_.
Referenced by ~KVStore().
void dodo::persist::KVStore::commitTransaction | ( | ) |
Commit a transaction.
Throws a dodo::common::Exception when no transaction has started.
Definition at line 68 of file kvstore.cpp.
References dodo::persist::sqlite::Database::commit(), and db_.
|
protected |
Create the SQLite schema.
Definition at line 72 of file kvstore.cpp.
References db_, dodo::persist::sqlite::DDL::execute(), dodo::persist::sqlite::Statement::prepare(), and dodo::persist::sqlite::Query::step().
Referenced by KVStore().
bool dodo::persist::KVStore::deleteKey | ( | const std::string & | key | ) |
Delete the key or return false if the key does not exist.
key | The key. |
Definition at line 90 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::DML::execute(), dodo::persist::sqlite::Statement::reset(), and stmt_delete_.
std::string dodo::persist::KVStore::ensureWithDefault | ( | const std::string & | key, |
const std::string & | def | ||
) |
If the key does not exist, create it with the default and return the default.
If the key exists, return its value (which may not be the default).
key | The key. |
def | The string value for the key if it does not exist. |
Definition at line 97 of file kvstore.cpp.
References getValue(), and insertKey().
double dodo::persist::KVStore::ensureWithDefault | ( | const std::string & | key, |
double & | def | ||
) |
If the key does not exist, create it with the default and return the default.
If the key exists, return its value (which may not be the default).
key | The key. |
def | The double value for the key if it does not exist. |
Definition at line 107 of file kvstore.cpp.
References getValue(), and insertKey().
int64_t dodo::persist::KVStore::ensureWithDefault | ( | const std::string & | key, |
int64_t & | def | ||
) |
If the key does not exist, create it with the default and return the default.
If the key exists, return its value (which may not be the default).
key | The key. |
def | The int64_t value for the key if it does not exist. |
Definition at line 117 of file kvstore.cpp.
References getValue(), and insertKey().
bool dodo::persist::KVStore::exists | ( | const std::string & | key | ) | const |
Check if the key exists.
Unless the value is not required , it is more efficient to get the value in one go by calling the getString(), getDouble(), getInt64() and getData() members, which return false if the key does not exist.
key | The key to check for. |
Definition at line 127 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::Query::getInt(), dodo::persist::sqlite::Statement::reset(), dodo::persist::sqlite::Query::step(), and stmt_exists_.
void dodo::persist::KVStore::filterKeys | ( | std::list< std::string > & | keys, |
const std::string & | filter | ||
) | const |
Return a list of keys that match the filter.
keys | The list that receives the keys. The list is cleared before assigning keys so it may turn up empty if the filter matches no keys. |
filter | The SQL-style case-sensitive filter as in 'match', 'match' |
Definition at line 135 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::Query::getText(), dodo::persist::sqlite::Statement::reset(), dodo::persist::sqlite::Query::step(), and stmt_key_filter_.
KVStore::MetaData dodo::persist::KVStore::getMetaData | ( | const std::string & | key | ) | const |
Get MetaData for the key.
key | The key. |
Definition at line 144 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::Query::getDataType(), dodo::persist::sqlite::Query::getDouble(), dodo::persist::sqlite::Query::getInt64(), dodo::persist::KVStore::MetaData::last_modified, dodo::persist::sqlite::Statement::reset(), dodo::persist::sqlite::Query::step(), stmt_metadata_, dodo::persist::KVStore::MetaData::type, and dodo::persist::KVStore::MetaData::update_count.
bool dodo::persist::KVStore::getValue | ( | const std::string & | key, |
common::Bytes & | value | ||
) | const |
If the key exists, returns true and copies (overwrites) data to the Bytes.
key | The key to get the value for. |
value | The Bytes that will receive the data. |
Definition at line 189 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::Query::getBytes(), dodo::persist::sqlite::Statement::reset(), dodo::persist::sqlite::Query::step(), and stmt_getvalue_.
bool dodo::persist::KVStore::getValue | ( | const std::string & | key, |
double & | value | ||
) | const |
If the key exists, returns true and sets the value parameter.
key | The key to get the value for. |
value | The destination value to set. |
Definition at line 167 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::Query::getDouble(), dodo::persist::sqlite::Statement::reset(), dodo::persist::sqlite::Query::step(), and stmt_getvalue_.
bool dodo::persist::KVStore::getValue | ( | const std::string & | key, |
int64_t & | value | ||
) | const |
If the key exists, returns true and sets the value parameter.
key | The key to get the value for. |
value | The destination value to set. |
Definition at line 178 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::Query::getInt64(), dodo::persist::sqlite::Statement::reset(), dodo::persist::sqlite::Query::step(), and stmt_getvalue_.
bool dodo::persist::KVStore::getValue | ( | const std::string & | key, |
std::string & | value | ||
) | const |
If the key exists, returns true and sets the value parameter.
key | The key to get the value for. |
value | The destination value to set. |
Definition at line 156 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::Query::getText(), dodo::persist::sqlite::Statement::reset(), dodo::persist::sqlite::Query::step(), and stmt_getvalue_.
Referenced by ensureWithDefault().
bool dodo::persist::KVStore::insertKey | ( | const std::string & | key, |
const common::Bytes & | value | ||
) |
Insert a (key, Bytes) pair.
Note that the size of the data cannot exceed INT_MAX, and exception is thrown if the Bytes is larger.
key | The key. |
value | The Bytes to insert. |
Definition at line 226 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::DML::execute(), dodo::persist::sqlite::Statement::reset(), and stmt_insert_.
bool dodo::persist::KVStore::insertKey | ( | const std::string & | key, |
const double & | value | ||
) |
Insert a (key, double) pair.
key | The key. |
value | The double value to set. |
Definition at line 208 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::DML::execute(), dodo::persist::sqlite::Statement::reset(), and stmt_insert_.
bool dodo::persist::KVStore::insertKey | ( | const std::string & | key, |
const int64_t & | value | ||
) |
Insert a (key, int64_t) pair.
key | The key. |
value | The int64_t value to set. |
Definition at line 217 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::DML::execute(), dodo::persist::sqlite::Statement::reset(), and stmt_insert_.
bool dodo::persist::KVStore::insertKey | ( | const std::string & | key, |
const std::string & | value | ||
) |
Insert a (key, string) pair.
key | The key. |
value | The string value to set. |
Definition at line 200 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::DML::execute(), dodo::persist::sqlite::Statement::reset(), and stmt_insert_.
Referenced by ensureWithDefault().
void dodo::persist::KVStore::optimize | ( | ) |
Optimzime, preferably called after workload and implicitly called by the destructor.
Definition at line 235 of file kvstore.cpp.
References db_, dodo::persist::sqlite::DDL::execute(), and dodo::persist::sqlite::Statement::prepare().
Referenced by ~KVStore().
|
protected |
Prepare all SQL statements.
Definition at line 241 of file kvstore.cpp.
References db_, dodo::persist::sqlite::DDL::execute(), dodo::persist::sqlite::Statement::prepare(), stmt_delete_, stmt_exists_, stmt_getvalue_, stmt_insert_, stmt_key_filter_, stmt_key_value_filter_, stmt_metadata_, and stmt_update_.
Referenced by KVStore().
void dodo::persist::KVStore::rollbackTransaction | ( | ) |
Rollback a transaction.
Throws a dodo::common::Exception when no transaction has started.
Definition at line 256 of file kvstore.cpp.
References db_, and dodo::persist::sqlite::Database::rollback().
bool dodo::persist::KVStore::setKey | ( | const std::string & | key, |
const common::Bytes & | value | ||
) |
Set the binary data/Bytes value of an existing key.
The function returns false if the key does not exist.
key | The key. |
value | The Bytes to set. |
Definition at line 284 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::DML::execute(), dodo::persist::sqlite::Statement::reset(), and stmt_update_.
bool dodo::persist::KVStore::setKey | ( | const std::string & | key, |
const double & | value | ||
) |
Set the double value of an existing key.
The function returns false if the key does not exist.
key | The key. |
value | The double value to set. |
Definition at line 268 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::DML::execute(), dodo::persist::sqlite::Statement::reset(), and stmt_update_.
bool dodo::persist::KVStore::setKey | ( | const std::string & | key, |
const int64_t & | value | ||
) |
Set the int64_t value of an existing key.
The function returns false if the key does not exist.
key | The key. |
value | The int64_t value to set. |
Definition at line 276 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::DML::execute(), dodo::persist::sqlite::Statement::reset(), and stmt_update_.
bool dodo::persist::KVStore::setKey | ( | const std::string & | key, |
const std::string & | value | ||
) |
Set the string value of an existing key.
The function returns false if the key does not exist.
key | The key. |
value | The string value to set. |
Definition at line 260 of file kvstore.cpp.
References dodo::persist::sqlite::DML::bind(), dodo::persist::sqlite::DML::execute(), dodo::persist::sqlite::Statement::reset(), and stmt_update_.
void dodo::persist::KVStore::startTransaction | ( | ) |
Start a transaction.
If insertKey or setKey calls are not inside a started transaction, each will commit automatically and indvidually, which is mach (much!) slower for bulk operations.
Definition at line 292 of file kvstore.cpp.
References db_, dodo::persist::sqlite::DDL::execute(), and dodo::persist::sqlite::Statement::prepare().
void dodo::persist::KVStore::vacuum | ( | ) |
Vaccum - clean and defragment, which may increase efficiency after heavy deletion and/or modification.
Definition at line 298 of file kvstore.cpp.
References db_, dodo::persist::sqlite::DDL::execute(), and dodo::persist::sqlite::Statement::prepare().
|
protected |
The database handle.
Definition at line 306 of file kvstore.hpp.
Referenced by checkpoint(), commitTransaction(), createSchema(), KVStore(), optimize(), prepareSQL(), rollbackTransaction(), startTransaction(), vacuum(), and ~KVStore().
|
protected |
The filesystem path to the kvstore.
Definition at line 303 of file kvstore.hpp.
Referenced by KVStore().
|
protected |
Delete key pair statement handle.
Definition at line 318 of file kvstore.hpp.
Referenced by deleteKey(), KVStore(), prepareSQL(), and ~KVStore().
|
protected |
check key existence statement handle.
Definition at line 309 of file kvstore.hpp.
Referenced by exists(), KVStore(), prepareSQL(), and ~KVStore().
|
protected |
Get-value-for-key statement handle.
Definition at line 312 of file kvstore.hpp.
Referenced by getValue(), KVStore(), prepareSQL(), and ~KVStore().
|
protected |
Insert key pair statement handle.
Definition at line 315 of file kvstore.hpp.
Referenced by insertKey(), KVStore(), prepareSQL(), and ~KVStore().
|
protected |
Key filter statement handle.
Definition at line 324 of file kvstore.hpp.
Referenced by filterKeys(), KVStore(), prepareSQL(), and ~KVStore().
|
protected |
Key + value filter statement handle.
Definition at line 327 of file kvstore.hpp.
Referenced by KVStore(), prepareSQL(), and ~KVStore().
|
protected |
Get metadata statement handle.
Definition at line 330 of file kvstore.hpp.
Referenced by getMetaData(), KVStore(), prepareSQL(), and ~KVStore().
|
protected |
Update key pair statement handle.
Definition at line 321 of file kvstore.hpp.
Referenced by KVStore(), prepareSQL(), setKey(), and ~KVStore().