From reillyhekazusa at gmail.com Wed Apr 26 05:17:17 2023 From: reillyhekazusa at gmail.com (Reilly He) Date: Wed Apr 26 05:09:27 2023 Subject: [odb-users] Questions about releasing/closing CONNECTION POOL under WAL mode Message-ID: Dear All/Boris, Sorry to bother you guys, recently I have some questions about closing/releasing connection-pool in the ODB framework. I understand this email will be a little bit long to read, so thanks a lot for your patience. Deeply appreciated if you guys can kindly provide us some help. 1.Env The general environment is: - *Compiler*: CXX11 - *OS*: Android&IOS - *Inner DB*: SQLite 2. Background information In our project, we are using *odb::sqlite::database *under WAL mode. And since we are using it under WAL mode, we managed to insert our own implementation of *odb::sqlite::connection_factory* in order to fully activate the power of *WAL*. Why we want to implement our own connection pool: > Because if we open a database by default, the odb framework will create a > database under SHARED_CACHE mode, > with a *odb::sqlite::connection_pool_factory* with *min* and *max* equals > to ZERO. I found this via reading odb source code, *correct me if I am > wrong*. So as a result, we just wrap a *odb::sqlite::connection_pool_factory *and a * odb::sqlite::single_connection_factory* as our new connection pool: > /** > * a connection pool which support multi connections under WAL mode > * noted that by default: > * - reader`s connection will be provided by a connection_pool_factory > which consists of [getMinReaderConnectionCount, > getMaxReaderConnectionCount] connections > * - writer`s connection will be provided by a single_connection_factory > which ONLY HAVE ONE connection > * requesting a connection when either pool is empty will result in > WAITING for any threads > */ > class RWConnectionPool: public odb::sqlite::connection_factory{ > typedef std::unique_ptr > ReaderPool; > typedef std::unique_ptr > WriterPool; > public: > RWConnectionPool(std::shared_ptr delegate, > std::shared_ptr config); > odb::sqlite::connection_ptr connect() override; > void database(database_type & db) override; > ~RWConnectionPool(); > > protected: > ReaderPool m_Reader{}; > WriterPool m_Writer{}; > }; > We set our m_Reader`s min = 4, and max = 4, which means during the runtime we will hold 4 connections forever for database reading, and m_Writer is a *odb::sqlite::single_connection_factory, *so it will hold 1 connection forever for database writing. 3. Problems Since we just made a wrap of *odb::sqlite::single_connection_factory *and *odb::sqlite::connection_pool_factory *to come up with our own connection pool. *Q:When?* I wanna make sure whether we should release/close all the connections in these 2 pools *explicitly*? *Q:How?* If we really need to do that, how to do it? I don`t think there are any *existing APIs *in your docs for me to call directly. After checking the source codes, are *odb::sqlite::single_connection_factory *and *odb::sqlite::connection_pool_factory`s *deallocator just doing those things for us? *Q:Process related issue?* Our program also runs on *Android* platform, and in android platform, we got NO hint/callbacks before process exiting. So in this case, the pool's *deallocator* will not be working (since user just kill our program`s process), is that OK? Since we released this feature to the market, we received 17 Android database corruption issues compared to just 1 iOS database corruption issue on IOS. Again, really appreciated for any kind of help. Best regards, Reilly He From boris at codesynthesis.com Wed Apr 26 09:14:15 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Apr 26 09:06:07 2023 Subject: [odb-users] Questions about releasing/closing CONNECTION POOL under WAL mode In-Reply-To: References: Message-ID: Reilly He writes: > > class RWConnectionPool: public odb::sqlite::connection_factory > > { > > typedef std::unique_ptr ReaderPool; > > typedef std::unique_ptr WriterPool; > > > > [...] > > > > protected: > > ReaderPool m_Reader{}; > > WriterPool m_Writer{}; > > }; > > > > I wanna make sure whether we should release/close all the connections in > these 2 pools *explicitly*? The only way to release/close connections is to destroy the corresponding connection object. So in your case the natural thing to do if you want to explicitly close both types of connections is to destroy your two pools: m_Reader.reset (); m_Writer.reset (); > Our program also runs on *Android* platform, and in android platform, we > got NO hint/callbacks before process exiting. So in this case, the pool's > *deallocator* will not be working (since user just kill our program`s > process), is that OK? Since we released this feature to the market, we > received 17 Android database corruption issues compared to just 1 iOS > database corruption issue on IOS. SQLite should normally be able to handle this provided the underlying storage is "sane" (i.e., actually durable, not lying to SQLite that a write has been committed to physical media, etc). If that's not the case and there is no notification for a graceful exit, then I believe your only option is to only hold open transactions and connections while writing, closing everything as soon as you are done, and then re-opening when you need to write again. An alternative to closing everything would be to force the WAL checkpointing using sqlite3_wal_checkpoint_v2()[1] after each write batch. But generally, if your storage is not "sane", it will be hard to achieve 100% reliability and your only recourse is to try to minimize corruptions. [1] https://www.sqlite.org/wal.html#application_initiated_checkpoints From reillyhekazusa at gmail.com Wed Apr 26 22:19:01 2023 From: reillyhekazusa at gmail.com (Reilly He) Date: Wed Apr 26 22:11:11 2023 Subject: [odb-users] Questions about releasing/closing CONNECTION POOL under WAL mode In-Reply-To: References: Message-ID: Dear Boris, The only way to release/close connections is to destroy the corresponding > connection object. So in your case the natural thing to do if you want to > explicitly close both types of connections is to destroy your two pools: > > m_Reader.reset (); > m_Writer.reset (); > Got it, this is *exactly what we did* in order to close the connection pool. An alternative to > closing everything would be to force the WAL checkpointing using > sqlite3_wal_checkpoint_v2()[1] after each write batch. > Speaking about checkpoints, we are using auto_checkpoint with a threshold of *150* sqlite pages. But generally, if your storage is not "sane", it will be hard to > achieve 100% reliability and your only recourse is to try to minimize > corruptions. > And I have heard some *dirty tricks* that a harddrive might play (i.e., return "finished" for flushing data while it has not even begun). So the storage might actually be an issue for Android users. We will keep looking for any further evidence to proof that. Thanks a lot for the help boris. Best Regards, Reilly He On Wed, Apr 26, 2023 at 9:14?PM Boris Kolpackov wrote: > Reilly He writes: > > > > class RWConnectionPool: public odb::sqlite::connection_factory > > > { > > > typedef std::unique_ptr > ReaderPool; > > > typedef std::unique_ptr > WriterPool; > > > > > > [...] > > > > > > protected: > > > ReaderPool m_Reader{}; > > > WriterPool m_Writer{}; > > > }; > > > > > > > I wanna make sure whether we should release/close all the connections in > > these 2 pools *explicitly*? > > The only way to release/close connections is to destroy the corresponding > connection object. So in your case the natural thing to do if you want to > explicitly close both types of connections is to destroy your two pools: > > m_Reader.reset (); > m_Writer.reset (); > > > > Our program also runs on *Android* platform, and in android platform, we > > got NO hint/callbacks before process exiting. So in this case, the pool's > > *deallocator* will not be working (since user just kill our program`s > > process), is that OK? Since we released this feature to the market, we > > received 17 Android database corruption issues compared to just 1 iOS > > database corruption issue on IOS. > > SQLite should normally be able to handle this provided the underlying > storage is "sane" (i.e., actually durable, not lying to SQLite that a > write has been committed to physical media, etc). > > If that's not the case and there is no notification for a graceful exit, > then I believe your only option is to only hold open transactions and > connections while writing, closing everything as soon as you are done, > and then re-opening when you need to write again. An alternative to > closing everything would be to force the WAL checkpointing using > sqlite3_wal_checkpoint_v2()[1] after each write batch. > > But generally, if your storage is not "sane", it will be hard to > achieve 100% reliability and your only recourse is to try to minimize > corruptions. > > [1] https://www.sqlite.org/wal.html#application_initiated_checkpoints > From reillyhekazusa at gmail.com Wed Apr 26 23:02:00 2023 From: reillyhekazusa at gmail.com (Reilly He) Date: Wed Apr 26 22:54:07 2023 Subject: [odb-users] Strange behavior of odb::details::transfer_ptr Message-ID: Dear Boris, Sorry to bother you again, I wrote this to you just to make sure *the usage of transfer_ptr*. 1.Env The general environment is: - *Compiler*: CXX11 - *OS*: Android&IOS - *Inner DB*: SQLite 2. Background information Below is the code in our project for database opening. As you can see, we use our own implementation of connection-pool. > //create database under wal > > //create connection pool for database > > auto connectionPoolDelegate = std::make_shared< > StandardDelegate>(); > > //the pool will be access ONLY by ODB database after creation > > std::unique_ptr rwPool(new > RWConnectionPool(connectionPoolDelegate, config)); > > //the transfer pointer is a WEIRD pointer used by ODB > > odb::details::transfer_ptr > poolInTransP = odb::details::transfer_ptr >(std::move(rwPool)); > > auto db = std::shared_ptr(new odb:: > sqlite::database(dbFile, > > > SQLITE_OPEN_READWRITE > > > | SQLITE_OPEN_CREATE > > > | SQLITE_OPEN_FULLMUTEX > > > | SQLITE_OPEN_PRIVATECACHE, > > > false, > > > "", > > > poolInTransP)); > > //the tricky part is you have to release the transfer pointer > in here > > //otherwise it will cause double release for this connection > pool > > poolInTransP.transfer(); > > return db; > I was really confused about transfer_ptr, I reckon this is *very buggy* because you have to t*ransfer() the ptr everytime* in its scope. Then reason why I have to do this is after checking the s*ource code of transfer_ptr*, I realized that it will d*eallocate the inner ptr it was holding in transfer_ptr`s deallocator*: > ~transfer_ptr () {delete p_;} > > > T* > > transfer () > > { > > T* r (p_); > > p_ = 0; > > return r; > > } > And in the constructor of odb::sqlite::database, *it takes a transfer_ptr via its copy constructor:* > database:: > database (const string& name, > int flags, > bool foreign_keys, > const string& vfs, > transfer_ptr factory) > : odb::database (id_sqlite), > name_ (name), > flags_ (flags), > foreign_keys_ (foreign_keys), > vfs_ (vfs), > factory_ (factory.transfer ()) > { > if (!factory_) > factory_.reset (new connection_pool_factory ()); > > factory_->database (*this); > } > > Thus, during this scope, *2 transfer_ptrs existed in memory,* and as a result, when we leave the current scope. The *real ptr will be deallocated by the first transfer_ptr* if we do not transfer it explicitly. 3. Question Is it *better* to take an *r-value* of transfer_ptr in the constructor of odb::sqlite::database? And I reckon it is even better to just *delete transfer_ptr`s copy constructor*. Since odb supports C++ 11, why just allow us to move our connection pool into the factory_ member via *std::unique_ptr*? Thanks a lot for the help, really appreciated. Best Regards, Reilly He