From boris at codesynthesis.com Mon May 1 12:34:36 2017 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon May 1 12:34:44 2017 Subject: [odb-users] safe usage odb connection pool in multi threaded environment (odb 2.4.0) In-Reply-To: References: Message-ID: Hi, ?????????? ??????? writes: > I want to figure out a proper way to manage database connection pool > assuming I want to use connection in different threads. You are over-thinking and over-complicating things ;-). ODB database is a thread-safe connection pool. So just pass a single instance of the database to all your threads and execute your database operations on it. All the connection management will be taken care of in a thread-safe manner. If you do want to work with connections explicitly for some reason (for example you want to make sure two transactions are executed on the same connection), then keep them local to the same thread. That is, get it in the thread, use for as long as necessary, then release in the same thread. Boris From pustovalovdmit at gmail.com Tue May 2 04:55:17 2017 From: pustovalovdmit at gmail.com (=?UTF-8?B?0J/Rg9GB0YLQvtCy0LDQu9C+0LIg0JTQvNC40YLRgNC40Lk=?=) Date: Tue May 2 04:55:40 2017 Subject: [odb-users] safe usage odb connection pool in multi threaded environment (odb 2.4.0) In-Reply-To: References: Message-ID: Thanks, Boris! Am I understand right that I should pass odb::database between the threads and when I begin transaction on some thread passing to odb::transaction return value from odb::database::begin I ensure that this transaction will work on other database connection if needed? ??, 1 ??? 2017 ?. ? 19:34, Boris Kolpackov : > Hi, > > ?????????? ??????? writes: > > > I want to figure out a proper way to manage database connection pool > > assuming I want to use connection in different threads. > > You are over-thinking and over-complicating things ;-). ODB database > is a thread-safe connection pool. So just pass a single instance of the > database to all your threads and execute your database operations > on it. All the connection management will be taken care of in a > thread-safe manner. > > If you do want to work with connections explicitly for some reason > (for example you want to make sure two transactions are executed > on the same connection), then keep them local to the same thread. > That is, get it in the thread, use for as long as necessary, then > release in the same thread. > > Boris > From boris at codesynthesis.com Tue May 2 09:05:56 2017 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue May 2 09:06:05 2017 Subject: [odb-users] safe usage odb connection pool in multi threaded environment (odb 2.4.0) In-Reply-To: References: Message-ID: Hi, ?????????? ??????? writes: > Am I understand right that I should pass odb::database between the threads > and when I begin transaction on some thread passing to odb::transaction > return value from odb::database::begin I ensure that this transaction will > work on other database connection if needed? When you call database::begin() it gets the connection from the pool and returns it together with the transaction object. Once you finalize the transaction (e.g., commit or roll it back, or let it go out of scope in which case it is rolled back automatically), then the connection is automatically returned back into the pool. Boris From pustovalovdmit at gmail.com Tue May 2 11:47:06 2017 From: pustovalovdmit at gmail.com (=?UTF-8?B?0J/Rg9GB0YLQvtCy0LDQu9C+0LIg0JTQvNC40YLRgNC40Lk=?=) Date: Tue May 2 11:47:30 2017 Subject: [odb-users] safe usage odb connection pool in multi threaded environment (odb 2.4.0) In-Reply-To: References: Message-ID: I got it! Thank you very much, Boris. ??, 2 ??? 2017 ?. ? 16:06, Boris Kolpackov : > Hi, > > ?????????? ??????? writes: > > > Am I understand right that I should pass odb::database between the > threads > > and when I begin transaction on some thread passing to odb::transaction > > return value from odb::database::begin I ensure that this transaction > will > > work on other database connection if needed? > > When you call database::begin() it gets the connection from the pool and > returns it together with the transaction object. Once you finalize the > transaction (e.g., commit or roll it back, or let it go out of scope in > which case it is rolled back automatically), then the connection is > automatically returned back into the pool. > > Boris > From haupt.wolfgang at gmail.com Sun May 7 07:19:45 2017 From: haupt.wolfgang at gmail.com (Wolfgang Haupt) Date: Sun May 7 07:20:09 2017 Subject: [odb-users] Crash on Windows startup Message-ID: Hi, odb-version: 2.4.0 official release I experienced a "schema not found" exception when calling migrate on windows. Similar problems happend on linux/osx without having --whole-archive. Tried to switch everything to build statically, but now I have an instant crash. See: https://pastebin.com/3fY7nxXj It seems that "win32_once_cs_" is a global and tried to be locked before globals are initialized. Is there anything I can do about it? Could we patch it, so that this lock is not global? Best Regards, Wolfgang From boris at codesynthesis.com Mon May 8 12:30:32 2017 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon May 8 12:30:41 2017 Subject: [odb-users] Crash on Windows startup In-Reply-To: References: Message-ID: Hi Wolfgang, Wolfgang Haupt writes: > I experienced a "schema not found" exception when calling migrate on > windows. > Similar problems happend on linux/osx without having --whole-archive. > Tried to switch everything to build statically, There is a way to make it work on Windows: http://www.codesynthesis.com/pipermail/odb-users/2013-May/001289.html > [...] but now I have an instant crash This is probably due to the libodb initialization that happens in DllMain() is not being performed. If you want to continue with this route (link everything statically), check what it calls and do the same from your main(). Boris From haupt.wolfgang at gmail.com Tue May 9 15:23:50 2017 From: haupt.wolfgang at gmail.com (Wolfgang Haupt) Date: Tue May 9 15:24:13 2017 Subject: [odb-users] Crash on Windows startup In-Reply-To: References: Message-ID: Nice. Thanks for your help, if this approach works I will use dynamic linking. As for the static approach - looking at the stacktrace I doubt my main gets even called, so not sure if I could fix the initialization problem that way. Best Regards, Wolfgang Boris Kolpackov schrieb am Mo., 8. Mai 2017 um 18:30 Uhr: > Hi Wolfgang, > > Wolfgang Haupt writes: > > > I experienced a "schema not found" exception when calling migrate on > > windows. > > Similar problems happend on linux/osx without having --whole-archive. > > Tried to switch everything to build statically, > > There is a way to make it work on Windows: > > http://www.codesynthesis.com/pipermail/odb-users/2013-May/001289.html > > > > [...] but now I have an instant crash > > This is probably due to the libodb initialization that happens in DllMain() > is not being performed. If you want to continue with this route (link > everything statically), check what it calls and do the same from your > main(). > > Boris > From anton.paymyshev at gmail.com Tue May 23 04:22:22 2017 From: anton.paymyshev at gmail.com (Anton Paymyshev) Date: Tue May 23 04:22:56 2017 Subject: [odb-users] odb crashes when querying object with section Message-ID: Hello, I'm having issues with odb 2.5.0.a11 sqlite 3.13.0 vs2015 update 3 windows 10 both release and debug Schema and code below. Test case requires two runs. On the first run sqlite database file is created and filled with a single row. On the second run it tries to query that single row and crashes in *result.begin() line, crash stack(note garbage in capacity argument value) is similar to: KernelBase.dll!_RaiseException@16() Unknown > vcruntime140.dll!_CxxThrowException(void * pExceptionObject=0x008ff654, const _s__ThrowInfo * pThrowInfo=0x0f73a2fc) Line 131 C++ odb-2.5.dll!__scrt_throw_std_bad_alloc() Line 33 C++ odb-2.5.dll!operator new(unsigned int size=0xcdcdcdcd) Line 19 C++ odb-2.5.dll!odb::details::basic_buffer_base::capacity(unsigned int c=0xcdcdcdcd, unsigned int data_size=0x00000000) Line 21 C++ odbbugtest.exe!odb::sqlite::object_result_impl::load_image() Line 132 C++ odbbugtest.exe!odb::sqlite::object_result_impl::load_id() Line 95 C++ odbbugtest.exe!odb::object_result_impl::load() Line 25 C++ My analysis: - db->query() line runs object_statements construction which leaves image_ field filled with garbage from heap. (Test case requires "prepared" heap to be reproducible in release build) - then object_result_impl::load_image() gets "truncated" result from "statement_->load()" and goes to access::object_traits_impl< ::Test, id_sqlite >::grow which tries to set sec1_value capacity to previously uninitialized value(0xcdcdcdcd). //-----schema (Schema.h): #include #include #pragma db object class Test { public: #pragma db id unsigned long id; std::string str; #pragma db section(mysection) std::string sec1; #pragma db load(lazy) odb::section mysection; }; //-----code (main.cpp): #include #include #include #include "Schema-odb.hxx" int main() { std::unique_ptr db; odb::connection_ptr c; try { db.reset(new odb::sqlite::database(L"test.db", SQLITE_OPEN_READWRITE)); c = db->connection(); } catch (odb::exception &) //first run goes here, setup schema and a single Test row { db.reset(new odb::sqlite::database(L"test.db", SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE)); c = db->connection(); odb::transaction t(c->begin()); odb::schema_catalog::create_schema(*db); Test test; test.id = 12345; test.str = std::string(1024, 'x'); // comment that line and test case runs OK db->persist(test); t.commit(); return 0; } odb::transaction t(c->begin()); { // fill heap with garbage for (auto i = 1; i < 10240; ++i) std::vector ttt(i, 0xcdcdcdcd); } auto result = db->query(odb::query::id == 12345); auto &row = *result.begin(); // <--- crash here //ASSERT(row.id == 12345) t.commit(); return 0; } From boris at codesynthesis.com Tue May 23 09:50:34 2017 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue May 23 09:50:43 2017 Subject: [odb-users] odb crashes when querying object with section In-Reply-To: References: Message-ID: Hi Anton, Anton Paymyshev writes: > My analysis: > > - db->query() line runs object_statements construction which > leaves image_ field filled with garbage from heap. (Test case requires > "prepared" heap to be reproducible in release build) > - then object_result_impl::load_image() gets "truncated" result from > "statement_->load()" and goes to access::object_traits_impl< ::Test, > id_sqlite >::grow which tries to set sec1_value capacity to previously > uninitialized value(0xcdcdcdcd). Thanks for the testcase and the analysis. This appears to be the second part of the bug that I (thought I) fixed earlier in this area[1]. I've fixed it some more[2] (hopefully for good this time) and have built you a pre-release: http://codesynthesis.com/~boris/tmp/odb/pre-release/b.3/ Let me know if there are any issues. [1] http://scm.codesynthesis.com/?p=odb/odb.git;a=commit;h=5d969f916178eb5e223d658dd55f5f2bc9ab526d [2] http://scm.codesynthesis.com/?p=odb/odb.git;a=commit;h=7ebcc590a5ca302f35805a025545a8d10ce4cf5f Boris From oliver.reid at otago.ac.nz Sun May 28 21:40:15 2017 From: oliver.reid at otago.ac.nz (Oliver Reid) Date: Mon May 29 07:58:09 2017 Subject: [odb-users] Vector causing unexpected EOF error Message-ID: <0282509e-4060-43f3-8e81-da7fc35745e9@Spark> Hey team, I?m trying to use ODB in a project that I am currently working on. The current setup is four classes: - Event - Timeline - TimelineItem - User A User holds a reference to Timeline*. A Timeline* holds a reference to a vector of TimelineItem*. Each TimelineItem* holds a reference to an Event* and a reference to another TimelineItem*(if the item is a repeated item - this links an item that is a repeating item on the timeline, back to the original item) or if the item is an item that is being repeated (it is an original item, that is being repeated by other items, so we hold a reference to which items repeat it), it holds a reference to a vector of TimelineItem*. At the moment, I am having no issues adding a new TimelineItem which isn?t repeated to the database: 1. unique_ptr timeline(db->query_one (timeline_query::id == tl_id)); 2. 3. Event *new_event = new Event(type, description, location); 4. TimelineItem *new_item = new TimelineItem(new_event, start, end); 5. 6. // Add the new item to the timeline 7. timeline->addTimelineItem(new_item); 8. 9. // Persist TimelineItem and update Timeline 10. db->persist(new_event); 11. db->persist(new_item); 12. db->update(*timeline); However, as soon as I try to add a TimelineItem that holds a vector of TimelineItem that repeats it, Postgres is throwing: ?LOG: could not receive data from client: Connection reset by peer" ?LOG: unexpected EOF on client connection with an open transaction? I store it like this: 1. // Create intial event 2. Event *new_event = new Event(type, description, location); 3. TimelineItem *new_item = new TimelineItem(new_event, start, end); 4. 5. // Add the new item to the timeline 6. timeline->addTimelineItem(new_item); 7. 8. // Persist TimelineItem 9. db->persist(new_event); 10. unsigned long update_id = db->persist(new_item); 11. db->update(*timeline); 12. 13. // Declare repeated items 14. vector repeat_items; 15. 16. // Create repeats (repeats - 1 because we make one less repeat because of new_item) 17. for(int i = 0; i < (repeats - 1); i++) { 18. TimelineItem *item = new TimelineItem(new_event, start, end, new_item); 19. repeat_items.push_back(item); 20. db->persist(item); 21. } 22. 23. // Get 24. unique_ptr update_item(db->query_one (timeline_item_query::id == update_id)); 25. 26. // Update initial item 27. update_item->setLinkedItems(repeat_items); 28. 29. db->update(*update_item); You can view the entire source file here: https://pastebin.com/1yrRLmt1 The function that I am particularly interested in is createEvent starting at line 99, however, any other advice is welcome (but be kind, this is my first time writing C++!). Thanks, Oliver Reid From boris at codesynthesis.com Mon May 29 08:53:07 2017 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon May 29 08:53:17 2017 Subject: [odb-users] Vector causing unexpected EOF error In-Reply-To: <0282509e-4060-43f3-8e81-da7fc35745e9@Spark> References: <0282509e-4060-43f3-8e81-da7fc35745e9@Spark> Message-ID: Hi Oliver, Oliver Reid writes: > ?LOG: could not receive data from client: Connection reset by peer" > ?LOG: unexpected EOF on client connection with an open transaction? What happens with your program? Is there an exception? Does it crash? What you are seeing on the server is that the client resets the connection unexpectedly which would normally indicate a crash. If it indeed crashes, run it under a debugger and try to understand what's going on. Boris From aongeeno at openmailbox.org Wed May 31 05:05:20 2017 From: aongeeno at openmailbox.org (aongeeno@openmailbox.org) Date: Wed May 31 05:05:38 2017 Subject: [odb-users] [Feature request] ODB should support CREATE TABLE IF NOT EXISTS for SQLite (and, preferably, for other drivers) Message-ID: <21082477.SB8ZGWT8Bl@99d44166af322fc16a4774097a5f271d> Good day! As I know there is no possibility to generate schema SQL with keywords "IF NOT EXISTS". This can be useful for embedded. In official examples there is schema/embedded. > // The following alternative version only creates the schema if it > // hasn't already been created. To detect the existence of the schema > // this version tries to query the database for a person object. If > // the corresponding table does not exist, then an exceptions will be > // thrown in which case we proceed to creating the schema. > // > /* > { > transaction t (db->begin ()); > try > { > db->query (false); > } > catch (const odb::exception& e) > { > schema_catalog::create_schema (*db); > } > t.commit (); > } > */ This looks like as dirty hack, because we can catch different odb::exception and need to know that this is not a driver problem. This makes code complicated and not so robust. Much better will be use supported feature CREATE TABLE IF NOT EXISTS. Any suggestions? From boris at codesynthesis.com Wed May 31 11:11:00 2017 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed May 31 11:11:08 2017 Subject: [odb-users] [Feature request] ODB should support CREATE TABLE IF NOT EXISTS for SQLite (and, preferably, for other drivers) In-Reply-To: <21082477.SB8ZGWT8Bl@99d44166af322fc16a4774097a5f271d> References: <21082477.SB8ZGWT8Bl@99d44166af322fc16a4774097a5f271d> Message-ID: Hi, aongeeno@openmailbox.org writes: > Much better will be use supported feature CREATE TABLE IF NOT EXISTS. This is not supported by all databases/versions. We could implement this for those that support it but we will soon have another problem: what if the table exists but from some older version of our application? To solve this you can use ODB's database schema evolution support (Chapter 13) which also takes care of properly detecting whether the schema does not yet exist. Boris